How Do I Choose an Appropriate Number of TDMQ for CKafka (CKafka) Replicas?
It is recommended that you select 2 or 3 replicas for data storage when creating a topic to ensure data reliability. A created topic has 2 replicas by default. If your business requires higher availability, you can select 3 replicas. If you need more replicas for your topic, submit a ticket for assistance.
To improve data security, CKafka does not recommend creating single-replica topics. If you have a single-replica topic in your account, it is recommended to migrate it as follows:
1. Create a topic, select the same partition, and select two replicas.
2. Produce messages in the new topic while the existing single-replica topic continues to be consumed.
3. Modify the consumer configuration after consumption is completed to subscribe to the new topic for consumption.
Why Can Messages Still Be Queried After the Message Retention Period Set for the Topic Has Elapsed?
1. A timestamp field and timestamp types are added to messages. Currently, two timestamp types are supported: CreateTime and LogAppendTime. The former indicates the time when the message is created by the producer, and the latter indicates the time when the broker receives the message. If the timestamp data of the time when a client produces a message is invalid, data deletion on the broker server will be affected.
2. If there are too many partitions in the topic and too little message data, and only one log segment file exists in the partition, messages will not be deleted.
3. The log deletion task checks whether the current log size exceeds the set threshold, namely 1 GB per segment. If the maximum timestamp data in the log segment is still within the retention period, messages will not be deleted.
Why Is the Number of Topics (Total Number of Partitions) Limited?
A large number of topics (total number of partitions) in Kafka will compromise cluster performance and stability.
As the number of partitions that a server can sustain is limited, if you want more partitions, you need to add more nodes, which incur higher costs. The storage and coordination mechanisms of Kafka work by partition. If there are too many partitions, storage fragmentation will be severe, random writes on individual servers will increase, the efficiency of leader switching in the cluster will decrease, controller node failover will slow down, and other problems may occur, which lower cluster performance and stability.
What Is the Relationship Between the Number of Topics and the Number of Partitions?
CKafka uses partitions as an assignment unit.
Total number of partitions = Topic A x Number of replicas + Topic B x Number of replicas + ... + Topic N x Number of replicas.
The number of replicas is also counted into the total number of partitions. For example, if you create 1 topic with 6 partitions and 2 replicas for each partition, then you have a total of 12 partitions. The formula is: 1 x 6 x 2 = 12.
Number of partitions: a concept in physical partitioning. One topic can contain one or more partitions.
Number of replicas: number of partition replicas. It is used to ensure the high availability of the partition. To ensure data reliability, creating a single-replica topic is not currently supported. Two replicas are enabled by default.
What Should I Do If Topic Creation Fails?
The sum of partition numbers of all created topics has reached the maximum number of partitions for the instance specification.
It is recommended to scale out the Kafka instance or delete unnecessary topics.
Create a topic with the same name during topic deletion.
Topic deletion is an asynchronous operation. After the deletion command is issued, the system will asynchronously delete the metadata of the topic. If you create a topic with the same name during this period, the system will notify you that the topic already exists. Try again later at that time. Then, you need to wait about 1 minute before proceeding with the recreation operation.
The topic already exists in the cluster.
The topic already exists in the cluster. Creating a topic with the same name will prompt that the topic already exists. It is recommended to reset the topic name.
