What Should I Do If No Data Is Visible When I Test the Kafka Console Client?
When consumers choose to consume data from the latest offset, they will only obtain the latest data. They need to maintain production at the same time to see the corresponding data.
Switch to consuming data from the earliest offset.
What Should I Do If Production or Consumption Errors Occur When a New Client Is Connected?
Check whether telnet is connected (network issue: whether Kafka and the producer are in the same network environment).
Check whether the accessed VIP address and port number are configured correctly.
How Do I Ensure That the Client Produces Messages in Order in the Same Partition?
If a topic has only one partition, messages are stored in the order they are received by the server. Therefore, the data is ordered in the partition.
If a topic has multiple partitions, you can specify a key for such messages on the producer. When messages are sent with the same key, TDMQ for CKafka (CKafka) will select one of the partitions for storage based on the hash modulo of the key. Since a partition can only be listened to and consumed by a single consumer, this ensures ordered message consumption.
For a single producer, production to a single partition remains ordered.
How Many Connections Does a Client Typically Establish with a Broker When Producing Messages?
From the perspective of a single client instance (creating a producer object), the total number of connections established with all servers is calculated as follows: Total number of connections = 1–n (where n is the number of brokers).
Each Java producer manages TCP connections in the following way:
1. When a KafkaProducer instance is created, a sender thread is started to create a TCP connection with all brokers in bootstrap.servers.
2. After the KafkaProducer instance updates its metadata information for the first time, it will create TCP connections with all brokers in the cluster again.
3. If the producer finds no TCP connection to a broker when sending a message to it, the connection will be created immediately.
4. If the connections.max.idle.ms parameter on the producer is set to be greater than 0, the TCP connections created in Step 1 will be automatically closed. By default, this parameter is set to 9 minutes, meaning that if no requests pass through a TCP connection within 9 minutes, Kafka will proactively close it. If the parameter is set to -1, the TCP connections created in Step 1 cannot be closed, resulting in zombie connections.
After Sending a Message via the Client, How Do I Determine Whether It Has Been Sent Successfully?
Most clients return Callback or Future after sending a message. If the callback is successful, it indicates that the message was sent successfully.
You can also verify whether the message was sent successfully in the console using the following methods:
By checking the partition status of a topic, you can view the number of messages in each partition in real time.
By checking the traffic monitoring of a topic, you can view the traffic curve for message production.
You can confirm whether the message was sent successfully by printing the partition information returned by the send method:
If the partition and offset can be printed, it indicates that the sent message has been correctly saved on the server. You can then use a message query tool to query messages at relevant offsets.
If the partition and offset cannot be printed, it indicates that the message was not saved on the server, and the client needs to retry.
What Is a Leader Switch?
When creating a topic, the Kafka broker cluster assigns a leader for each partition, distributing the partitions of the current topic evenly across each broker.
However, after a period of usage, partitions may become unevenly distributed across brokers.
Or the client may throw exceptions such as BrokerNotAvailableError and NotLeaderForPartitionError during production or consumption.
This is usually caused by a partition leader switch. Typical scenarios are as follows:
When unexpected events occur on the broker where a partition leader resides, such as network interruptions, program crashes, or hardware faults, and result in the inability to communicate with the broker controller:
The current topic partition will undergo a leader switch, with the leader migrating to a follower partition.
When auto.leader.rebalance.enable = true is set for the Kafka cluster and a rebalance is performed automatically, or when brokers are manually added/removed and a rebalance is manually triggered:
A leader switch will also occur at this time since an automatic partition rebalance is involved.
When a leader switch occurs due to an unexpected broker outage:
If the client sets ack = all and min.insync.replicas > 1, the message will not be lost because it is acknowledged on both the leader partition and follower partition.
If the client sets ack = 1, messages within the replica.lag.time.max.ms period may not have been synchronized to the follower partition, potentially causing message loss.
When a leader switch is triggered by manual/automatic rebalancing (such as during instance upgrades, switching from single-availability zone (AZ) deployment to cross-AZ deployment, or instance migration) while brokers are functioning normally, it does not cause message loss for the following reasons:
If the client sets ack = all and min.insync.replicas > 1, the message will not be lost because it is acknowledged on both the leader partition and follower partition.
If the client sets ack = 1, the leader switch will automatically synchronize the offsets in the partition. Therefore, messages will not be lost.
Can Kafka Clients Connect Directly to the CKafka Service?
You can connect to the CKafka service using mainstream Apache Kafka clients and deploy your code in Tencent Cloud services to produce or consume messages.
