tencent cloud

Producer-Consumer Practice Tutorial
Last updated:2026-01-20 17:10:13
Producer-Consumer Practice Tutorial
Last updated: 2026-01-20 17:10:13
This article introduces the best practices for producing and consuming messages in CKafka, helping you reduce the possibility of message consumption errors.

Production Message

Using and Creating Topics

Configuration requirements: It is recommended to set the replica count as an integer multiple of the number of nodes to reduce data skew issues, and the minimum number of in-sync replicas for synchronous replication is 2, and the number of in-sync replicas must not equal the number of Topic replicas; otherwise, if one replica fails, it may cause inability to produce messages.
Before using CKafka for message production and consumption, you need to create a Topic resource in your existing CKafka instance. Producers send messages to a specified Topic, while consumers obtain messages by subscribing to the Topic for consumption. This article describes the procedure for creating a Topic in the CKafka console.

Partition Count Estimation

Accurate estimation of the partition count enables balanced data distribution. To achieve this, it is recommended to set the partition count as an integer multiple of the number of nodes. Additionally, configure it based on the estimated traffic, calculated at a standard of 10MB/s per partition. For example, if a Topic's estimated throughput is 100MB/s, set the partition count to 10. This ensures messages are evenly distributed across partitions under high traffic, preventing overload on any single partition.

Retry upon Failure

In distributed environments, message delivery may occasionally fail due to network issues. This could occur either because the message was successfully delivered but the ACK mechanism failed, or because the message was not delivered at all. To address this, it is essential to configure an appropriate retry policy. Based on your business requirements, you may set the following retry parameters:
retries: Used to set the number of retries, default value is 3. If retries are unsuccessful, it will trigger an error. If the customer does not accept message loss, it is recommended to change the number of retries or manually retry.
retry.backoff.ms: Sets the retry interval, recommended value is 1000. This interval allows the producer to wait for a period before retrying, avoiding frequent retries within a short time.
This approach can handle situations where the Broker's Leader partition cannot immediately respond to Producer requests.

Asynchronous sending

The message sending interface is typically asynchronous, meaning producers can proceed to other tasks without waiting for messages to be fully processed after sending. To receive delivery outcomes, use the Callback interface in the Send method to obtain the sending results.

One Producer Corresponds to One Application

Producers are thread-safe and can send messages to any Topic. One Producer per Application

Acks

Kafka's ACK mechanism refers to the Producer's message delivery acknowledgment mechanism. In Kafka version 0.10.x, the setting is Acks, while in version 0.8.x, it is request.required.acks. The Acks parameter determines how the Producer waits for server responses after sending messages, directly impacting the Kafka cluster's throughput and message reliability.
The parameters of Acks are described as follows:
When Acks=0, producers operate without an acknowledgment mechanism. After sending a message, they proceed without waiting for responses from any Broker nodes. This mode delivers the highest throughput performance but poses a high risk of data loss due to the absence of write guarantees.
When Acks=1, with the primary node single-acknowledgment mechanism, the producer only needs to wait for the Leader replica to complete message writing before receiving an acknowledgment response. This mode balances performance and reliability. Note: If the Leader node fails before synchronization completes, messages that have been sent but not yet replicated may be partially lost.
When Acks=all, with the full-replica acknowledgment mechanism enabled, the producer must wait for both the Leader replica and all in-sync replicas (ISR set) to persist the message before receiving an acknowledgment. While this mode achieves the highest level of data security through multiple redundancies (data loss occurs only if the entire ISR cluster fails simultaneously), the latency introduced by cross-node synchronization results in relatively lower throughput performance.
Generally, Acks=1 is recommended; for critical services, Acks=all can be set.

Batch

Typically, CKafka Topics are configured with multiple partitions. When a Producer client sends messages to the server, it must first determine which partition of which Topic to send the messages to. When sending multiple messages to the same partition, the Producer client aggregates these messages into a Batch and transmits them to the server in bulk. However, processing Batches incurs additional overhead for the Producer client. Generally, small batches cause the Producer client to generate a large number of requests, leading to request pile-up and queuing on both client and server sides. This also increases CPU utilization on related machines, thereby increasing the overall latency of message production and consumption. Setting an appropriate Batch size reduces the number of requests sent from the client to the server, thereby improving the overall throughput of message delivery.
The following describes the Batch-related parameters:
Batch.size: This is the threshold for the message cache size per partition. When the cached message size reaches this set value, it triggers a network request, and the Producer client then sends the messages to the server in batches.
Linger.ms: This specifies the maximum linger time for messages in the cache. If a message remains in the cache longer than this value, the Producer client no longer adheres to the Batch.size limit and immediately sends the messages to the server.
Buffer.memory: When the total size of all cached messages exceeds this value, it triggers sending messages to the server, bypassing the limitations of both batch.size and linger.ms. The default value of Buffer.memory is 32MB, which is sufficient to ensure the performance of a single Producer.

Key and Value

Messages in a message queue contain two fields: Key (message identifier) and Value (message content). Setting a unique Key for messages facilitates tracking. By viewing the sending and consumption logs, you can monitor the production and consumption status of messages. For example, in an e-commerce order system, using the order ID as the Key allows easy tracking of the message flow. For high-volume message production, it is recommended to omit keys and adopt a sticky partitioning policy.

Sticky Partitioning

In Kafka message queues, only messages sent to the same partition are grouped into the same Batch. Therefore, the partitioning policy configured on the Kafka Producer side is one of the key factors affecting Batch formation. Kafka Producer allows users to implement custom Partitioner classes to select appropriate partitioning methods that align with business requirements. When a message specifies a Key, Kafka Producer by default first hashes the message's Key, then selects the partition based on the hash result, ensuring messages with the same Key are sent to the same partition.
When messages do not specify a Key, prior to Kafka version 2.4, the default partitioning policy cycled sequentially through all partitions under a topic, distributing messages to each partition in a round-robin manner. However, this default policy performed poorly in Batch aggregation, often generating numerous small batches in practice, thus increasing actual message processing latency. To address the inefficiency of partitioning messages without Keys, Kafka introduced the Sticky Partitioning Strategy in version 2.4.
The Sticky Partitioning Strategy specifically addresses the issue of messages without Keys being scattered across different partitions, which leads to numerous small batches. Its core mechanism is that once a Batch in a partition is processed, it randomly selects another partition and then directs subsequent messages to this newly chosen partition as much as possible. From a short-term perspective, messages are concentrated in the same partition; however, over long-term operation, messages remain evenly distributed across all partitions. This approach avoids partition skew while also reducing latency and enhancing overall service performance.
If your Kafka Producer client version is 2.4 or above, the sticky partitioning policy is enabled by default. For client versions below 2.4, you can implement a custom partitioning policy based on the sticky partitioning principle and specify it via the partitioner.class parameter.
The following provides a Java code example for implementing the Sticky Partitioning Strategy. Its core logic involves switching partitions at fixed intervals.
public class MyStickyPartitioner implements Partitioner {

// Record the time of the last partition switch.
private long lastPartitionChangeTimeMillis = 0L;
// Record the current partition.
private int currentPartition = -1;
// Partition switching interval. You can select the interval for switching partitions based on actual business needs.
private long partitionChangeTimeGap = 100L;

public void configure(Map<String, ?> configs) {}

/**
* Compute the partition for the given record.
*
* @param topic The topic name
* @param key The key to partition on (or null if no key)
* @param keyBytes serialized key to partition on (or null if no key)
* @param value The value to partition on or null
* @param valueBytes serialized value to partition on or null
* @param cluster The current cluster metadata
*/
public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {

// Get all partition information.
List<PartitionInfo> partitions = cluster.partitionsForTopic(topic);
int numPartitions = partitions.size();

if (keyBytes == null) {
List<PartitionInfo> availablePartitions = cluster.availablePartitionsForTopic(topic);
int availablePartitionSize = availablePartitions.size();

// Determine the currently available partitions.
if (availablePartitionSize > 0) {
handlePartitionChange(availablePartitionSize);
return availablePartitions.get(currentPartition).partition();
} else {
handlePartitionChange(numPartitions);
return currentPartition;
}
} else {
// For messages with Keys, select partitions based on the hash value of the Key.
return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;
}
}

private void handlePartitionChange(int partitionNum) {
long currentTimeMillis = System.currentTimeMillis();

// If the partition switching interval is exceeded, switch to the next partition, otherwise select the previous partition.
if (currentTimeMillis - lastPartitionChangeTimeMillis >= partitionChangeTimeGap
|| currentPartition < 0 || currentPartition >= partitionNum) {
lastPartitionChangeTimeMillis = currentTimeMillis;
currentPartition = Utils.toPositive(ThreadLocalRandom.current().nextInt()) % partitionNum;
}
}

public void close() {}

}

Partition Order

Within a single partition, messages are stored in the order they are sent, making them essentially sequentially ordered. Each topic has several partitions. If messages are assigned to different partitions, the order between different partitions cannot be guaranteed.
To ensure message consumption ordering, you can specify a Key for such messages on the producer side. When messages with the same Key are published, CKafka will select a partition for storage based on the hash modulo of the Key. Since a partition can only be consumed by a single consumer, this ensures ordered message consumption.

CKafka Ordered Messages Scenarios: A Hands-on Tutorial

Ordered Message Scenario

In TDMQ for CKafka, the primary approach to ensuring message ordering relies on its partition design and the use of message Keys. Client-side scenarios involving ordered messages fall into two categories: global ordering scenarios and partition ordering scenarios. For these scenarios, CKafka's best practices are as follows:
1. Global Order: To ensure global ordering, in the CKafka console, you need to set the Topic partition to 1. The number of replicas can be specified based on specific usage scenarios, availability requirements, and cost balance. It is recommended to set it to 2.
Global Order: Due to the throughput limit of a single partition, the overall throughput is not very high. For the throughput metrics of a single partition, see Usage Limits.
2. Partition Ordering: To ensure partition ordering, you can calculate the number of partitions by dividing the estimated business traffic by the traffic per partition and rounding up. To avoid data skew, round the partition count to a multiple of the number of nodes to determine a reasonable final partition count. For single-partition throughput, see Usage Limits. When sending Kafka messages, specify a Key. Kafka calculates a hash value based on the Key to ensure messages with the same Key are sent to the same partition, maintaining internal ordering. It is also recommended to distribute business Keys as widely as possible. If all produced messages use the same Key, partition ordering degrades to global ordering, reducing overall write throughput.

Parameter Best Practices

Due to the requirements of ordered and non-duplicated messages in sequential messaging scenarios, the default Kafka producer parameters may cause message duplication or disorder when encountering network jitter, Kafka Broker node changes, or partition Leader elections. Therefore, for sequential scenarios, Kafka producer parameters must be specifically configured with the following key settings:
Enable.idempotence
Enable.idempotence indicates whether to enable the idempotence feature. For sequential scenarios, enabling idempotence is recommended to address issues such as out-of-order messages and message duplication in the aforementioned scenarios. It is advised to set Kafka Producer's enable.idempotence to true. Note that this feature requires Kafka Broker version 0.11 or higher (i.e., Kafka versions >= 0.11). Additionally: Starting from Kafka 3.0 (including 3.0), Kafka Producers default to enable.idempotence=true and Acks=All. For Kafka versions >=0.11 but <3.0, idempotence is disabled by default. Therefore, in sequential scenarios, explicitly set this parameter to ensure idempotence is enabled.
Acks
After enabling idempotency, Acks must be set to All. If not explicitly set to All, the configuration will fail validation and result in an error.
Max.in.flight.requests.per.connection
By default, the Kafka producer attempts to send records as quickly as possible. Max.in.flight.requests.per.connection specifies the maximum number of requests that can be sent concurrently on a single connection, with a default value of 5. For Kafka versions including 0.11 and later but earlier than 1.1 (i.e., Kafka >= 0.11 & < 1.1), the Kafka Broker lacks optimizations in this aspect and requires configuration.
When max.in.flight.requests.per.connection is set to 1, for Kafka versions >=1.1, optimizations have been made for throughput in idempotent scenarios. The Broker maintains a queue to sequentially validate messages from five concurrent batches, allowing max.in.flight.requests.per.connection to be set to 5 but not exceeding 5.
Therefore, it is recommended:
For Kafka >= 0.11 & < 1.1: explicitly set max.in.flight.requests.per.connection to 1.
For Kafka>=1.1: explicitly set max.in.flight.requests.per.connection to 1 <= max.in.flight.requests.per.connection <= 5; it is recommended to be set to 5.
Retries
In sequential scenarios, it is recommended to specify the retry parameter. The default behavior of Retries varies across versions: for Kafka <= 2.0, the default is 0; for Kafka >= 2.1, the default is Integer.MAX_VALUE (2147483647). It is advised to set a reasonable number of retries based on business requirements.
Summary
In sequential scenarios, the following is an example of producer properties that need to be enabled:
Kafka >= 0.11 & < 1.1:
// create Producer properties
Properties properties = new Properties();
properties.setProperty("enable.idempotence", "true");
properties.setProperty("acks", "all");
properties.setProperty("max.in.flight.requests.per.connection", "1");
properties.setProperty("retries", Integer.toString(Integer.MAX_VALUE));
Kafka>=1.1:
// create Producer properties
Properties properties = new Properties();
properties.setProperty("enable.idempotence", "true");
properties.setProperty("acks", "all");
properties.setProperty("max.in.flight.requests.per.connection", "5");
properties.setProperty("retries", Integer.toString(Integer.MAX_VALUE));

Data Skew

Kafka Broker data skew issues are typically caused by uneven partition distribution or uneven Key distribution in producer-sent data, which can lead to several problems:
1. Overall traffic is not rate-limited, but localized rate limiting occurs at the node level.
2. Excessive load on certain nodes leads to low overall Kafka utilization, impacting the overall throughput.

Such issues can be optimized through the following methods:
1. Use an appropriate number of partitions, ensuring the partition count is a multiple of the number of nodes.
2. Reasonable partition policies, such as RoundRobin (round-robin), Range (range), and Sticky (sticky) or custom partition policies, to evenly distribute messages.
3. Check whether keys are used for message sending. If keys are employed, design strategies to achieve better partition balancing for the keys.

Consuming Messages

The basic process of consuming messages is straightforward. First, Poll data: consumers pull messages from the message queue. Next, execute consumption logic to process the retrieved messages. After processing, Poll data again, repeating this cycle. For example, in an e-commerce order processing system, consumers pull order messages from the queue, perform operations such as inventory deduction and order status updates based on the order information, and then proceed to pull the next batch of order messages.

CLB Mechanism

Load Balancing

Load balancing plays a critical role in the consumption process. Each Consumer Group can contain multiple Consumers. By setting the parameter group.id to the same value, these Consumers belong to the same Consumer Group and collectively consume the subscribed Topics.
For example, Consumer Group A subscribes to Topic A and enables three consumer instances C1, C2, and C3. Each message sent to Topic A will ultimately be delivered to only one of C1, C2, or C3. By default, TDMQ CKafka evenly distributes messages to consumer instances to achieve consumption load balancing.
The internal mechanism of load balancing in TDMQ CKafka Edition involves evenly distributing partitions of subscribed Topics among Consumers. Therefore, the number of Consumers should not exceed the number of partitions; otherwise, some Consumer instances may be allocated no partitions and remain idle. It is advisable to ensure the number of consumers is evenly divisible by the total number of partitions. Other than during initial startup, any subsequent changes—such as Consumer instance restarts, additions, reductions, or partition count increases—will trigger a rebalance.

Handling Rebalancing

Frequent occurrences of rebalance may stem from various reasons.
1. Consumer message processing is time-consuming, for example, when handling complex business logic which may require multiple database queries or remote interface calls, resulting in slower consumption.
2. Consuming an abnormal message may also cause the consumer to block or fail, for example, due to a malformed message that cannot be parsed.
3. Heartbeat timeout can also trigger rebalance.
4. For Kafka clients in open-source Java versions prior to v0.10.2, the Consumer lacked a dedicated thread to maintain heartbeats. Instead, heartbeat maintenance was coupled with the Poll interface. If message consumption stalled, this could cause the Consumer's heartbeat to time out, triggering rebalance. In clients from v0.10.2 onward, if consumption is too slow and exceeds the configured time (set by max.poll.interval.ms, defaulting to 5 minutes) without a Poll operation to fetch messages, the client will actively leave the group, initiating Rebalance.

Solutions include optimizing consumption processing, increasing consumption speed, and adjusting parameters:
1. The consumer must maintain the same version as the Broker.
2. You can adjust parameter values by referring to the following instructions:
session.timeout.ms: For versions prior to v0.10.2, moderately increase this parameter value to exceed the time required to consume a batch of data, but do not exceed 30s. A setting of 25s is recommended. For v0.10.2 and later versions, remain at the default value of 10s;
max.poll.records: Reduce this parameter value. It is recommended to be significantly less than the number of messages consumed per second per thread * the number of consumer threads * max.poll.interval.ms / 1000;
max.poll.interval.ms: This value must be greater than max.poll.records / (the number of messages consumed per second per thread * the number of consumer threads).
3. Endeavor to enhance the client's consumption speed by processing consumption logic in a separate thread and monitoring time consumption.
4. Reduce the number of Topics subscribed by a Group. It is recommended that a Group subscribes to no more than 5 Topics, with a single Topic per Group being the optimal configuration.

Topic Subscription Relationship

Regarding subscription relationships, within the same Consumer Group, it is recommended that clients maintain consistent Topic subscriptions—ideally one Consumer Group per Topic—to avoid introducing additional complexity in troubleshooting.

Consumer Group Subscribing to Multiple Topics

A Consumer Group can subscribe to multiple Topics, where messages from these Topics will be evenly consumed by Consumers within the Consumer Group. For instance, if Consumer Group A subscribes to Topic A, Topic B, and Topic C, messages from all three Topics will be uniformly consumed by Consumers in the Group.
The sample code for a Consumer Group subscribing to multiple Topics is as follows:
String topicStr = kafkaProperties.getProperty("topic");
String[] topics = topicStr.split(",");
for (String topic: topics) {
subscribedTopics.add(topic.trim());
}
consumer.subscribe(subscribedTopics);

Topics Are Subscribed to by Multiple Consumer Groups

A Topic can be subscribed to by multiple Consumer Groups, with each group independently consuming all messages under the Topic. For example, if Consumer Group A subscribes to Topic A and Consumer Group B also subscribes to Topic A, every message sent to Topic A will be delivered to both consumer instances in Group A and Group B. These two processes operate independently and remain unaffected by each other.

One Consumer Group per Application

It is recommended that one Consumer Group corresponds to one application, meaning different applications should use distinct code implementations. If you need to implement different logic within the same application, prepare multiple separate kafka.properties files, such as kafka1.properties and kafka2.properties.

Consumer Offset Explanation

Each Topic has multiple partitions. Each partition counts the total number of messages in the current partition, which is called the maximum offset (MaxOffset).
TDMQ CKafka Consumer sequentially processes each message within a partition and records the number of messages consumed, known as the ConsumerOffset.
The number of unconsumed messages (also known as message backlog) = MaxOffset-ConsumerOffset.

Offset Commit

TDMQ CKafka edition's Consumer has two related parameters:
enable.auto.commit: Default value is true.
auto.commit.interval.ms: Default value is 5000, i.e. 5s.
The combined effect of these two parameters is: Before each Poll operation, the system checks the time elapsed since the last offset commit. If this duration exceeds the interval specified by auto.commit.interval.ms, the client will initiate an offset commit.
Therefore, if enable.auto.commit is set to true, you must ensure that the data from the previous Poll operation has been fully consumed during each subsequent Poll. Otherwise, it may cause offset skipping.
If you wish to manually control offset commits, set enable.auto.commit to false and use the Commit(Offsets) function to manage offset submissions.
Note:
Avoid excessively frequent commit offset requests, as this can lead to high Broker CPU utilization and impact normal service. For example, when auto.commit.interval.ms is set to 100ms for automatic offset commits, or when offsets are manually committed after processing each message in high-throughput scenarios.

Reset Offset

There are two cases where the consumer offset will be reset:
When the server-side does not have a previously committed offset (for example, during the client's first launch or the initial consumption of newly expanded partitions).
When messages are pulled from an invalid offset (For example, the maximum offset of a partition is 10, but the client starts pulling messages from 11).
The Java client can configure the reset strategy through auto.offset.reset. There are three main strategies:
Latest: Start consumption from the latest offset.
Earliest: Start consumption from the earliest offset.
None: Take no action, meaning no reset will be performed.
Note:
It is recommended to set it to Latest instead of Earliest to avoid starting consuming data from the beginning when the offset is invalid, which could result in a large number of duplicates.
When partitions are scaled out, the initial consumption of new partitions triggers offset initialization. Since producers may detect new partitions before consumers and start producing data, using the Latest configuration may cause messages in new partitions to be skipped. The following solutions are recommended:
Option 1: directly set auto.offset.reset=earliest.
Option 2: To maintain the Latest configuration, follow the steps below when partitions are scaled out:
1.1.1 Temporarily modify the consumer-side configuration to earliest before scaling out partitions.
1.1.2 Restart the consumer client.
1.1.3 Scale out partitions;
1.1.4 After scaling out partitions is completed, you can optionally revert to the latest configuration.
If your business operations manage consumption offsets independently, you can set it to None.

Message Pulling Optimization

Pulling Messages

The consumption process is initiated by the client to pull messages from the server. When pulling large messages, it is necessary to control the pulling speed and pay attention to the following parameter settings:
max.poll.records: If a single message exceeds 1MB, it is recommended to set it to 1.
max.partition.fetch.bytes: Set it to a value slightly larger than the size of a single message.
fetch.max.bytes: Set it to a value slightly larger than the size of a single message.
When consuming messages via a public network, connections are often disconnected due to public network bandwidth limitations. In such cases, it is necessary to control the pulling speed and modify the configuration:
fetch.max.bytes: It is recommended to set it to half of the public network bandwidth (note that the unit of this parameter is bytes, while public network bandwidth is measured in bits).
max.partition.fetch.bytes: It is recommended to set it to one-third or one-fourth of fetch.max.bytes.

Pulling Large Messages

The consumption process is initiated by the client to pull messages from the server. When pulling large messages, it is necessary to control the pulling speed and modify the configuration:
max.poll.records: The maximum number of messages obtained per Poll. If a single message exceeds 1 MB, it is recommended to set this value to 1.
fetch.max.bytes: Set it to a value slightly larger than the size of a single message.
max.partition.fetch.bytes: Set it to a value slightly larger than the size of a single message.
The core of pulling large messages lies in pulling them one by one.

Exception Handling in Message Consumption

Message Duplication and Message Idempotence

The consumption semantics of TDMQ CKafka edition is at least once, meaning messages are delivered at least once to ensure no loss, but duplication cannot be guaranteed. A small number of duplicate messages may occur during network issues or client restarts. If the application consumer is sensitive to message duplication (e.g., in order transactions), it should implement message idempotency.
Taking database applications as an example, the common practice are:
When sending a message, pass in a Key as the unique serial ID.
When consuming a message, determine whether the Key has been consumed. If it has been consumed, skip it; if not, consume it once.
Of course, if the application itself is not sensitive to a small number of duplicate messages, there is no need to perform such idempotency checks.

Consumption Failure

TDMQ CKafka edition processes messages sequentially by partition. If the consumer fails to execute the consumption logic after receiving a message—for example, due to dirty data on the application server causing message processing failure and requiring manual intervention—the following two approaches can be adopted:
Keep trying to execute the consumption logic after failure. This method may cause the consumer thread to be blocked on the current message and unable to move forward, resulting in a message backlog.
Since Kafka does not have a built-in mechanism for handling failed messages, common practice involves logging failed messages or storing them in a dedicated service (such as creating a special Topic for failed messages). Periodically check these failed messages, analyze the causes of failure, and handle them accordingly.

Consumption Delay

The consumption process is initiated by the client to pull messages from the server. In general, if the client can consume in time, there will be no significant delay. If there is a significant delay, please first check if there is a backlog and pay attention to improve the consumption speed.

Consumption Backlog

Common causes of message backlog are:
The consumption speed cannot keep up with the production speed, so the consumption speed should be improved.
There is a blockage on the consumer.
After receiving the message, the consumer executes the consumption logic, which usually involves some remote calls. If the consumer waits for the result synchronously at this time, the consumer may continue to wait and the consumption process cannot move forward.
The consumer should try to avoid blocking the consumption thread. If there is a situation where the call result is waiting, it is recommended to set a timeout for the wait. After the timeout, it is treated as a consumption failure.

Improving Consumption Speed

Increasing the number of Consumer instances enhances parallel processing capabilities. If the ratio of consumers to partitions is already 1:1, consider increasing the number of partitions (Note: For scenarios where Flink automatically maintains partitions, it may not automatically detect newly added partitions, potentially requiring code modifications and restarts). You can scale within the process (ensuring one thread per instance) or deploy multiple consumer instance processes.
Note:
When the number of instances exceeds the number of partitions, the speed can no longer be increased and some consumer instances will not work.
Increase the number of consumer threads.
1. Define a thread pool.
2. Poll data.
3. Submit the data to the thread pool for concurrent processing.
4. After the concurrent results are successfully returned, re-attempt polling data for execution.

Message Broadcasting

Kafka currently does not support broadcast semantics for messages, but this can be simulated by creating different consumer groups.

Message Filtering

Kafka itself does not support message filtering semantics. In practice, the following two approaches can be adopted:
If there are not many types of filtering, adopting multiple Topics can achieve the filtering purpose.
If there are numerous types of filtering, it is preferable to perform filtering at the client-side business logic layer.
In practice, make a choice based on your specific business scenario, or combine both approaches mentioned above.

Some Partitions Consumed While Others Not

During message consumption, a consumer may remain online while the offsets of certain partitions fail to advance. Possible reasons include:
1. Encountering an abnormal message, possibly a large message or format exception, causes issues when the consumer pulls the message during conversion to the business offset.
2. When using public network bandwidth, the bandwidth is limited, and pulling large messages can saturate the bandwidth, resulting in failure to pull messages within the timeout period.
3. When a consumer enters a zombie state, it fails to poll messages.
Solution Approach:
Shut down the consumer, set the offset in the TDMQ CKafka console to skip certain exception messages, or optimize the consumption code and restart the consumer.

Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback