The relationship between consumer group and consumer is as follows:
A consumer can subscribe to multiple topics at the same time.
A topic can contain one or multiple partitions.
A partition can be consumed only by one consumer at a time.
Therefore, the maximum number of consumers in a consumer group is the total number of partitions in all topics.
Consumer refers to a consumer object in programming. There can be multiple consumers on a server. This is because a thread contains a consumer, and if multiple threads are initiated, there will be multiple consumers as shown below:
ConsumerRecords<String, String> records = consumer.poll(100);
}
You can configure the initial number of consumers based on the client resources and configure heap alarms for consumer groups. If messages are heaped, you can add more consumers. For more information on how to configure an alarm, see Configuring Alarm.
Why You Will Not See Consumer Detail
Consumer detail includes the Topic consumed by the consumer, partition progress, client information, etc. This information is stored by the server after the consumer processes and submits the offset. The client will not automatically submit the offset storage request to the server in the following scenarios:
1. Use the custom partition consumption mode.
2. Using Flink + CKafka for big data scenarios, the Flink cluster does not submit offsets or perform partition rebalance through the coordinator, so the server has no consumption status or pod IP information.
The CKafka consumer group list shows group names, but no consumption details when opened. For example: the CR consumer group in the figure below displays no details.
Possible Causes
Kafka supports two data consumption modes: consumer group mode and custom partition consumption mode.
When using consumer group mode, the client coordinates consumption through the consumption coordinator. Upon completion, it submits a storage request for the offset to the server. At this point, the server stores information such as the consumed Topic, partition progress, and client details.
When using custom partition consumption mode, the client does not automatically submit offset storage requests to the server. If the client does not actively initiate an offset submission request, the server will not see the relevant consumption information.
After a Topic has ACL set up, specific instances may be unable to view the consumer group details. If you cannot view the details, please first check whether there is an ACL. If there is, you are required to submit a ticket for processing.
Note: This will not show information about old Zookeeper-based consumers.
You will find no details for this consumer group. This indicates the consumer client does not use the consumerGroup mechanism to consume data. The client does not submit consumption details to the server, and the server does not store consumption data, so it will not display normally.
3. Locate whether the issue is on the server side.
4. Using the built-in consumer group command, specify the consumer group name test1 to consume, as follows:
]$ bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --from-beginning --topic test --group test1
For consumer groups that display normally on the console, you can view details using the --describe command, as shown below:
Subscribe Type Consumer Group and Assign Type Consumer Group Can Reuse the Same Group Id
In CKafka, the management methods for subscribe-type and assign-type consumer groups are entirely different. The information of subscribe-type consumer groups is fully managed by the CKafka server, including consumer meta-information, submitted offsets, and rebalance management. In contrast, for assign-type consumer groups, the logic of partition assignment for each consumer is entirely managed by the client itself, with the server only storing the offsets submitted by the assign-type consumer group. Therefore, if subscribe-type and assign-type consumer groups reuse the same group id, some unexpected behaviors may occur as follows:
1. If a subscribe-type consumer group is in stable status and consuming, then starting an assign-type consumer with the same group id may cause the assign-type consumer's offset submission to fail.
2. If an assign-type consumer group is consuming and submitting offsets, then starting a subscribe-type consumer group with the same group id may cause the subscribe-type group's submitted offsets to overwrite those of the assign-type group, leading to offset confusion and submission failure for the assign-type consumer group.
3. If a subscribe-type consumer group stops consumption after use and its status is empty, then starting an assign-type consumer with the same group id may allow normal consumption and offset submission. However, since the server stores this group id as a subscribe type, including its consumer meta-information, when the meta-information expires (default: 7 days), the consumer group will be directly deleted via subscribe-type expiration cleanup. This can cause synchronous deletion of the group's offsets, and the ongoing assign-type consumer group may encounter issues due to offset loss.
Therefore, we should avoid this improper usage. If you must use the same group id name while the consumer group types differ before and after, you need to first delete the consumer group in the console and then enable a new type of consumer group for consumption.
What to Do If Consumer Group Persistently Occurs in PreparingRebalance State
Possible Causes
1. A new consumer joins the consumer group.
2. When a running consumer stops functioning and leaves the consumer group. Common situations include consumer restart, consumer application crash, or heartbeat timeout reported by the consumer process (For details, refer to CKafka Common Parameter Configuration Guide).
3. When the number of partitions changes (add or delete).
Solution
Two situations, 1 and 3, cannot avoid rebalance. Under normal conditions, rebalance completes within 30s. If a long-term rebalance occurs, submit a ticket to handle it.
If the issue is caused by heartbeat timeout or two times poll interval too large, you can adjust the following parameters:
# When using the Kafka consumer grouping mechanism, the consumer timeout period. When the Broker does not receive a heartbeat from the consumer within this time, the consumer is considered to have failed, and the Broker initiates the Rebalance process. Currently, this value must be set in the Broker configuration between group.min.session.timeout.ms=6000 and group.max.session.timeout.ms=300000.
session.timeout.ms=10000
# When using the Kafka consumer grouping mechanism, the interval for consumers to send heartbeat. This value must be less than session.timeout.ms, normally less than one-third of it.
heartbeat.interval.ms=3000
# When using the Kafka consumer grouping mechanism, the maximum interval for subsequent calls to poll. If poll is not called again within this time, the consumer is considered to have failed, and the Broker will initiate Rebalance again to assign its partitions to other consumers.
max.poll.interval.ms=300000
If a consumer subscribes to many topics, you can try to reduce the number of topics subscribed by the Group.
