CKafka provides rate limiting protection at both the cluster level and Topic Level. The configuration instructions are as follows:
Configuring Cluster-Level Rate Limiting Rules
After purchasing the specified peak bandwidth, the system will monitor business traffic usage in real time and automatically determine whether the cluster exceeds the specified usage based on the throttling mechanism. If traffic exceeding the purchase specification threshold is detected, the system will automatically trigger rate limiting protection with no special configuration required. For detailed throttling mechanism, refer to Throttling Mechanism Description.
If you have purchased a Pro Edition instance, you can choose to enable elastic bandwidth to handle sudden traffic spikes and avoid business interruption due to throttling. For specific steps, refer to Enable Elastic Bandwidth.
Configuring Topic Throttling Rules
You can set throttling rules for a Topic to prevent excessive traffic on one Topic from impacting others.
Note
Only Broker Version 1.1.1, 2.4.1, and 2.8.1 support Topic throttling rules.
2. Select instance list in the left sidebar, click the ID of the target instance to go to the instance basic information page.
3. On the instance basic information page, select the topic list tab, click more in the Operation column of the target Topic, then select traffic throttling.
4. Enable traffic throttling in the pop-up window and configure the throttling rule.
Maximum production traffic for a topic: excluding replica traffic, value range from 1MB/s to the maximum bandwidth of the instance purchase divided by the number of topic replicas.
Maximum consumption traffic per topic: value range from 1MB/s to the maximum bandwidth of the purchased instance. The underlying layer performs rate limiting against brokers. The actual rate limit value (equal to an integral multiple of the broker count) may vary slightly from the set throttling value. For details about the soft throttling mechanism, please refer to Throttling Mechanism Description.
CKafka Traffic Throttling Best Practices
Partition Quantity Planning and Local Traffic Throttling
CKafka instances have multiple nodes in distributed deployment mode to provide overall production and consumption services, and fixed read and write traffic throttling quotas are set for each node. To better utilize CKafka traffic, customers need to ensure the number of partitions is a multiple of that of nodes (in such case, CKafka will try to configure the same number of partitions on each node) to balance traffic as much as possible. (The write traffic may be uneven in special scenes such as specifying the message key. By default, the CKafka client tries to balance traffic across partitions when sending requests to the server). This effectively avoids local traffic throttling issues caused by local hotspots.
Explanation of Instance Traffic Throttling Count and Delayed Packet Response Monitoring
The instance traffic throttling count of CKafka refers the sum of traffic throttling operations on all nodes. It does not reflect the overall production and consumption performance of the instance, nor does it indicate that all nodes have triggered traffic throttling. Therefore, when the number of traffic throttling operations is high but the overall throttled traffic appears lower than the specification, customers can view the traffic throttling count on each node on the Advanced Monitoring tab to check the nodes on which traffic throttling has been performed.
When such issue occurs, it is recommended to adjust the number of partitions to be a multiple of the number of nodes to improve the overall bandwidth utilization of the instance. Currently, CKafka adopts the delayed packet response policy. Therefore, after traffic throttling is performed, you need to pay attention to the metric packet response delay . This metric can be viewed on the Advanced Monitoring tab in Pro Edition.
Traffic Throttling Difference Between Production and Consumption
Production involves replica synchronization. Therefore, the number of replicas is counted. Consumption only pulls data from the leader, so the number of replicas is not counted. Maximum production traffic of a single node = specification / number of nodes / number of replicas; maximum consumption traffic of a single node = specification / number of nodes.
Explanation of Occasional Throttling
Replicas consume traffic, which means that the write traffic decreases as the number of replicas increases. Throttling is actually performed on nodes and monitored by second, while customers view the data of monitoring by minute. Therefore, traffic throttling is more sensitive. When the overall write traffic exceeds 70% (excluding replicas) of the specification, second-level local throttling may occur on some nodes. Customers can view the specific node traffic on the Advanced Monitoring tab. If the requirement for response time is high, try to reserve 30% of the specification as buffer.
Explanation of Continuous Throttling count
Instance traffic throttling has been performed, Advanced Monitoring shows that continuous traffic throttling occurs on all nodes and the instance traffic is 10% higher than the specification, and the impact of Topic traffic throttling rules is excluded. This is unexpected. For such issues, submit a ticket.
FAQs
Why Is the Actual Rate Limit Value Less Than the Configured Threshold
The configured throttling quota will be distributed to each Broker (single machine rate limiting). Therefore, the instance dimension throttling will be distributed as throttling value/Broker count, and the Topic throttling value will also be distributed as throttling value/Broker count.
When the Topic partition count < Broker count or partition traffic imbalance occurs, it may cause advance throttling. In such cases, it is advisable to add partitions or optimize partition traffic. For specific steps, refer to Changing Instance Specifications and Configuring Topic Partition Balancing.
Why is traffic throttling triggered when the monitored production/consumption traffic is lower than the instance specification?
As mentioned above, traffic throttling is measured in milliseconds, but the monitoring platform in the console collects data at the second level and aggregates the data at the minute level (maximum or average value).
According to the principle of token bucket, a single bucket does not forcibly throttle the traffic. Suppose the bandwidth specification of instance A is 100 MB/sec, then the traffic throttling threshold of each 100-ms time bucket is 100 MB/10 = 10 MB/bucket. If the production traffic of instance A reaches 30 MB in the first 100-ms time bucket of a certain second (3 times the threshold), then the broker's traffic throttling policy will be triggered to increase the delayed response time. Suppose the original normal TCP response time is 100 ms, then the delay may be increased by 500 ms before response after the threshold is exceeded. The final traffic in this second is 30 MB 1 + 0 MB 5 + 10 MB * 4 = 70 MB, so the traffic speed in this second (70 MB/sec) is lower than the instance specification (100 MB/sec).
Why is the peak production/consumption traffic higher than the instance specification?
Suppose again the bandwidth specification of instance A is 100 MB/sec, then the traffic throttling threshold of each 100-ms time bucket is 10 MB. If the production traffic of instance A reaches 70 MB in the first 100-ms time bucket of a certain second (7 times the threshold), then the broker's traffic throttling policy will be triggered to increase the delayed response time. Suppose the original normal TCP response time is 100 ms, then the delay may be increased by 800 ms before response after the threshold is exceeded. After the response is returned at the 900th ms, the client immediately injects 70 MB of traffic into the 10th time bucket. The final traffic in this second is (70 MB 1 + 0 MB 8 + 70 MB * 1) = 140 MB, so the traffic speed in this second (140 MB/sec) is higher than the instance specification (100 MB/sec).
Why does the number of traffic throttling events surge?
The number of traffic throttling events is counted based on TCP requests. If instance A exceeds the traffic threshold in the first time bucket in a certain second, all TCP requests in the remaining time of this time bucket after the threshold is exceeded will be throttled and counted as traffic throttling events.
How does CKafka throttle traffic?
To ensure service stability, CKafka implement network traffic control strategies on both input and output messages.
Throttling occurs when the total traffic of a user’s replicas exceeds the purchased peak traffic.
When the producer traffic is throttled, CKafka will extend the response time of a TCP connection. The delay period depends on how much the instantaneous traffic exceeds the limit. It is similar to the principle of road traffic control. The more traffic flow, the higher the delay value from the delay algorithm, up to 5 minutes.
When the consumer traffic is throttled, CKafka will reduce the size of each fetch.request.max.bytes request to control the traffic.
How do I determine whether CKafka has been throttled?
1. In the instance list, you can see the health status of each cluster. If it’s “Warning”, you can hover your mouse over it to view the detailed data. The data displays your peak traffic and the number of throttling occurrences, based on which you can determine whether this instance has been throttled.
2. Users can open the monitoring data to view the maximum traffic value. If (maximum traffic value × number of replicas) > peak bandwidth at purchase, throttling has occurred at least once. Configure traffic throttling alarms to check whether throttling occurs. For alarm rules and configuration steps, refer to Configure Alarm Policy.
View instance monitoring on the CKafka console's monitoring page. If the throttling count is above 0, it indicates throttling has occurred. For specific steps, refer to View Monitoring Data.
