tencent cloud

TDMQ for CKafka

Release Notes and Announcements
Release Notes
Broker Release Notes
Announcement
Product Introduction
Introduction and Selection of the TDMQ Product Series
What Is TDMQ for CKafka
Strengths
Scenarios
Technology Architecture
Product Series Introduction
Apache Kafka Version Support Description
Comparison with Apache Kafka
High Availability
Use Limits
Regions and AZs
Related Cloud Services
Billing
Billing Overview
Pricing
Billing Example
Changing from Postpaid by Hour to Monthly Subscription
Renewal
Viewing Consumption Details
Overdue Payments
Refund
Getting Started
Guide for Getting Started
Preparations
VPC Network Access
Public Domain Name Access
User Guide
Usage Process Guide
Configuring Account Permission
Creating Instance
Configuring Topic
Connecting Instance
Managing Messages
Managing Consumer Group
Managing Instance
Changing Instance Specification
Configuring Traffic Throttling
Configuring Elastic Scaling Policy
Configuring Advanced Features
Viewing Monitoring Data and Configuring Alarm Rules
Synchronizing Data Using CKafka Connector
Use Cases
Cluster Resource Assessment
Client Practical Tutorial
Log Integration
Open-Source Ecosystem Integration
Replacing Supporting Route (Old)
Migration Guide
Migration Solution Overview
Migrating Cluster Using Open-Source Tool
Troubleshooting
Topics
Clients
Messages
​​API Reference
History
Introduction
API Category
Making API Requests
Other APIs
ACL APIs
Instance APIs
Routing APIs
DataHub APIs
Topic APIs
Data Types
Error Codes
SDK Reference
SDK Overview
Java SDK
Python SDK
Go SDK
PHP SDK
C++ SDK
Node.js SDK
SDK for Connector
Security and Compliance
Permission Management
Network Security
Deletion Protection
Event Record
CloudAudit
FAQs
Instances
Topics
Consumer Groups
Client-Related
Network-Related
Monitoring
Messages
Agreements
CKafka Service Level Agreements
Contact Us
Glossary

Sarama Client Exceptions

PDF
포커스 모드
폰트 크기
마지막 업데이트 시간: 2026-01-20 17:19:22

Overview

Sarama is a Kafka client written in Golang, with high message throughput performance.
When performance reaches a bottleneck and TDMQ for CKafka (CKafka) partitions are actively scaled out, the Sarama client may fail to detect the partition rebalance, causing the failure to produce and consume the information of new partitions normally.

Possible Causes

After CKafka performs partition rebalancing due to various causes, Sarama requires approximately a 10-minute interval to detect the partition change, fetch the topic's metadata, and parse the latest partition data. Due to the considerable time taken for fetching, this process is sometimes perceived by users as a fetch failure.
Note:
By default, sarama.Metadata.RefreshFrequency is set to 10 minutes. Therefore, it takes Sarama 20 minutes to detect partition updates in the worst-case scenario. To accelerate partition update detection, decrease this configuration value. For example, setting it to 10 seconds ensures detection within 20 seconds.
In addition, during the long-term maintenance and usage by the CKafka team, it has also been observed that occasionally the Sarama client fails to fetch the latest partition information, resulting in message backlogs and randomly thrown exceptions.
This issue has been raised multiple times in the Sarama community and has received attention and fixes. In the latest version, the frequency of errors has decreased, but the issue still persists.

Solutions

If users are sensitive to the rebalance and use the Golang technology stack, it is recommended to migrate to using the Confluent-Kafka-go client as soon as possible.

Comparison of Common Golang Clients

Golang Client
Advantage
Limitation
Sarama
Highly active in the community. Sarama has a large user base, and questions raised in the community receive prompt responses and fixes.
High performance. Sarama is written in native Golang and provides robust support for asynchronous operations and high concurrency.
Moderate stability. Sarama may encounter unknown errors after scaling out and rebalancing partitions.
Confluent-Kafka-go (recommended)
High stability. Since the client is essentially a wrapper around the librdkafka C++ library, and the librdkafka library has been running in multiple languages for years, it delivers sufficient reliability.
High performance. Since the underlying implementation uses C++, it requires fewer resources and delivers a fast computational speed.
Increased compilation complexity. Due to the import of C++ libraries, the Golang compiler requires additional compilation configurations, which increases compilation dependencies and compilation complexity.
Additionally, due to differences in underlying implementations of C++ libraries across compiling environments, introducing the librdkafka library may impact cross-compilation for Golang projects.
kafka-go
Complete APIs. kafka-go provides not only high-level APIs but also exposes underlying ones, enabling users to configure and use kafka clients with greater flexibility.
Easy to use. kafka-go requires minimal code for basic production and consumption operations, with extensive default configurations provided.
Compared with the two clients above, it exhibits inferior performance and may not meet large-scale concurrent demands.


도움말 및 지원

문제 해결에 도움이 되었나요?

피드백