Connector overview
Last updated: 2025-09-29 10:40:30
What Is a Connector
CKafka Connector is a SaaS-based data access and processing tool on Tencent Cloud, enabling all-in-one efficient data access, processing and distribution. It mainly provides two data flow plug-ins: Kafka to Kafka and Kafka to Elasticsearch Service, with built-in simple configurable data cleansing (ETL) capacity to help users build stable reliable data flow chains at low cost.
Introducing the Kafka to Kafka Plug-In
Use Cases
The data is already present in the caching layer's TDMQ Kafka and must be scrubbed, formatted, and processed via ETL before being stored in the downstream Kafka.
Cross-cluster data replication (Tencent Cloud CKafka cluster → Tencent Cloud CKafka cluster): real-time hot standby between production environment and disaster recovery cluster, unified aggregation of dispersed business data.
Multi-cloud disaster recovery (self-built Kafka cluster/other cloud vendors' Kafka clusters → Tencent Cloud CKafka cluster): disaster recovery between IDC environment and Cloud Environment, dual-cloud or multi-cloud disaster recovery scenario.
Feature Overview
Based on the native Kafka protocol, it provides high-throughput, millisecond-level latency transmission capacity, supports seamless data stream synchronization, and intelligent routing rules (by Topic, field content filtering). It ensures data consistency through checkpoint restart and automatic retry mechanism, compatible with self-built or cross-cloud Kafka clusters.
Business Example
E-commerce order data is distributed to different business departments (such as risk control, logistics, data analysis) for analysis.
Build an active-active architecture for core business in the financial industry.
Introducing the Kafka to ES Plug-In
Use Cases
The CKafka Connector supports streaming data transformation into searchable, analyzable structured storage, suitable for various scenarios such as real-time log monitoring, user behavior retrieval, or IoT device status analysis. For example:
Business logs are synchronized to Elasticsearch Service via Kafka and leverage Kibana for monitoring visualization.
Write user clickstream or order data to Elasticsearch Service in real time, supporting fast retrieval and queries with complex conditions.
Feature Overview
The connector provides near real-time writing (data visible within seconds), automated index management (dynamically adapting to JSON structure), and data preprocessing capacity (JSON parsing, field trimming). It effectively isolates abnormal data through DLQ mechanism, combined with retry policy to guarantee transmission reliability.
Business Example
Analysis of user learning behavior on online education platforms.
Quick location and troubleshooting of abnormal data from sensors in smart hardware scenarios.
CKafka Connector Plug-In Lifecycle Explanation
Message Queue CKafka currently offers commercial plug-ins: Kafka to Kafka and Kafka to Elasticsearch Service (ES).
Plug-ins in the commercialization plan:
Plug-In Type | Plug-In Name | Lifecycle Stage | EOM Time | EOS Time |
data export | GA | None | None | |
| Elasticsearch Service | GA | None | None |
Data Access | TDMQ MQTT | open beta | None | None |
Plug-ins to be decommissioned after open beta:
Plug-In Type | Plug-In Name | Lifecycle Stage | EOM Time | EOS Time |
data export | CTSDB | EOM | October 31, 2024 | expected 2025-10-31 |
| TDSQL-C PostgreSQL | EOM | October 31, 2024 | expected 2025-10-31 |
| Data Warehouse ClickHouse | EOM | October 31, 2024 | expected 2026-06-30 |
| Analytical Database Doris | EOM | October 31, 2024 | expected 2026-06-30 |
| TencentDB for MySQL | EOM | October 31, 2024 | expected 2026-06-30 |
| PostgreSQL | EOM | October 31, 2024 | expected 2026-06-30 |
| Cloud Data Warehouse PostgreSQL | EOM | October 31, 2024 | expected 2026-06-30 |
| TDSQL for PostgreSQL | EOM | October 31, 2024 | expected 2026-06-30 |
| TDW Distributed Data Warehouse | EOM | October 31, 2024 | expected 2026-06-30 |
| Cloud Object Storage (COS) | EOM | October 31, 2024 | expected 2026-06-30 |
| Cloud Log Service (CLS) | EOM | October 31, 2024 | expected 2026-06-30 |
| SCF (Serverless Cloud Function) | EOM | October 31, 2024 | expected 2026-06-30 |
| EventBridge | EOM | October 31, 2024 | expected 2026-06-30 |
Data Access | TencentDB for MariaDB data subscription | EOM | October 31, 2024 | expected 2025-10-31 |
| TDSQL-C for PostgreSQL data subscription | EOM | October 31, 2024 | expected 2025-10-31 |
| TencentDB for MongoDB data subscription | EOM | October 31, 2024 | expected 2026-06-30 |
| TencentDB for MySQL Data Subscription | EOM | October 31, 2024 | expected 2026-06-30 |
| MySQL Data Access | EOM | October 31, 2024 | expected 2026-06-30 |
| PostgreSQL data subscription | EOM | October 31, 2024 | expected 2026-06-30 |
| SQL Server Data Subscription | EOM | October 31, 2024 | expected 2026-06-30 |
| TDSQL-C for MySQL data subscription | EOM | October 31, 2024 | expected 2026-06-30 |
| Data reporting over HTTP | EOM | October 31, 2024 | expected 2026-06-30 |
| Data Transfer Service (DTS) | EOM | October 31, 2024 | expected 2026-06-30 |
Note:
EOM: End of Marketing. Sale of this version is disabled.
EOS: End of Service & Support. Service for this version is terminated. After this date, no technical support is provided for the installed software version.
Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No
Feedback