What Is a Connector
The TDMQ for CKafka Connector is a SaaS-based data access and processing tool on Tencent Cloud, providing a one-stop solution for efficient data access, processing, and distribution. It primarily offers two data transfer plugins: Kafka to Kafka and Kafka to Elasticsearch Service (ES), with built-in configurable data cleaning (ETL) capabilities to help users build stable and reliable data transfer linkages at low cost.
Introduction to the Kafka to Kafka Plugin
Scenarios
The data is already stored in Kafka of the cache layer and needs to undergo ETL processing for cleansing and formatting before being stored to a downstream Kafka instance.
Cross-cluster data replication (from a CKafka cluster to another CKafka cluster): real-time hot backup for production environments and disaster recovery clusters, and unified aggregation of dispersed business data.
Cross-cloud disaster recovery (from a self-built Kafka cluster/Kafka cluster from other cloud vendors to a CKafka cluster): disaster recovery for IDC environments and cloud environments, and dual-cloud or multi-cloud disaster recovery.
Feature Description
Based on the native Kafka protocol, the connector provides high-throughput, millisecond-level latency transmission capabilities and supports seamless data stream synchronization and intelligent routing rules (filtering by topic or field). It ensures data consistency through resumable transfers and automatic retry mechanisms, and is compatible with self-built or cross-cloud Kafka clusters.
Business Examples
Real-time distribution of e-commerce order data to different business departments (such as risk control, logistics, and data analysis) for analysis.
Building multi-site active-active architectures for core businesses in the financial industry.
Introduction to the Kafka to ES Plugin
Scenarios
The CKafka connector supports transforming streaming data into searchable and analyzable structured storage, suitable for various scenarios such as real-time log monitoring, user behavior search, and IoT device status analysis. For example:
Business logs are synchronized to ES via Kafka, and Ops monitoring is visualized via Kibana.
User clickstream or order data is written to ES in real time. Quick searches and queries under complex conditions are supported.
Feature Description
The connector provides near-real-time writing (data visible within seconds), automatic index management (dynamically adapting to JSON structures), and data preprocessing capabilities (JSON parsing and field pruning), and effectively isolates abnormal data through a dead letter queue mechanism, ensuring transmission reliability with a retry policy.
Business Examples
User learning behavior analysis on online education platforms.
Quick localization and troubleshooting of abnormal sensor data in smart hardware scenarios.
Lifecycle of CKafka Connector Plugins
The CKafka connector currently provides commercially available plugins Kafka to Kafka and Kafka to ES.
Plugins within the commercialization plan:
|
Data output | | General Availability (GA) | None | None |
| | GA | None | None |
Data access | | Open beta | None | None |
Plugins planned to be taken offline after the open beta:
|
Data output | CTSDB | EOM | October 31, 2024 | Estimated date: October 31, 2025 |
| TDSQL-C PostgreSQL | EOM | October 31, 2024 | Estimated date: October 31, 2025 |
| Data warehouse ClickHouse | EOM | October 31, 2024 | Estimated date: June 30, 2026 |
| Analytical database Doris | EOM | October 31, 2024 | Estimated date: June 30, 2026 |
| Database MySQL | EOM | October 31, 2024 | Estimated date: June 30, 2026 |
| PostgreSQL | EOM | October 31, 2024 | Estimated date: June 30, 2026 |
| Cloud data warehouse PostgreSQL | EOM | October 31, 2024 | Estimated date: June 30, 2026 |
| TDSQL for PostgreSQL | EOM | October 31, 2024 | Estimated date: June 30, 2026 |
| Distributed data warehouse TDW | EOM | October 31, 2024 | Estimated date: June 30, 2026 |
| Cloud Object Storage (COS) | EOM | October 31, 2024 | Estimated date: June 30, 2026 |
| Cloud Log Service | EOM | October 31, 2024 | Estimated date: June 30, 2026 |
| Serverless Cloud Function (SCF) | EOM | October 31, 2024 | Estimated date: June 30, 2026 |
| EventBridge | EOM | October 31, 2024 | Estimated date: June 30, 2026 |
Data access | MariaDB data subscription | EOM | October 31, 2024 | Estimated date: October 31, 2025 |
| TDSQL-C PostgreSQL data subscription | EOM | October 31, 2024 | Estimated date: October 31, 2025 |
| MongoDB data subscription | EOM | October 31, 2024 | Estimated date: June 30, 2026 |
| MySQL data subscription | EOM | October 31, 2024 | Estimated date: June 30, 2026 |
| MySQL data access | EOM | October 31, 2024 | Estimated date: June 30, 2026 |
| PostgreSQL data subscription | EOM | October 31, 2024 | Estimated date: June 30, 2026 |
| SQL Server data subscription | EOM | October 31, 2024 | Estimated date: June 30, 2026 |
| TDSQL-C for MySQL data subscription | EOM | October 31, 2024 | Estimated date: June 30, 2026 |
| Data reporting HTTP | EOM | October 31, 2024 | Estimated date: June 30, 2026 |
| Data Transfer Service (DTS) | EOM | October 31, 2024 | Estimated date: June 30, 2026 |
Note:
EOM: indicates that a version is no longer offered for sale.
EOS: indicates that services are no longer provided for the version. After this date, no technical service support will be provided for this software version.