Tencent Cloud Kafka (CKafka) is a distributed, high-throughput and highly scalable messaging system. It is fully compatible with the open-source Apache Kafka API v0.9 and v0.10. Based on the publish/subscribe mode, CKafka enables async interaction between message producers and consumers by decoupling messages and thus eliminating wait time. CKafka supports data compression as well as offline and real-time data processing, making it ideal for collecting compressed logs and aggregating monitoring data.
Message Queue CKafka
Distributed high-throughput messaging system for log collection and data aggregation
Overview
Benefits
Open-Source Component Compatibility
Work well with open-source kafka components like Kafka Streams, Kafka Connect, and KSQL, thus reducing the cloudification costs.
Upstream and Downstream Ecosystems
Connect to over 13 Tencent Cloud products, including EMR, COS, CIS, SCS, SCF, and CLS, for quick one-click deployment.
High Reliability
Surpass the productivity of open-source solutions and provide a distributed deployment to ensure cluster stability.
High Scalability
Support automatic horizontal scaling of clusters and seamless upgrade of instances without affecting the user experience.
Business Security
Isolate tenants at the network level among different accounts, and support CAM for management streams and SASL for data streams to enhance security.
Unified OPS Monitoring
Provide a comprehensive set of Ops services, including multidimensional monitoring and alarm services such as tenant isolation, access control, message retention query, and consumer details query.
Features
Message Decoupling
CKafka decouples message producers and consumers, allowing you to scale or modify the production-consumption processing procedure independently as long as the same API constraints are followed. In this sense, CKafka can replace traditional messaging middleware because it has higher throughput, a stronger partition replication mechanism, and better fault tolerance.
Peak Shifting
It is critical that the system be able to respond to access surges. However, if traffic surges are infrequent, resources invested based on peak traffic will be wasted. CKafka can help your system handle sudden access surges and reduce the risk of system crash caused by request overload.
Horizontal Scaling
Message queuing and processing can become more efficient when the decoupled message processing procedure is horizontally scaled. For example, a CKafka topic can be partitioned and distributed to one or more brokers. A consumer can subscribe to one or more partitions, while the producer evenly distributes messages to corresponding partitions. In this sense, adding more brokers means having a higher cluster throughput.
One-Time Production for Repeated Consumption
CKafka supports queue and publish/subscribe modes. A CKafka topic can be partitioned to reside in different brokers for higher throughput. CKafka also offers a multi-queue mode that uses the consumer group policy. Specifically, one topic stores only one copy of data at a node, and different consumer groups keep their own consumption records. This is ideal for scenarios where the messages are produced once but consumed by multiple consumer groups.
Scenarios
Log Analysis System
Streaming Data Processing Platform
CKafka can work with EMR to create a complete log analysis system. The client-side agent collects logs and aggregates them to CKafka, where the data is computed and consumed repeatedly by the backend big data suite, such as Spark. The original logs are then cleaned, stored, or graphically displayed.
CKafka can work with Stream Compute Service (SCS) to process data in a real-time or offline manner and detect exceptions in various scenarios:
- Analyze real-time data for exception detection to troubleshoot system issues.
- Store historical consumption data and analyze it offline for secondary processing and trend report generation.
Pricing
CKafka supports two billing modes: pay-as-you-go and monthly subscription. For more information, see Billing Overview.