Using Kafka Protocol to Consume Logs
Last updated: 2025-11-25 19:15:50
By using the consumption over Kafka feature, you can consume a log topic as a Kafka topic. This allows you to stream log data collected in CLS to downstream big data components or data warehouses.
Such as Flink, Flume, Logstash, Splunk, and Tencent Cloud Oceanus.
Prerequisites
Activated Cloud Log Service (CLS), created a logset and log topic, and log data has been successfully collected.
Ensure the current operation account has permission to enable Kafka protocol consumption. For permission problems, see CLS Access Policy Template.
Intranet Consumption and Public Network Consumption Description
Consumption over the private network: A private network domain name is used to consume logs, and the traffic price is 0.032 USD/GB. If your raw log is 100 GB and you choose Snappy for compression, around 50 GB will be billable, and the private network read traffic fees will be 50 GB * 0.032 USD/GB = 1.6 USD. In general, you can consume logs over the private network if your consumer and log topic are in the same VPC or region.
Consumption over the public network: A public network domain name is used to consume logs, and the traffic price is 0.141 USD/GB. If your raw log is 100 GB and you choose Snappy for compression, around 50 GB will be billable, and the public network read traffic fees will be 50 GB * 0.141 USD/GB = 7.05 USD. In general, you need to consume logs over the public network if your consumer and log topic are in different VPCs or regions.
Operation Steps
1. Log in to the CLS console, select Log Topic in the left sidebar.
2. On the Log Topic page, click the log topic ID/name that needs to consume using Kafka protocol to enter the log topic management page.
3. On the log topic management page, click the Consumption over Kafka tab.
4. Click on the right Edit, set the Status switch button to open status, edit the following configuration, then click OK.
Configuration Item | Description | Rule |
Consumption data format | JSON: Consume logs in JSON data format. Original content: Consume logs in their original format. | Select |
Data Range | Historical + latest: New version, can consume all data within the log topic lifecycle. Latest: Earlier version, can only consume the latest data. Note: Two log topics with different data ranges cannot use the same consumption group. For example: if Log Topic A is configured as Historical + latest and Topic B as latest, Log Topic A and B cannot use the same consumption group. | Select |
Consume log fields | Please select the log fields you need to consume. JSON format escape/non-escaping as follows: Escape, convert the value of the first-layer JSON node to String. If the value of your first-level nodes is Struct, you can choose this option to convert the Struct to String in advance during downstream storage or calculation. Non-escaping, do not modify your JSON structure and hierarchy, keep the log format consistent with that on the collection side. Note: When the first-layer node of JSON contains a numeric value, it will automatically convert to int or float after consumption. Log: {"a":123, "b":"123", "c":"-123", "d":"123.45", "e":{"e1":123,"f1":"123"}} Consumption: {"a":123,"b":123,"c":-123,"d":123.45,"e":{"e1":123,"f1":"123"}} Flatten or not flatten __TAG__ metadata as follows. __TAG__ meta information: {"__TAG__":{"fieldA":200,"fieldB":"text"}} tiling: {"__TAG__.fieldA":200,"__TAG__.fieldB":"text"} Not tiling: {"__TAG__":{"fieldA":200, "fieldB":"text"}} | Select |
Data compression format | Supports SNAPPY, LZ4, and No compression. | Select |
Extranet Consumption | After disabling, you cannot access consumption logs from the public network and can only consume over the private network. | Switch |
Consumption Log Preview | Preview your consumed log data. | - |
Service log | Related logs for consumption monitoring charts. Data is provided free by CLS. | Switch |
5. The console provides Topic and Host+Port information. You can copy this information to construct your consumer (KafkaConsumer). You can also use the Automatically Generate Consumers widget in the upper-right corner of the Basic info Tab to generate a runnable consumer client. If any other business logic is required, modify the code.
When a log topic has multiple partitions, it is recommended to start up multiple consumers to consume data and avoid backlog. You can view the partition count in log topic details.
Consumer Parameter Description
Parameter | Description |
User authentication mode | Currently, only SASL_PLAINTEXT is supported. |
hosts | Intranet consumption: kafkaconsumer-${region}.cls.tencentyun.com:9095 Public network consumption: kafkaconsumer-${region}.cls.tencentcs.com:9096. For details, see Log Consumption - Consumption Over Kafka Protocol. |
topic | Consumption topic ID, please copy it from the console for consumption over Kafka. Example: XXXXXX-633a268c-XXXX-4a4c-XXXX-7a9a1a7baXXXX. |
username | Configured as ${LogSetID}, the logset ID. Example: 0f8e4b82-8adb-47b1-XXXX-XXXXXXXXXX, please copy it from the console for consumption over Kafka. |
password | Configured as ${SecretId}#${SecretKey}. For example: XXXXXXXXXXXXXX#YYYYYYYY, please log in to Cloud Access Management and click Access Keys in the left sidebar. The API key or project key can be used.If your sub-account needs to use this feature, it is recommended to use a sub-account key. When authorizing the sub-account, configure both the action and resource in the access policy to the minimum permissible range. For more details, see Kafka Protocol Consumption Authorization. |
Consumer Demo
Consumer Group Management
Reset Consumption Point: When your choice of data scope is Historical + latest, you may select a consumption group on the Consumer Tab page and perform a restart operation for the consumption point. The following three consumption points are available:
Latest position: Consume from the latest data.
Earliest position: Consume from the earliest data.
Specified time point. Note this time point must be within the log topic lifecycle.
Note:
The consumption group state is Empty when the consumption point can be reset. After disabling the consumer client program, the consumption group status will be recognized as Empty by the server.
Monitoring detail: Click the monitoring detail to navigate to the monitoring information of this consumption group.
Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No
Feedback