Creating Shipping Task

Last updated: 2025-07-03 20:10:49

Creating Shipping Task

Last updated: 2025-07-03 20:10:49

Application Scenario
You can ship log topic data to CKafka for real-time stream computing and storage. If you haven't purchased a CKafka instance, you can consider using the Consumption over Kafka feature of CLS.
Prerequisites
You have activated the CKafka service.
Note:
If you enable ACL for CKafka, please note the version limitations: CKafka 1.X requires version 1.1.1 or higher; CKafka 2.X requires version 2.4.2 or higher. Enabling ACL on other versions will cause delivery failure. You can disable ACL to resolve the issue.
Currently only support real-time log delivery. Historical logs are not supported.
Delivery to CKafka elastic Topics is not supported.
CKafka support routing access method requires configuration as PLAINTEXT, otherwise delivery will fail.
Billing Description
Log shipping incurs private network read traffic fees (cross-region shipping is not supported for now), which CLS charges based on the size after Snappy or LZ4 compression. If your raw log is 100 GB and you choose Snappy for compression, around 50 GB will be billable. As the read traffic price is 0.032 USD/GB, the fees will be 50 GB * 0.032 USD/GB = 1.6 USD.
Directions
1. Prepare and configure a CKafka Topic.
1.1 Create a CKafka instance in the same region as the log topic. For more information, see Creating Instance.
1.2 Create a topic. For details, see Creating Topic. You need to make the following changes to the topic configuration. Other parameters can be configured as needed.
Preset ACL Policy: Disable this option.
cleanup.policy: Select delete; otherwise, shipping will fail.
max.message.bytes: Set this value to 12 MB or above. Otherwise, when the size of a single message in CLS exceeds the specified limit, the message cannot be written to the CKafka topic, and shipping will fail.
2. Go to the CLS console. There are two ways to enter the delivery task management.
Entry 1: In the left navigation bar, click Shipping Job Management, select the region, log set, and log topic, and then select the Ship to CKafka tab.
Entry 2: In the log topic list, select and enter the log topic that needs to deliver to CKafka. On the log topic details page, select the Ship to CKafka tab on the left side.
3. Click Edit on the right, enable the switch of delivering to CKafka, select the appropriate CKafka instance and corresponding Topic, then start shipping. You can choose to ship logs in original content or JSON format.
Deliver original content to CKafka. Configuration items are described:
Configuration Item
Description
Rule
Required
Target CKafka Topic Ownership
Current root account
Another root account
Deliver CLS logs to the current root account's CKafka Topic.
Deliver CLS logs to another root account's CKafka. For example, if account A ships logs to account B's CKafka Topic via CLS, account B must configure an access role in Cloud Access Management (CAM). After configuration, account A needs to enter the Role ARN and external ID in the CLS console to enable cross-account delivery. The steps to configure the role are as follows:
1. Create new role. Account B logged in to the CAM role management page.
1.1 Create an access policy, with a policy name such as cross_shipper. For policy syntax, see the following:
Note:
Note: The authorization in the example follows the minimum permission principle, with the resource configured as shipping to only the CKafka instance (ckafka-12abcde3) in the Guangzhou region. Please authorize according to the actual situation.
  {
    "statement": [
        {
            "action": [
                "cam:GetRole"
            ],
            "effect": "allow",
            "resource": [
                "*"
            ]
        },
        {
            "action": [
                "ckafka:ListRoute",
                "ckafka:AddRoute",
                "ckafka:DescribeInstanceAttributes",
                "ckafka:AuthorizeToken",
                "ckafka:CreateToken",
                "ckafka:DescribeTopicAttributes"
            ],
            "effect": "allow",
            "resource": [
                "qcs::ckafka:ap-guangzhou:uin/100001234567:ckafkaId/ckafka-12abcde3"
            ]//resource configured as shipping to CKafka: ckafka-12abcde3 in the Guangzhou region only, please authorize according to the actual situation
        }
    ],
    "version": "2.0"
}
1.2 Create a new role, select Tencent Cloud account as the role carrier, choose other root account for the cloud account type, then input Account A's ID, such as 100012345678, check enable verification and configure the external ID, for example: Hello123.
1.3 Configure role policy, configure access policy for the role, and select the pre-configured access policy cross_shipper (example).
1.4 Save the role, for example: uinA_writeCLS_to_CKafka.
2. Configure the carrier for the role. In the CAM role list, find uinA_writeCLS_to_CKafka (example), click the role, select role carrier > management carrier > add product service > CLS, then click refresh.
The current role's carriers are two: account A and cls.cloud.tencent.com (CLS log service).
3. Account A logs in to CLS and fills in Role ARN and external ID.
The two items of info need to be provided by account B:
Account B finds the role uinA_writeCLS_to_CKafka (example) in the CAM role list, clicks to view the RoleArn of the role, such as qcs::cam::uin/100001112345:roleName/uinA_writeCLS_to_CKafka.
The external ID, such as Hello123, is visible in the role carrier.
Note:
Fill in the Role ARN and external ID. Note: Do not enter extra spaces, as this will cause permission verification to fail.
Cross-account delivery will generate read traffic fees for the log topic under Account A.
current root account other root account
No
CKafka instance
The CKafka Topic in the same region as the current log topic is used as the delivery target.
In the cross-account delivery scenario, the user manually fills in the CKafka instance ID and Topic name.
List selection
Required
Format of Data to Ship
Select Original content to deliver the user's raw logs.
List selection
Required
Data compression format
no compression\\SNAPPY\\LZ4.
List selection
Required
Shipping log preview
Preview your delivered log data.
-
-
Deliver in JSON format to CKafka. Configuration items are described:
Configuration Item
Description
Rule
Required
Target CKafka Topic Ownership
Current Root Account
Other Root Account
Deliver CLS logs to the current root account's CKafka Topic.
Ship CLS logs to another root account's CKafka. For example, if account A ships logs from CLS to account B's CKafka Topic, account B must configure a role in CAM (Access Management). After configuration, account A needs to enter the Role ARN and external ID in the CLS console to enable cross-account delivery. The steps to configure the role are as follows:
1. Create role. Account B logged in to the CAM role management page.
1.1 Create an access policy, with a policy name such as cross_shipper. For policy syntax, see the following:
Note:
Note: The authorization in the example follows the minimum permission principle, with the resource configured as shipping to only the CKafka instance (ckafka-12abcde3) in the Guangzhou region. Please authorize according to the actual situation.
  {
    "statement": [
        {
            "action": [
                "cam:GetRole"
            ],
            "effect": "allow",
            "resource": [
                "*"
            ]
        },
        {
            "action": [
                "ckafka:ListRoute",
                "ckafka:AddRoute",
                "ckafka:DescribeInstanceAttributes",
                "ckafka:AuthorizeToken",
                "ckafka:CreateToken",
                "ckafka:DescribeTopicAttributes"
            ],
            "effect": "allow",
            "resource": [
                "qcs::ckafka:ap-guangzhou:uin/100001234567:ckafkaId/ckafka-12abcde3"
            ]//resource configured as shipping to CKafka: ckafka-12abcde3 in the Guangzhou region only, please authorize according to the actual situation
        }
    ],
    "version": "2.0"
}
1.2 Create a new role, select Tencent Cloud account as the role carrier, choose other root account for the cloud account type, then input Account A's ID, such as 100012345678, check enable verification and configure the external ID, for example: Hello123.
1.3 Configure role policy, configure access policy for the role, and select the pre-configured access policy cross_shipper (example).
1.4 Save the role, for example: uinA_writeCLS_to_CKafka.
2. Configure a carrier for the role. Find uinA_writeCLS_to_CKafka (example) in the CAM role list, click the role, select role carrier > Entity management > add product service > CLS, then click update.
The current role's carriers are two: account A and cls.cloud.tencent.com (CLS log service).
3. Account A logs in to CLS and fills in Role ARN and external ID.
The two items of info need to be provided by account B:
Account B finds the role uinA_writeCLS_to_CKafka (example) in the CAM role list, clicks to view the RoleArn of the role, such as qcs::cam::uin/100001112345:roleName/uinA_writeCLS_to_CKafka.
The external ID, such as Hello123, is visible in the role carrier.
Note:
Fill in the Role ARN and external ID. Note: Do not enter extra spaces, as this will cause permission verification to fail.
Note: Cross-account delivery will generate read traffic fees for the log topic under Account A.
current root account other root account
No
CKafka instance
The CKafka Topic in the same region as the current log topic is used as the delivery target. 
List selection
Required
Format of Data to Ship
Option JSON, deliver logs in JSON format.
List selection
Required
Escape/Do not escape in JSON format
Escape: Convert the value of the first-level nodes in the JSON to String. If the value of your first-level nodes is Struct and you need to convert the Struct to String in downstream storage or computation, you can select this option. Examples:
Original log: {"a":"aa", "b":{"b1":"b1b1", "c1":"c1c1"}}
Deliver to CKafka: {"a":"aa","b":"{\\"b1\\":\\"b1b1\\", \\"c1\\":\\"c1c1\\"}"} 
Non-escaping, do not modify your JSON structure and hierarchy, keep the log format consistent with that on the collection side. Example:
Original log: {"a":"aa", "b":{"b1":"b1b1", "c1":"c1c1"}}
Deliver to CKafka: {"a":"aa", "b":{"b1":"b1b1", "c1":"c1c1"}}
Note:
When the first-layer node of JSON contains a numeric value, it will automatically convert to int or float after delivery.
Original log: {"a":123, "b":"123", "c":"-123", "d":"123.45", "e":{"e1":123,"f1":"123"}}
Deliver to CKafka: {"a":123,"b":123,"c":-123,"d":123.45,"e":{"e1":123,"f1":"123"}}
List selection
Required
Log Fields to Ship
Flatten or not flatten the __TAG__ metadata based on your actual business scenario.
__TAG__ meta information: {"__TAG__":{"fieldA":200,"fieldB":"text"}}
Flatten: {"__TAG__.fieldA":200,"__TAG__.fieldB":"text"}
Not Tiled: {"__TAG__":{"fieldA":200, "fieldB":"text"}}
﻿
﻿
Data compression format
no compression\\SNAPPY\\LZ4.
List selection
Required
Shipping log preview
Preview your delivered log data.
-
-
4. Click OK to start shipping to CKafka. 
FAQs
What should I do if the log data cannot be shipped to CKafka?
If ACL authentication is enabled in CKafka, the log data cannot be shipped. In this case, you need to disable the ACL of the topic.
What should I do if the system prompts that I have no permissions to read/write the CKafka topic?
If you directly use an API to ship data to CKafka, you may not have the read/write permissions of the CKafka topic. If you ship data in the console, the system will guide you through the authorization process, but if you directly call an API for shipping, you need to authorize manually. For more information, see CLS Service Role Authorization.

Was this page helpful?

You can also Contact Sales or Submit a Ticket for help.

Yes

Feedback

Configuration Item	Description	Rule	Required
Target CKafka Topic Ownership	Current root account Another root account Deliver CLS logs to the current root account's CKafka Topic. Deliver CLS logs to another root account's CKafka. For example, if account A ships logs to account B's CKafka Topic via CLS, account B must configure an access role in Cloud Access Management (CAM). After configuration, account A needs to enter the Role ARN and external ID in the CLS console to enable cross-account delivery. The steps to configure the role are as follows: 1. Create new role. Account B logged in to the CAM role management page. 1.1 Create an access policy, with a policy name such as cross_shipper. For policy syntax, see the following: Note: Note: The authorization in the example follows the minimum permission principle, with the resource configured as shipping to only the CKafka instance (ckafka-12abcde3) in the Guangzhou region. Please authorize according to the actual situation. { "statement": [ { "action": [ "cam:GetRole" ], "effect": "allow", "resource": [ "" ] }, { "action": [ "ckafka:ListRoute", "ckafka:AddRoute", "ckafka:DescribeInstanceAttributes", "ckafka:AuthorizeToken", "ckafka:CreateToken", "ckafka:DescribeTopicAttributes" ], "effect": "allow", "resource": [ "qcs::ckafka:ap-guangzhou:uin/100001234567:ckafkaId/ckafka-12abcde3" ]//resource configured as shipping to CKafka: ckafka-12abcde3 in the Guangzhou region only, please authorize according to the actual situation } ], "version": "2.0" } 1.2 Create a new role, select Tencent Cloud account* as the role carrier, choose other root account for the cloud account type, then input Account A's ID, such as 100012345678, check enable verification and configure the external ID, for example: Hello123. 1.3 Configure role policy, configure access policy for the role, and select the pre-configured access policy cross_shipper (example). 1.4 Save the role, for example: uinA_writeCLS_to_CKafka. 2. Configure the carrier for the role. In the CAM role list, find uinA_writeCLS_to_CKafka (example), click the role, select role carrier > management carrier > add product service > CLS, then click refresh. The current role's carriers are two: account A and cls.cloud.tencent.com (CLS log service). 3. Account A logs in to CLS and fills in Role ARN and external ID. The two items of info need to be provided by account B: Account B finds the role uinA_writeCLS_to_CKafka (example) in the CAM role list, clicks to view the RoleArn of the role, such as qcs::cam::uin/100001112345:roleName/uinA_writeCLS_to_CKafka. The external ID, such as Hello123, is visible in the role carrier. Note: Fill in the Role ARN and external ID. Note: Do not enter extra spaces, as this will cause permission verification to fail. Cross-account delivery will generate read traffic fees for the log topic under Account A.	current root account other root account	No
CKafka instance	The CKafka Topic in the same region as the current log topic is used as the delivery target. In the cross-account delivery scenario, the user manually fills in the CKafka instance ID and Topic name.	List selection	Required
Format of Data to Ship	Select Original content to deliver the user's raw logs.	List selection	Required
Data compression format	no compression\\SNAPPY\\LZ4.	List selection	Required
Shipping log preview	Preview your delivered log data.	-	-

tencent cloud

Application Scenario

Prerequisites

Billing Description

Directions

FAQs

What should I do if the log data cannot be shipped to CKafka?

What should I do if the system prompts that I have no permissions to read/write the CKafka topic?