Creating Shipping Task

Last updated:2025-12-03 18:30:48

Creating Shipping Task

Last updated: 2025-12-03 18:30:48

Application Scenario
After logs are collected to CLS, the log data can be delivered to COS for storage or analysis. This section describes how to create a log shipping task to COS.
Prerequisite
1. You have activated CLS, created a logset and a log topic, and successfully collected the log data.
2. You have activated COS and created a bucket in the target region for log topic shipping. For more information, see Creating Bucket.
3. Sub-accounts and collaborators need to be authorized by the root account. For more information on granting permissions, see CAM Access Management. For authorization policy, see CLS Access Policy Template.
4. You have authorized the CLS service role to access COS. If you perform operations in the console, the system will guide you through the authorization process. If you directly call APIs, manual authorization will be required. For more information, see CLS Service Role Authorization.
Directions
1. Log in to the CLS console.
2. Click Log Topic on the left sidebar.
3. Click the desired log topic ID/name to go to the log topic management page.
4. Click the Ship to COS tab to enter the shipping to COS configuration page, then click Add Shipping Configuration.
5. If you need to archive logs, you can use the log settlement feature of CLS to transition cold data to infrequent storage. After settlement, storage fees can drop significantly while logs remain retrievable. For example, you can set the total storage time to 180 days, with 30 days for standard storage (hot data) and 150 days for infrequent storage (cold data), meeting audit requirements. If you still choose to deliver to COS, refer to the following configuration:
Configuration Item
Description
Rule
Required
Shipping Task Name
Name of the delivery task.
/
Required
Time range
Start time: The start time of the log data you want to ship. The default provides the earliest time point in the log topic lifecycle.
End time: The end time of the log data you want to ship. Future time cannot be selected. Not specified means continuous shipping of logs.
Note:
If your start time is historical time and end time is not specified, the task will continuously ship both historical logs and real-time logs.
For example: If you choose to deliver data from 00:03 on January 1, 2023 to ∞, and submit the delivery task at 19:05 on February 13, 2023, then the historical logs will be from 00:03 on January 1, 2023 to 19:05 on February 13, 2023, and the real-time logs will be from 19:05 on February 13, 2023 to ∞. The two kinds of data will be delivered to COS simultaneously. You may view the shipping progress and required duration of historical data delivery in the delivery task list.
﻿After task submission, the delivery time range cannot be modified
.
Select Time
Required
File Size
The size of the raw log file to be delivered works in conjunction with the delivery interval time. Whichever condition is met first will trigger the rule to compress the file, and then deliver it to COS.
For example, if you configure 256MB and 15 minutes, and the file size reaches 256MB in 5 minutes, then the file size condition will trigger the delivery task first.
5 - 256, unit: MB.
Required
Shipping Interval
Specify the interval to trigger a delivery. This works with the delivery file size. Whichever condition is met first will trigger the rule to compress the file and then deliver it to COS.
For example, if you configure 256MB and 15 minutes, and the file size is only 200MB after 15 minutes, then the interval time condition will trigger the delivery task first.
5 - 15 minutes
Required
6. Click Next to enter the target bucket
 configuration.
Configuration Item
Description
Rule
Required
Target COS Bucket Ownership
Current Root Account
Other Root Account
Deliver CLS logs to the current root account's COS bucket.
Deliver CLS logs to another root account's COS bucket. For example, to deliver CLS logs from account A to account B's COS bucket, account B must configure an access role in Cloud Access Management (CAM). After configuration, account A needs to enter the Role ARN and external ID in the CLS console to enable cross-account delivery. The steps to configure the role are as follows:
1. Create role. Account B is logged in to the CAM role management page.
1.1 Create an access policy, with a policy name such as cross_shipper. See the following for policy syntax:
Note:
Note: The authorization in the example follows the minimum permission principle, with the resource configured as shipping to only the COS bucket test123-123456789 in the Guangzhou region. Please authorize according to the actual situation.
{
    "statement": [
        {
            "action": [
                "cam:GetRole"
            ],
            "effect": "allow",
            "resource": [
                "*"
            ]
        },
        
        {
            "action": [
                "cos:PutObject"
            ],
            "effect": "allow",
            "resource": [
                "qcs::cos:ap-guangzhou:uid/123456789:test123-123456789/*"
            ]//Shipping to COS bucket test123-123456789 in Guangzhou region, please authorize according to the actual situation.
        }
    ],
    "version": "2.0"
}
1.2 Create a new role, select Tencent Cloud account as the role carrier, choose other root account for the cloud account type, then input the ID of Account A, such as 100012345678, check enable verification and configure the external ID, for example Hello123.
1.3 Configure the access policy for the role, select the pre-configured access policy cross_shipper (example).
1.4 Save the role, for example: uinA_writeCLS_to_COS.
2. Configure the carrier for the role. In the CAM role list, find uinA_writeCLS_to_COS (example), click the role, select role carrier > management carrier > add product service, choose CLS, then click update.
See current role-based carriers: one is account A, the other is cls.cloud.tencent.com (CLS log service).
3. Log in to CLS with account A and fill in Role ARN and external ID.
The following two items need to be provided by account B:
Account B finds the role uinA_writeCLS_to_COS (example) in the CAM role list, clicks to view the RoleArn of the role, such as qcs::cam::uin/100001112345:roleName/uinA_writeCLS_to_COS.
The external ID, such as Hello123, can be viewed in the role carrier.
Note:
Enter Role ARN and external ID. Be careful not to include extra spaces, otherwise lead to permission verification failed.
Cross-account delivery will generate read traffic fees for log topics under account A.
current root account other root account
Optional
COS Bucket
The destination bucket for shipping logs. For cross-account delivery, the user must manually fill in the target bucket's name.
List selection
Required
File naming
Delivery time naming: default option, such as 202208251645_000_132612782.gz means delivery time_log topic partition_offset. Hive can load this file.
Random number naming: the old naming method, Hive may not recognize it. Hive does not recognize files starting with an underscore. You can add a custom prefix in the COS path configuration, such as /%Y%M%d/%H/Yourname.
/
Required
File
Compression
No compression/snappy/lzop/gzip
/
Required
COS path
The path for storing logs in a COS storage bucket. By default, logs are saved in the format /year/month/day/hour/, for example /2022/7/31/14/. Path configuration supports strftime syntax, such as:
The path generated by /%Y/%m/%d/ is /2022/7/31/.
The path generated by /%Y%M%d/%H/ is /20220731/14/.
Do not start with /
Optional
Storage Class
The log storage types in a COS bucket: standard storage, infrequent storage, intelligent tiering storage, archive storage, deep archive storage. For details, see storage type overview.
Note:
Cross-account delivery does not support selection of COS storage type.
List selection
Required
7. Click Next, enter advanced configuration, you can select Format of Data to deliver in JSON/CSV/Parquet.
Format of Data to Ship
Application Scenarios
﻿Deliver in CSV format﻿
Applicable to Tencent Cloud DLC data ingestion calculation
Use CSV delivery to implement original log text (single-line, multi-line, logs collected by delimiter).
﻿Deliver in JSON format﻿
For common data formats, see your business scenarios for selection.
﻿Deliver in Parquet format﻿
Log data must be structured data, supporting data type conversion (not single-line or multi-line collection), and is mostly used for Hive. 
Delivery in JSON Format 
1. Select the log content (field) to be delivered. _CONTENT_ is the user's log, and _SOURCE_, _FILENAME_, _HOSTNAME_, _TIMESTAMP_, and _TAG_ are metadata fields of CLS. Select based on your conditions.
2. ﻿Escape option:
﻿
Escape Option
Description
Do not escape
Make no changes to your JSON structure and hierarchy, and keep the log format consistent with that on the collection side.
Example:
Original log text: {"a":"aa", "b":{"b1":"b1b1", "c1":"c1c1"}}
Deliver to COS: {"a":"aa", "b":{"b1":"b1b1", "c1":"c1c1"}}
Note:
When the first-layer node in JSON contains a numeric value, it will automatically convert to int or float after delivery.
Original Log Text: {"a":123, "b":"123", "c":"-123", "d":"123.45", "e":{"e1":123,"f1":"123"}}
Deliver to COS: {"a":123,"b":123,"c":-123,"d":123.45,"e":{"e1":123,"f1":"123"}}
Escape
Convert the value of the first-layer JSON node to String. If your node value is Struct, you need to convert it into String in advance during downstream storage or calculation. You can select this option.
Example 1:
Original log text: {"a":"aa", "b":{"b1":"b1b1", "c1":"c1c1"}}
Deliver to COS: {"a":"aa","b":"{\\"b1\\":\\"b1b1\\", \\"c1\\":\\"c1c1\\"}"} 
Example 2:
Original Log Text: {"a":123, "b":"123", "c":"-123", "d":"123.45", "e":{"e1":123,"f1":"123"}}
Deliver to COS:{"a":"123","b":"123","c":"-123","d":"123.45","e":"{\\"e1\\":123,\\"f1\\":\\"123\\"}"}
Delivery in CSV Format 
Choose to deliver logs in CSV format to COS.
Note:
Unneeded CLS metadata fields can be deleted.
Configuration items are described as follows:
Configuration Item
Description
Rule
Required
Key
Specify the key-value (key) field to be written in CSV file (the filled key must be the structured key name or reserved field of the log, otherwise it will be regarded as an invalid key)
Only letters, digits, and _-./@ are supported.
Required
Separator
Separator between fields in the CSV file.
List selection
Required
Escape Character
If a delimiter character appears in the normal field, it needs to be wrapped with an escape character to prevent misidentification when reading data.
List selection
Required
Invalid Field Filling
If the configured key-value field (key) does not exist, it will be filled with invalid fields. 
List selection
Required
Key in First Line
Add a description of the field name to the first line of the CSV file, that is, write the key-value (key) into the first line of the CSV file. It is not written by default.
ON/OFF
Required
Delivery in Parquet Format 
Parquet files can be loaded by Hive and are commonly used for big data computing and analysis. The following is a detailed introduction on how to create a Parquet format log delivery task.
Note:
Note: Parquet files are mostly used for big data platforms. Since Parquet itself has a certain compression rate and uses file compression formats (snappy/lzop/gzip), the delivery file size should be configured larger. It is recommended to set it to not less than 200MB (approximately 50MB when delivered to COS).
Parquet format does not support previewing delivered data.
Note: If you need to use import COS data, deliver the logs to COS, then import them to a CLS log topic. Parquet format does not support importing to CLS log topics, while CSV and JSON formats are supported.
Configuration items are described as follows:
Configuration Item
Description
Required
Key name
Write key-value fields to the Header part of a Parquet file. If the key-value from logs automatically pulled by the system does not meet your requirements, you can add fields (up to 100). Field names only support letters, digits, and _-./@.
If a certain line of log lacks a defined Key, there will be no such Key in the Parquet file Body corresponding to that log. This will not affect your big data computing frameworks such as Spark and Flink.
Required
Data type
The field data type in Parquet file: String, Boolean, Int32, Int64, Float, Double
Required
Assignment information returned upon resolution failure
The default value when data parsing fails. For example, if a field value (String type) fails to parse, you can specify it as an empty String "", NULL, or a custom String. The same applies to other data types: boolean, integer, and floating-point.
Required
﻿

Was this page helpful?

You can also Contact Sales or Submit a Ticket for help.

Yes

Feedback

Configuration Item	Description	Rule	Required
Shipping Task Name	Name of the delivery task.	/	Required
Time range	Start time: The start time of the log data you want to ship. The default provides the earliest time point in the log topic lifecycle. End time: The end time of the log data you want to ship. Future time cannot be selected. Not specified means continuous shipping of logs. Note: If your start time is historical time and end time is not specified, the task will continuously ship both historical logs and real-time logs. For example: If you choose to deliver data from 00:03 on January 1, 2023 to ∞, and submit the delivery task at 19:05 on February 13, 2023, then the historical logs will be from 00:03 on January 1, 2023 to 19:05 on February 13, 2023, and the real-time logs will be from 19:05 on February 13, 2023 to ∞. The two kinds of data will be delivered to COS simultaneously. You may view the shipping progress and required duration of historical data delivery in the delivery task list. After task submission, the delivery time range cannot be modified .	Select Time	Required
File Size	The size of the raw log file to be delivered works in conjunction with the delivery interval time. Whichever condition is met first will trigger the rule to compress the file, and then deliver it to COS. For example, if you configure 256MB and 15 minutes, and the file size reaches 256MB in 5 minutes, then the file size condition will trigger the delivery task first.	5 - 256, unit: MB.	Required
Shipping Interval	Specify the interval to trigger a delivery. This works with the delivery file size. Whichever condition is met first will trigger the rule to compress the file and then deliver it to COS. For example, if you configure 256MB and 15 minutes, and the file size is only 200MB after 15 minutes, then the interval time condition will trigger the delivery task first.	5 - 15 minutes	Required

Format of Data to Ship	Application Scenarios
Deliver in CSV format	Applicable to Tencent Cloud DLC data ingestion calculation Use CSV delivery to implement original log text (single-line, multi-line, logs collected by delimiter).
Deliver in JSON format	For common data formats, see your business scenarios for selection.
Deliver in Parquet format	Log data must be structured data, supporting data type conversion (not single-line or multi-line collection), and is mostly used for Hive.

Escape Option	Description
Do not escape	Make no changes to your JSON structure and hierarchy, and keep the log format consistent with that on the collection side. Example: Original log text: {"a":"aa", "b":{"b1":"b1b1", "c1":"c1c1"}} Deliver to COS: {"a":"aa", "b":{"b1":"b1b1", "c1":"c1c1"}} Note: When the first-layer node in JSON contains a numeric value, it will automatically convert to int or float after delivery. Original Log Text: {"a":123, "b":"123", "c":"-123", "d":"123.45", "e":{"e1":123,"f1":"123"}} Deliver to COS: {"a":123,"b":123,"c":-123,"d":123.45,"e":{"e1":123,"f1":"123"}}
Escape	Convert the value of the first-layer JSON node to String. If your node value is Struct, you need to convert it into String in advance during downstream storage or calculation. You can select this option. Example 1: Original log text: {"a":"aa", "b":{"b1":"b1b1", "c1":"c1c1"}} Deliver to COS: {"a":"aa","b":"{\\"b1\\":\\"b1b1\\", \\"c1\\":\\"c1c1\\"}"} Example 2: Original Log Text: {"a":123, "b":"123", "c":"-123", "d":"123.45", "e":{"e1":123,"f1":"123"}} Deliver to COS:{"a":"123","b":"123","c":"-123","d":"123.45","e":"{\\"e1\\":123,\\"f1\\":\\"123\\"}"}

tencent cloud

Application Scenario

Prerequisite

Directions

Delivery in JSON Format

Delivery in CSV Format

Delivery in Parquet Format