- Release Notes and Announcements
- User Guide
- Product Introduction
- Purchase Guide
- Getting Started
- Operation Guide
- Resource Management
- Permission Management
- Log Collection
- Collection Overview
- Collecting Logs in Self-Built Kubernetes Cluster
- Collecting Syslog
- Collection by LogListener
- Collecting Text Log
- Uploading Log over Kafka
- Uploading Logs via Anonymous Write
- Uploading Logs via Logback Appender
- Uploading Logs via Log4j Appender
- Uploading Log via SDK
- Uploading Log via API
- Importing Data
- Tencent Cloud Service Log Access
- Log Storage
- Search and Analysis
- Syntax and Rules
- Statistical Analysis (SQL)
- Quick Analysis
- SQL Syntax
- SQL Functions
- String Function
- Date and Time Functions
- IP Geographic Function
- URL Function
- Mathematical Calculation Functions
- Mathematical Statistical Function
- General Aggregate Function
- Geospatial Function
- Binary String Function
- Estimation Function
- Type Conversion Function
- Logical Function
- Operators
- Bitwise Operation
- Regular Expression Function
- Lambda Function
- Conditional Expressions
- Array Functions
- Interval-Valued Comparison and Periodicity-Valued Comparison Functions
- JSON Functions
- Window Functions
- Sampling Analysis
- Configuring Indexes
- Reindexing
- Context Search and Analysis
- Downloading Log
- Dashboard
- Data Processing documents
- Data Processing
- Creating Processing Task
- Viewing Data Processing Details
- Data Processing Functions
- Function Overview
- Key-Value Extraction Functions
- Enrichment Functions
- Flow Control
- Row Processing Functions
- Field Processing Functions
- Value Structuring Functions
- Regular Expression Processing Functions
- Time Value Processing Functions
- String Processing Functions
- Type Conversion Functions
- Logical and Mathematical Functions
- Encoding and Decoding Functions
- IP Parsing Functions
- Processing Cases
- Scheduled SQL Analysis
- SCF
- Data Processing
- Shipping and Consumption
- Monitoring Alarm
- Historical Documentation
- Best Practices
- Developer Guide
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- Topic Management APIs
- Log Set Management APIs
- Index APIs
- Topic Partition APIs
- Machine Group APIs
- Collection Configuration APIs
- Log APIs
- Metric APIs
- Alarm Policy APIs
- Data Processing APIs
- Kafka Protocol Consumption APIs
- CKafka Shipping Task APIs
- Kafka Data Subscription APIs
- COS Shipping Task APIs
- SCF Delivery Task APIs
- Scheduled SQL Analysis APIs
- COS Data Import Task APIs
- Data Types
- Error Codes
- FAQs
- CLS Service Level Agreement
- CLS Policy
- Contact Us
- Glossary
- Release Notes and Announcements
- User Guide
- Product Introduction
- Purchase Guide
- Getting Started
- Operation Guide
- Resource Management
- Permission Management
- Log Collection
- Collection Overview
- Collecting Logs in Self-Built Kubernetes Cluster
- Collecting Syslog
- Collection by LogListener
- Collecting Text Log
- Uploading Log over Kafka
- Uploading Logs via Anonymous Write
- Uploading Logs via Logback Appender
- Uploading Logs via Log4j Appender
- Uploading Log via SDK
- Uploading Log via API
- Importing Data
- Tencent Cloud Service Log Access
- Log Storage
- Search and Analysis
- Syntax and Rules
- Statistical Analysis (SQL)
- Quick Analysis
- SQL Syntax
- SQL Functions
- String Function
- Date and Time Functions
- IP Geographic Function
- URL Function
- Mathematical Calculation Functions
- Mathematical Statistical Function
- General Aggregate Function
- Geospatial Function
- Binary String Function
- Estimation Function
- Type Conversion Function
- Logical Function
- Operators
- Bitwise Operation
- Regular Expression Function
- Lambda Function
- Conditional Expressions
- Array Functions
- Interval-Valued Comparison and Periodicity-Valued Comparison Functions
- JSON Functions
- Window Functions
- Sampling Analysis
- Configuring Indexes
- Reindexing
- Context Search and Analysis
- Downloading Log
- Dashboard
- Data Processing documents
- Data Processing
- Creating Processing Task
- Viewing Data Processing Details
- Data Processing Functions
- Function Overview
- Key-Value Extraction Functions
- Enrichment Functions
- Flow Control
- Row Processing Functions
- Field Processing Functions
- Value Structuring Functions
- Regular Expression Processing Functions
- Time Value Processing Functions
- String Processing Functions
- Type Conversion Functions
- Logical and Mathematical Functions
- Encoding and Decoding Functions
- IP Parsing Functions
- Processing Cases
- Scheduled SQL Analysis
- SCF
- Data Processing
- Shipping and Consumption
- Monitoring Alarm
- Historical Documentation
- Best Practices
- Developer Guide
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- Topic Management APIs
- Log Set Management APIs
- Index APIs
- Topic Partition APIs
- Machine Group APIs
- Collection Configuration APIs
- Log APIs
- Metric APIs
- Alarm Policy APIs
- Data Processing APIs
- Kafka Protocol Consumption APIs
- CKafka Shipping Task APIs
- Kafka Data Subscription APIs
- COS Shipping Task APIs
- SCF Delivery Task APIs
- Scheduled SQL Analysis APIs
- COS Data Import Task APIs
- Data Types
- Error Codes
- FAQs
- CLS Service Level Agreement
- CLS Policy
- Contact Us
- Glossary
Overview
This document describes how to ship the collected raw log to COS. Currently, only logs collected in a single line, in multiple lines, or with certain separators are supported in this scenario.
Notes
You can configure CSV shipping to COS to ship certain raw logs. The following table describes the options of CSV shipping to COS but doesn't apply to JSON or Parquet shipping.
Log Collection Format | Support for Raw Log Shipping | User Raw Log | CLS Storage Format | Format for Shipping to COS |
Full text in a single line | yourlog | __CONTENT__:yourlog | yourlog | |
Full text in multiple lines | yourlog | __CONTENT__:yourlog | yourlog | |
Separator (CSV) format | It depends. For more information, see CSV format. Only space, tab, comma, semicolon, and vertical bar are supported as raw log separators. | V1 separator V2 separator V3 separator...Vn For example, V1,V2,V3...Vn | K1:V1 K2:V2 K3:V3...Kn:Vn | V1 separator V2 separator V3 separator...Vn For example, V1,V2,V3...Vn |
JSON format | No | K1:V1 K2:V2 K3:V3...Kn:Vn | K1:V1 K2:V2 K3:V3...Kn:Vn | V1,V2,V3...Vn differs from the original JSON. |
Full regex | No | V11V22V33...Vnn | K1:V1 K2:V2 K3:V3...Kn:Vn | V1,V2,V3...Vn differs from the raw log. |
Directions
Log collected in a single line or multiple lines
For logs collected in a single line or multiple lines, you can configure parameters based on CSV shipping to implement raw log shipping.
1. Complete the first step of Basic Configuration as instructed in CSV Shipping.
2. Set Shipping Format to CSV, retain the
__CONTENT__ field, delete other fields, set Separator to Space, set Escape Character to Space, set Invalid Field Filling to None, and disable Key in First Line.
The parameters are described as follows:Configuration Item | Description | Remarks |
Key | __CONTENT__ | For full text in a single line or multiple lines, __CONTENT__ is used as the default key, and the raw log is used as the value. When the raw log is shipped, only the __CONTENT__ field is retained. |
Separator | Space | Set **Separator** to **Space** for the full text in a single line or multiple lines. |
Escape Character | None | To prevent the raw log from being modified due to escape characters, set Escape Character to None. |
Invalid Field Filling | None | Set Invalid Field Filling to None. |
Key in First Line | Disabled | You don't need to add a description of the field name in the first line of the CSV file for raw log shipping. |
3. Click OK to enable shipping.
Logs collected in CSV format
Note:
CSV shipping supports limited types of separators, including space, tab, comma, semicolon, and vertical bar. Therefore, you can ship log data in the raw format only when the separator in the raw log content is supported by CSV shipping.
1. Complete the first step of Basic Configuration as instructed in CSV Shipping.
2. Set Shipping Format to CSV. During field configuration, you need to delete CLS metadata fields.
The parameters are described as follows:
Configuration Item | Value | Description |
Key | Keys | Only the user field is retained. |
Separator | A value selected from the drop-down list | Select the separator of the raw log content. If separators are different, raw log shipping is not supported. Currently, only space, tab, comma, semicolon, and vertical bar are supported. |
Escape Character | None | To prevent the raw log from being modified due to escape characters, set Escape Character to None. |
Invalid Field Filling | None | Set Invalid Field Filling to None. |
Key in First Line | Disabled | You don't need to add a description of the field name in the first line of the CSV file for raw log shipping. |
3. Click OK to enable shipping.
Yes
No
Was this page helpful?