- Release Notes and Announcements
- User Guide
- Product Introduction
- Purchase Guide
- Getting Started
- Operation Guide
- Resource Management
- Permission Management
- Log Collection
- Collection Overview
- Collecting Logs in Self-Built Kubernetes Cluster
- Collecting Syslog
- Collection by LogListener
- Collecting Text Log
- Uploading Log over Kafka
- Uploading Logs via Anonymous Write
- Uploading Logs via Logback Appender
- Uploading Logs via Log4j Appender
- Uploading Log via SDK
- Uploading Log via API
- Importing Data
- Tencent Cloud Service Log Access
- Log Storage
- Search and Analysis
- Syntax and Rules
- Statistical Analysis (SQL)
- Quick Analysis
- SQL Syntax
- SQL Functions
- String Function
- Date and Time Functions
- IP Geographic Function
- URL Function
- Mathematical Calculation Functions
- Mathematical Statistical Function
- General Aggregate Function
- Geospatial Function
- Binary String Function
- Estimation Function
- Type Conversion Function
- Logical Function
- Operators
- Bitwise Operation
- Regular Expression Function
- Lambda Function
- Conditional Expressions
- Array Functions
- Interval-Valued Comparison and Periodicity-Valued Comparison Functions
- JSON Functions
- Window Functions
- Sampling Analysis
- Configuring Indexes
- Reindexing
- Context Search and Analysis
- Downloading Log
- Dashboard
- Data Processing documents
- Data Processing
- Creating Processing Task
- Viewing Data Processing Details
- Data Processing Functions
- Function Overview
- Key-Value Extraction Functions
- Enrichment Functions
- Flow Control
- Row Processing Functions
- Field Processing Functions
- Value Structuring Functions
- Regular Expression Processing Functions
- Time Value Processing Functions
- String Processing Functions
- Type Conversion Functions
- Logical and Mathematical Functions
- Encoding and Decoding Functions
- IP Parsing Functions
- Processing Cases
- Scheduled SQL Analysis
- SCF
- Data Processing
- Shipping and Consumption
- Monitoring Alarm
- Historical Documentation
- Best Practices
- Developer Guide
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- Topic Management APIs
- Log Set Management APIs
- Index APIs
- Topic Partition APIs
- Machine Group APIs
- Collection Configuration APIs
- Log APIs
- Metric APIs
- Alarm Policy APIs
- Data Processing APIs
- Kafka Protocol Consumption APIs
- CKafka Shipping Task APIs
- Kafka Data Subscription APIs
- COS Shipping Task APIs
- SCF Delivery Task APIs
- Scheduled SQL Analysis APIs
- COS Data Import Task APIs
- Data Types
- Error Codes
- FAQs
- CLS Service Level Agreement
- CLS Policy
- Contact Us
- Glossary
- Release Notes and Announcements
- User Guide
- Product Introduction
- Purchase Guide
- Getting Started
- Operation Guide
- Resource Management
- Permission Management
- Log Collection
- Collection Overview
- Collecting Logs in Self-Built Kubernetes Cluster
- Collecting Syslog
- Collection by LogListener
- Collecting Text Log
- Uploading Log over Kafka
- Uploading Logs via Anonymous Write
- Uploading Logs via Logback Appender
- Uploading Logs via Log4j Appender
- Uploading Log via SDK
- Uploading Log via API
- Importing Data
- Tencent Cloud Service Log Access
- Log Storage
- Search and Analysis
- Syntax and Rules
- Statistical Analysis (SQL)
- Quick Analysis
- SQL Syntax
- SQL Functions
- String Function
- Date and Time Functions
- IP Geographic Function
- URL Function
- Mathematical Calculation Functions
- Mathematical Statistical Function
- General Aggregate Function
- Geospatial Function
- Binary String Function
- Estimation Function
- Type Conversion Function
- Logical Function
- Operators
- Bitwise Operation
- Regular Expression Function
- Lambda Function
- Conditional Expressions
- Array Functions
- Interval-Valued Comparison and Periodicity-Valued Comparison Functions
- JSON Functions
- Window Functions
- Sampling Analysis
- Configuring Indexes
- Reindexing
- Context Search and Analysis
- Downloading Log
- Dashboard
- Data Processing documents
- Data Processing
- Creating Processing Task
- Viewing Data Processing Details
- Data Processing Functions
- Function Overview
- Key-Value Extraction Functions
- Enrichment Functions
- Flow Control
- Row Processing Functions
- Field Processing Functions
- Value Structuring Functions
- Regular Expression Processing Functions
- Time Value Processing Functions
- String Processing Functions
- Type Conversion Functions
- Logical and Mathematical Functions
- Encoding and Decoding Functions
- IP Parsing Functions
- Processing Cases
- Scheduled SQL Analysis
- SCF
- Data Processing
- Shipping and Consumption
- Monitoring Alarm
- Historical Documentation
- Best Practices
- Developer Guide
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- Topic Management APIs
- Log Set Management APIs
- Index APIs
- Topic Partition APIs
- Machine Group APIs
- Collection Configuration APIs
- Log APIs
- Metric APIs
- Alarm Policy APIs
- Data Processing APIs
- Kafka Protocol Consumption APIs
- CKafka Shipping Task APIs
- Kafka Data Subscription APIs
- COS Shipping Task APIs
- SCF Delivery Task APIs
- Scheduled SQL Analysis APIs
- COS Data Import Task APIs
- Data Types
- Error Codes
- FAQs
- CLS Service Level Agreement
- CLS Policy
- Contact Us
- Glossary
Overview
Row processing functions process log rows, such as filtering, distributing, and splitting log rows.
Function log_output
Function definition
This function is used to output a row of log to a specified log topic. It can be used independently or together with branch conditions.
Syntax description
log_output(Alias). The alias is defined when the processing task is configured.
Field description
Parameter | Description | Type | Required | Default Value | Value Range |
alias | Log topic alias | string | Yes | - | - |
Sample
Distribute the log to 3 different log topics according to the values (
waring, info, and error) of the loglevel field.Raw log:
[{"loglevel": "warning"},{"loglevel": "info"},{"loglevel": "error"}]
Processing rule:
// The `loglevel` field has 3 values (`waring`, `info`, and `error`) and therefore the log is distributed to 3 different log topics accordingly.t_switch(regex_match(v("loglevel"),regex="info"),log_output("info_log"),regex_match(v("loglevel"),regex="warning"),log_output("warning_log"),regex_match(v("loglevel"),regex="error"),log_output("error_log"))
Function log_split
Function definition
This function is used to split a row of log into multiple rows of logs based on the value of a specified field by using a separator and JMES expression.
Syntax description
log_split(Field name, sep=",", quote="\\"", jmes="", output="")
Field description
Parameter | Description | Type | Required | Default Value | Value Range |
field | Field to extract | string | Yes | - | - |
sep | Separator | string | No | , | Any single character |
quote | Characters that enclose the value | string | No | - | - |
jmes | string | No | - | - | |
output | Name of a single field | string | Yes | - | - |
Sample
Example 1. Split a log whose
field has multiple values{"field": "hello Go,hello Java,hello python","status":"500"}
Processing rule:
// Use the separator "," to split the log into 3 logs.log_split("field", sep=",", output="new_field")
Processing result:
{"new_field":"hello Go","status":"500"}{"new_field":"hello Java","status":"500"}{"new_field":"hello python","status":"500"}
Example 2. Use a JMES expression to split a log
{"field": "{\\"a\\":{\\"b\\":{\\"c\\":{\\"d\\":\\"a,b,c\\"}}}}", "status": "500"}
Processing rule:
// The value of `a.b.c.d` is `a,b,c`.log_split("field", jmes="a.b.c.d", output="new_field")
Processing result:
{"new_field":"a","status":"500"}{"new_field":"b","status":"500"}{"new_field":"c","status":"500"}
Example 3. Split a log that contains a JSON array
{"field": "{\\"a\\":{\\"b\\":{\\"c\\":{\\"d\\":[\\"a\\",\\"b\\",\\"c\\"]}}}}", "status": "500"}
Processing rule:
log_split("field", jmes="a.b.c.d", output="new_field")
Processing result:
{"new_field":"a","status":"500"}{"new_field":"b","status":"500"}{"new_field":"c","status":"500"}
Function log_drop
Function definition
This function is used to delete logs that meet a specified condition.
Syntax description
log_drop(Condition 1)
Field description
Parameter | Description | Type | Required | Default Value | Value Range |
condition | Function expression whose return value is of bool type | bool | Yes | - | - |
Sample
Delete logs where
status is 200 and retain other logs.Raw log:
{"field": "a,b,c", "status": "500"}{"field": "a,b,c", "status": "200"}
Processing rule:
log_drop(op_eq(v("status"), 200))
Processing result:
{"field":"a,b,c","status":"500"}
Function log_keep
Function definition
This function is used to retain logs that meet a specified condition.
Syntax description
log_keep(Condition 1)
Field description
Parameter | Description | Type | Required | Default Value | Value Range |
condition | Function expression whose return value is of bool type | bool | Yes | - | - |
Sample
Retain logs where
status is 500 and delete other logs.Raw log:
{"field": "a,b,c", "status": "500"}{"field": "a,b,c", "status": "200"}
Processing rule:
log_keep(op_eq(v("status"), 500))
Processing result:
{"field":"a,b,c","status":"500"}
Function log_split_jsonarray_jmes
Function definition
This function is used to split and expand the JSON array in the log according to JMES syntax.
Syntax description
log_split_jsonarray_jmes("field", jmes="items", prefix="")
Field description
Parameter | Description | Type | Required | Default Value | Value Range |
field | Field to extract | string | Yes | - | - |
Sample
Example 1
Raw log:
{"common":"common","result":"{\\"target\\":[{\\"a\\":\\"a\\"},{\\"b\\":\\"b\\"}]}"}
Processing rule:
log_split_jsonarray_jmes("result",jmes="target")fields_drop("result")
Processing result:
{"common":"common", "a":"a"}{"common":"common", "b":"b"}
Example 2
Raw log:
{"common":"common","target":"[{\\"a\\":\\"a\\"},{\\"b\\":\\"b\\"}]"}
Processing rule:
log_split_jsonarray_jmes("target",prefix="prefix_")fields_drop("target")
Processing result:
{"prefix_a":"a", "common":"common"}{"prefix_b":"b", "common":"common"}
Yes
No
Was this page helpful?