tencent cloud

Log Structuration-JSON
Download PDF
Last updated:2024-12-18 16:37:58
Log Structuration-JSON
Last updated: 2024-12-18 16:37:58
Download PDF

Scenario description

Xiaowang collects logs in JSON format to CLS (Cloud Log Service, CLS) under the following two conditions:
1. The JSON is multi-layer nested. Xiaowang wants to extract the user and App fields, where user is a secondary nested field.
2. Xiaowang's JSON log is an array and needs to split multiple logs from the array.

Raw Log

Log 1: Nested JSON
Log 2: JSON Array
[
    {
        "content": {
            "App": "App-1",
            "start_time": "2021-10-14T02:15:08.221",
            "resonsebody": {
                "method": "GET",
                "user": "Tom"
            },
            "response_code_details": "3000",
            "bytes_sent": 69
        }
    },
    {
        "content": {
            "App": "App-2",
            "start_time": "2222-10-14T02:15:08.221",
            "resonsebody": {
                "method": "POST",
                "user": "Jerry"
            },
            "response_code_details": "2222",
            "bytes_sent": 1
        }
    }
]
{
    "timestamp": 1732099684144000,
    "topic": "log-containers",
    "records": [
        {
            "category": "kube-request",
            "log": "{\\"requestID\\":\\"12345\\",\\"stage\\":\\"Complete\\"}"
        },
        {
            "category": "db-request",
            "log": "{\\"requestID\\":\\"67890\\",\\"stage\\":\\"Response\\"}"
        }
    ]
}

Processing result

Log 1
Log 2
[{"App":"App-1","user":"Tom"},
{"App":"App-2","user":"Jerry"}]
[
{
"category":"kube-request",
"requestID":"12345",
"stage":"Complete",
"timestamp":"1732099684144000",
"topic":"log-containers"
},
{
"category":"db-request",
"requestID":"67890",
"stage":"Response",
"timestamp":"1732099684144000",
"topic":"log-containers"
}
]

DSL Processing Function

Log 1:Processing statement
Log 2:Processing statement
//Use the ext_json function to extract structured data from JSON data, by default, it will flatten all fields
ext_json("content")
//Discard the content field
fields_drop("content")
//Discard unnecessary fields bytes_sent,method,response_code_details,start_time
fields_drop("bytes_sent","method","response_code_details","start_time")
//Split logs from the array, splitting into 2 logs
log_split_jsonarray_jmes("records")
//Discard the original field records
fields_drop("records")
//Expand the KV pairs of the log
ext_json("log")
//Discard the original field log
fields_drop("log")


Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback