The JSON extraction mode is applicable to the log parsing mode in which each line of log text contains one original log, and each log can be extracted as multiple key-values according to JSON parsing rules. If you do not need to extract the key-value, please refer to Full Text in a Single Line Mode for configuration. This article shows you how to collect logs using the JSON extraction mode.
Prerequisites
The server where the target file resides has LogListener installed. For details, see:
2. In the left sidebar, select Overview to go to the overview page.
3. In Fast Integration > Server and application, locate and click JSON-File log to enter the data collection configuration process.
4. On the Log Topic Creation page of the data collection configuration process, enter a log topic name, configure the log retention period according to actual needs, and click Next.
SIf you want to select an existing log topic, perform the following operations:
2. In the left sidebar, select Log Topic, then select the log topic to be delivered, click the specified log topic name, and enter the log topic management page.
3. Select the Collection Configuration tab, click Add under the LogListener collection configuration section, and enter the log data source selection.
4. On the log data source selection page, select Servers and applications, locate and click JSON-file logs to enter the data collection configuration process.
Step 2: Managing Machine Groups
If you need to collect data from a target server without LogListener installed, see:
If you want to create a machine group, perform the following operations:
1. Click Create Machine Group.
2. Fill in the machine group name, associate the target server with LogListener installed via the machine label. For details, see Machine Group, and then click OK.
3. After creation is completed, select the system environment of your created machine group from the Tab options, check your target machine group in the list, and click Next.
If you want to select existing machine groups, select the system environment of your created machine group from the Tab options, check your target machine group in the list, and click Next.
Step 3: Collection Configuration
Configuring the Log File Collection Path
On the Collection Configuration page, enter a collection rule name and specify the Collection Path based on the log collection path format. Example format:
Note:
For Linux systems, the log path must start with /. For Windows systems, the file path must start with a drive letter, such as C:\\.
Log path in a Linux system: /[Directory prefix expression]/**/[File name expression]. Example: /data/log/**/*.log.
Log path in a Windows system: [Drive letter]:\\[Directory prefix expression]\\**\\[File name expression]. Example: C:\\Program Files\\Tencent\\...\\*.log.
After the log collection path is filled in, LogListener will match all common prefix paths that meet the rules according to [directory prefix expression] and monitor all log files that meet the [file name expression] rule under these directories (including sub-layer directories). The parameters are detailed as follows:
Number
Directory Prefix Expression
File Name Expression
Description
1
/var/log/nginx
access.log
In this example, the log path is configured as /var/log/nginx/**/access.log, and LogListener will monitor the log files named with access.log in all subdirectories under the prefix of /var/log/nginx.
2
/var/log/nginx
*.log
In this example, the log path is configured as /var/log/nginx/**/*.log, and LogListener will monitor the log files ending with .log in all sub-directories under the prefix of /var/log/nginx.
3
/var/log/nginx
error*
In this example, the log path is configured as /var/log/nginx/**/error*, and LogListener will monitor the log files named starting with error in all subdirectories under the prefix of /var/log/nginx.
Note:
Windows environments do not support soft link collection.
Only LogListener 2.3.9 and later versions support adding multiple collection paths.
It is recommended to configure the collection path as log/*.log, and rename the rotated old log files as log/*.log.xxxx.
By default, a log file can only be collected by one log topic. If you need multiple collection configurations for a file and the file resides in a Linux environment, add a soft link to the source file and add it to another set of collection configurations.
Configuring the Blocklist of Data Collection Paths
Enable the blocklist of collection paths to ignore the specified directory prefix or complete file path during collection. Directory paths and file paths can be fully matched, and wildcard pattern matching is also supported.
The collection blocklist supports two filter types, which can be used simultaneously:
File name: In the collection path, the complete file path for the collection needs to be ignored. The wildcard * or ? is supported, and ** path fuzzy matching is supported.
Directory: In the collection path, the directory prefix for the collection needs to be ignored. The wildcard * or ? is supported, and ** path fuzzy matching is supported.
Note:
LogListener 2.3.9 or later is required.
The collection blocklist excludes paths under the collection path. Therefore, in both file name mode and directory mode, the specified path should be a subset of the collection path.
Configuring Collection Policy
All collection: When LogListener collects a file, it reads from the beginning of the file.
New Collection: When LogListener collects a file, it collects only the newly added content in the file.
Configuring Backtracking Collection
When Collection Policy is set to New Collection, you can further specify the starting point for backtracking collection herein, whether to start collecting from the position offset by the specified number of bytes from the latest position when LogListener is started.
Note:
Windows environments currently do not support custom metadata.
Encoding Mode
UTF-8: Select this option if your log file encoding mode is UTF-8.
GBK: Select this option if your log file encoding mode is GBK.
Configure JSON Mode
Note:
LogListener only parses one layer of JSON. For deeper-level JSON parsing, configure it via index configuration.
On the Collection Configuration page, set the "Extraction Mode" to JSON. As shown below:
Configuring Custom Metadata
You can configure custom metadata to distinguish logs. The following metadata configurations are supported. For details, see Custom Metadata.
Machine group metadata: Use the machine group metadata.
Collection path: Extract the value in the acquisition path as metadata through regularization.
Custom: customize key values as metadata.
Note:
Custom metadata can only be configured with LogListener 2.8.7 and later versions.
Configuring Log Timestamp Source
Log Collection Time
Specify Log Fields
Use the time of log collection as the log time.
Use the value of the specified field in the log as the log time.
1. Select the extracted Value from the log time field as the log time.
2. In time format parsing, manually enter or select the corresponding parsing expression. For example: the value representing time in logs is 07/Jul/2025:19:19:30 +0800, and the parsing format is %d/%b/%Y :%H:%M:%S %z. For more information, please refer to configure time format.
3. Click Verify.
Note:
If the time format is incorrect, the log time will be subject to the collection time.
Configure Filter Conditions
The purpose of the filter is to add log collection and filtering rules according to business requirements, so as to help you screen out valuable log data.
For JSON format logs, you can configure filtering conditions according to the parsed key-value pairs. The following filtering rules are supported:
Equal to: Only collect logs with specified field values matching the specified characters. Exact or regular matching is supported.
Not equal to: Only collect logs whose specified field values do not match the specified characters. Exact or regular matching is supported.
Field exists: Only logs where the specified field exists are collected.
Field does not exist: Only logs in which the specified field does not exist are collected.
For example, if you want all log data with response_code of 400 or 500 in the original JSON format log content to be collected, then configure response_code at key, select equals as filtering rule, and configure 400|500 at value.
Note:
Windows environments currently do not support custom metadata.
The filtering rules "Not equal to", "Field exists", and "Field does not exist" are only supported in LogListener 2.9.3 and later versions.
Multiple filtering conditions are in an AND relationship. If multiple filtering conditions are configured for the same key name, the rule will be overwritten.
Configure the Upload of Logs Failed to Be Parsed
It is recommended to enable upload parsing-failed logs. When enabled, LogListener will upload various logs failed to be parsed. If upload parsing-failed logs is disabled, the failed log will be discarded.
After this function is enabled, the key value (LogParseFailure by default) failed to be parsed needs to be configured. All logs failed to be parsed are uploaded with the input content as the key name (Key), and the original log content as the value (Value).
Upload Raw Logs
When enabled, LogListener will upload raw logs and parsed logs together. All raw logs are uploaded with the key name you specified, and the raw log content as the value (Value).
Advanced Configuration
Note:
Windows environments currently do not support custom metadata.
Select the advanced configuration you need to define by checking .
In JSON extraction mode, the following advanced settings are supported:
Name
Description
Configuration Item
Timeout property
This configuration controls the timeout period for the log file. If a log file has no updates within the specified time, it is timed out. LogListener will no longer collect the timed-out log file. When you have a large number of log files, it is advisable to shorten the timeout to avoid waste of LogListener performance.
No timeout: Log file never times out
Custom: The timeout for log files can be customized.
Maximum directory levels
This configuration controls the maximum directory depth for log collection. LogListener will not collect log files from directories whose hierarchy exceeds the specified maximum depth. When your target collection path includes fuzzy matching, configure an appropriate maximum directory depth to avoid waste of LogListener performance.
An integer greater than 0. 0 means no drilling down into subdirectories.
Step 4: Index Configuration
1. Click Next to enter the Index Configuration page.
2. On the "Index Configuration" page, configure the following information. For configuration details, please see index configuration.
Note:
Index configuration must be enabled before you can perform searches.
3. Click Submit to enter the edit index configuration confirmation page.
If you want the index configuration you have set to only come into effect for newly written logs, click Confirm. If you want this configuration to take effect for historical data, after clicking Confirm, for details, see Rebuilding an Index for further settings.
4. Operation succeeded. Complete the collection configuration.
