The delimiter-based mode is applicable to the log parsing mode in which each line of log text contains an original log, and each log can be extracted as multiple key-values according to the specified separator. If you do not need to extract the key-value, please refer to Full Text in a Single Line Mode for configuration. This article shows you how to collect logs using the delimiter extraction pattern.
Prerequisites
The server where the target file resides has LogListener installed. For details, see:
When the log parsing separator is specified as :::, this log will be divided into eight fields and each of these fields will be assigned a unique key, as shown below:
IP: 10.20.20.10 -
bytes: 35
host: 127.0.0.1
length: 647
referer: http://127.0.0.1/
request: GET /online/sample HTTP/1.1
status: 200
time: [Tue Jan 22 14:49:45 CST 2019 +0800]
Operation Steps
Step 1: Creating/Selecting a Log Topic
Create a Log Topic
Select an Existing Log Topic
If you want to create a new log topic, perform the following operations:
2. In the left sidebar, select Overview to go to the overview page.
3. IIn Fast Integration > Server and application, locate and click Separator - File log to enter the data collection configuration process.
4. On the Create Log Topic page, specify the log topic name, configure the log storage duration, select a logset based on your actual requirements, and click Next.
To select an existing log topic, perform the following steps:
2. In the left sidebar, select Log Topic, then select the log topic to be delivered, click the designated log topic name, and enter the log topic management page.
3. Select the Collection Configuration tab, click Add under the LogListener collection configuration section, and enter the log data source selection.
4. On the log data source selection page, select Server and application, locate and click Separator - File log to enter the data collection configuration process.
Step 2: Managing Machine Groups
If the target server where you need to collect logs does not have LogListener installed, see:
If you want to create a machine group, perform the following operations:
1. Click Create Machine Group.
2. Fill in the machine group name, associate the target server with LogListener installed via machine id (see Machine Group for details), and then click OK.
3. After creation is completed, select the system environment of your created machine group from the Tab options, check your target machine group in the list, and click Next.
If you want to select an existing machine group, select the system environment of your created machine group from the Tab options, check your target machine group in the list, and click Next.
Step 3: Collection Configuration
Configuring the Log File Collection Path
On the Collection Configuration page, fill in the collection rule name and the Collection Path according to the log collection path format. See the following for the log collection path format:
Note:
For Linux systems, the log path must start with /. For Windows systems, the file path must start with a drive letter, such as C:\\.
Linux log path in the system: /[directory prefix expression]/**/[file name expression], such as /data/log/**/*.log.
Log path in Windows systems: [drive letter]:\\[directory prefix expression]\\**\\[file name expression], such as C:\\Program Files\\Tencent\\...\\*.log.
After the log collection path is filled in, LogListener will match all common prefix paths that meet the rules according to [directory prefix expression] and monitor all log files that meet the [file name expression] rule under these directories (including sub-layer directories). The parameters are detailed as follows:
Number
Directory Prefix Expression
File Name Expression
Description
1
/var/log/nginx
access.log
In this example, the log path is configured as /var/log/nginx/**/access.log, and LogListener will monitor the log files named access.log in all subdirectories under the prefix path /var/log/nginx.
2
/var/log/nginx
*.log
In this example, the log path is configured as /var/log/nginx/**/*.log, and LogListener will monitor the log files ending with .log in all sub-directories under the prefix of /var/log/nginx.
3
/var/log/nginx
error*
In this example, the log path is configured as /var/log/nginx/**/error*, and LogListener will monitor the log files named starting with error in all subdirectories under the prefix path /var/log/nginx.
Note:
Windows environments do not support soft link collection.
Only LogListener 2.3.9 and later versions support adding multiple collection paths.
It is recommended to configure the collection path as log/*.log, and rename the rotated old log files as log/*.log.xxxx.
By default, a log file can only be collected by one log topic. If you need multiple collection configurations for a file and the file resides in a Linux environment, add a soft link to the source file and add it to another set of collection configurations.
Configuring the Blocklist of Data Collection Paths
Enable the blocklist of collection paths to ignore the specified directory prefix or complete file path during collection. Directory paths and file paths can be fully matched, and wildcard pattern matching is also supported.
The collection blocklist is divided into two types of filtering and can be used at the same time:
File name: In the collection path, the complete file path for the collection needs to be ignored. The wildcard * or ? is supported, and ** path fuzzy matching is supported.
Directory: In the collection path, the directory prefix for the collection needs to be ignored. The wildcard * or ? is supported, and ** path fuzzy matching is supported.
Note:
LogListener version 2.3.9 or later is required.
The collection blocklist excludes paths under the collection path. Therefore, in both file name mode and directory mode, the specified path should be a subset of the collection path.
Configuring Collection Policy
All Collection: When LogListener collects a file, it reads from the beginning of the file.
New Collection: When LogListener collects a file, it collects only the newly added content in the file.
Configuring Backtracking Collection
When Collection Policy is set to New collection, you can further set the starting point for backtracking collection herein and specify whether to start collecting from the position offset by the specified number of bytes from the latest position when LogListener starts.
Note:
Windows environments currently do not support custom metadata.
Encoding Mode
UTF-8: Select this option if your log file encoding mode is UTF-8.
GBK: Select this option if your log file encoding mode is GBK.
Configure Delimiter-Based Mode
1. Set Extraction Mode to Separator.
2. Select Separator, enter a log sample in the "Log Example" text box, and click Extract.
The system divides the log sample according to the determined delimiter and displays it in the extraction result column. You need to define a unique key for each field. Currently, log collection supports a variety of delimiters. Common delimiters include space, tab character, comma, semicolon, and vertical line. If your log data uses another symbol such as :::, you can also parse it through a custom separator.
Configuring Custom Metadata
Note:
Custom metadata can only be configured with LogListener 2.8.7 and later versions.
You can configure custom metadata to distinguish logs. The following metadata configurations are supported. For details, see Custom Metadata.
Machine group metadata: Use the machine group metadata.
Collection path: Extract the value in the acquisition path as metadata through regularization.
Custom: customize key values as metadata.
Configure Log Timestamp Source
LogLog Collection Time']
Specify Log Fields
Use the time of log collection as the log time.
Use the value of the specified field in the log as the log time.
1. Select the extracted Value from the Log Time Field as the log time.
2. In Time Parsing Format, manually enter or select the corresponding parsing expression. For example: the value representing time in logs is 07/Jul/2025:19:19:30 +0800, and the parsing format is %d/%b/%Y :%H:%M:%S %z. For more information, please refer to configure time format.
3. Click Verify.
Note:
If the time format is incorrect, the log time will be subject to the collection time.
Configure Filter Conditions
The purpose of the filter is to add log collection and filtering rules according to business requirements, so as to help you screen out valuable log data.
Delimiter format logs need to be configured with filtering criteria based on custom key-value pairs. The following filtering rules are supported:
Equal to: Only collect logs with specified field values matching the specified characters. Exact or regular matching is supported.
Not equal to: Only collect logs whose specified field values do not match the specified characters. Exact or regular matching is supported.
Field exists: Only logs where the specified field exists are collected.
Field does not exist: Only logs in which the specified field does not exist are collected.
For example, after the sample log is parsed in the separator mode, if you want all log data with status field of 400 or 500 to be collected, configure status at key, select equal to as the filtering rule, and configure 400|500 at value.
Note:
Windows environments currently do not support custom metadata.
Filter rules "not equal to", "field exists", and "field does not exist" are supported in LogListener 2.9.3 and above versions.
The relationship between multiple filter conditions is "AND" logic. If multiple filter conditions are configured for the same key name, rules will be overwritten.
Configure the Upload of Logs Failed to Be Parsed
It is recommended to enable upload parsing-failed logs. When enabled, LogListener will upload various logs that fail to be parsed. If upload parsing-failed logs is disabled, the failed log will be discarded.
After this function is enabled, the key value (LogParseFailure by default) failed to be parsed needs to be configured. All logs failed to be parsed are uploaded with the input content as the key name (Key), and the original log content as the value (Value).
Advanced Configuration
Note:
Windows environments currently do not support custom metadata.
Select the advanced configuration you need to define by checking .
In delimiter extraction mode, the following advanced settings are supported:
Name
Description
Configuration Item
Timeout property
This configuration controls the timeout period for the log file. If a log file has no updates within the specified time, it is timed out. LogListener will no longer collect the timed-out log file. When you have a large number of log files, recommend reducing timeout to avoid waste of LogListener performance.
No timeout: Log files never time out.
Custom: Timeout period of custom log files.
Maximum directory levels
The configuration controls the maximum directory depth for log collection. LogListener does not collect log files in directories that exceed the specified maximum depth. When your target collection path includes fuzzy matching, configure an appropriate maximum directory depth to avoid waste of LogListener performance.
An integer greater than 0. 0 represents no drill-down of subdirectories.
Step 4: Index Configuration
1. Click Next to enter the Index Configuration page.
2. On the "Index Configuration" page, configure the following information. For configuration details, please see Index Configuration.
Note:
Index configuration must be enabled before you can perform searches.
3. Click Submit to enter the edit index configuration confirmation page.
If you have set the index configuration to take effect only for newly written log, click Confirm. If you want this configuration to take effect for historical data, click Confirm, then see Rebuilds indexes for further settings.
4. Operation succeeded. Complete the collection configuration.
