NGINX, as a common reverse proxy service, handles a large number of service requests in actual business scenarios. While running, the service generates a massive amount of access logs. Typically, users face issues such as logs being scattered across the cluster and huge data volumes. Therefore, effectively collecting and managing these logs is crucial for both business Ops and operations. This document uses NGINX access logs as an example to illustrate how to integrate NGINX logs into Cloud Log Service (CLS).
NGINX Log Format
The format of NGINX access logs (access.log) can be defined using the log_format command. Example format:
log_format main '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent"';
The description of each field is as follows:
Field Name
Description
remote_addr
Client IP address.
remote_user
Client name.
time_local
Local server time.
request
HTTP request method and URL.
status
HTTP request status code.
body_bytes_sent
Number of bytes sent to the client.
http_referer
Page URL of the access source.
http_user_agent
Client browser information.
Operation Steps
Step 1: Creating/Selecting a Log Topic
Selecting a New Log Topic
Selecting an Existing Log Topic
If you want to select a new log topic, perform the following operations:
2. In the left sidebar, click Overview to go to the overview page.
3. In Fast Integration, find and click Nginx Log under Server and application to go to the collection configuration process page.
4. On the Creat Log Topic page of the data collection configuration process, enter a log topic name, configure the log retention period according to actual needs, and click Next.
If you want to select an existing log topic, perform the following operations:
2. In the left sidebar, click Log Topic. Then, select the log topic you want to ship and click its name to go to the log topic management page.
3. Select the Collection Configuration tab, click Add in the LogListener collection configuration section to go to the collection configuration process page.
4. On the log data source selection page, find and click Nginx Log under Server and application to go to the collection configuration process page.
Step 2: Managing Machine Groups
After creating or selecting a log topic, click Next to go to the Machine Group Management page.
If the target server from which you want to collect logs does not have LogListener installed, see:
If you want to create a machine group, perform the following operations:
(1) Click Create Machine Group, and in the pop-up window for creating a machine group, associate the target server with LogListener installed via the IP address or machine label. For details, see Machine Groups.
(2) After the machine group is created, find and select the created machine group from the list.
If you want to select an existing machine group, directly find your target machine group in the list:
Step 3: Collection Configuration
After completing the machine group configuration, click Next to go to the Collection Configuration page of the collection configuration process.
Configuring the Log File Collection Path
On the Collection Configuration page, enter a collection rule name and specify the Collection Path based on the log collection path format. Example format:
Note:
For Linux systems, the log path must start with /. For Windows systems, the file path must start with a drive letter, such as C:\\.
Log path in a Linux system: /[Directory prefix expression]/**/[File name expression]. Example: /data/log/**/*.log.
Log path in a Windows system: [Drive letter]:\\[Directory prefix expression]\\**\\[File name expression]. Example: C:\\Program Files\\Tencent\\...\\*.log.
After the log collection path is specified, LogListener will match all common prefix paths that meet the rules based on the [directory prefix expression], and listen to all log files under these directories (including subdirectories) that match the [file name expression] rules. The detailed parameter description is as follows:
Field
Description
Directory Prefix
Directory structure prefix of the log file. Only the wildcards * and ? are supported.
* matches multiple arbitrary characters.
? matches a single arbitrary character.
Commas (,) are not supported.
**
Indicates the current directory and all subdirectories.
File Name
Log file name. Only the wildcards * and ? are supported.
* matches multiple arbitrary characters.
? matches a single arbitrary character.
Commas (,) are not supported.
Common configuration patterns are as follows:
/[Common directory prefix]/**/[Common file name prefix]*
/[Common directory prefix]/**/*[Common file name suffix]
/[Common directory prefix]/**/[Common file name prefix]*[Common file name suffix]
/[Common directory prefix]/**/*[Common string]
Example configurations:
No.
Directory Prefix Expression
File Name Expression
Description
1.
/var/log/nginx
access.log
In this example, the log path is configured as /var/log/NGINX/**/access.log. LogListener will listen to all log files named access.log in the subdirectories under the /var/log/NGINX prefix path.
2.
/var/log/nginx
*.log
In this example, the log path is configured as /var/log/NGINX/**/*.log. LogListener will listen to all log files ending with .log in the subdirectories under the /var/log/NGINX prefix path.
3.
/var/log/nginx
error*
In this example, the log path is configured as /var/log/NGINX/**/error*. LogListener will listen to all log files starting with error in the subdirectories under the /var/log/NGINX prefix path.
Note:
Windows environments do not support soft link collection.
Only LogListener 2.3.9 and later versions support adding multiple collection paths.
In log rotation scenarios, it is recommended to configure the collection path as log/*.log and rename the old file after log rotation as log/*.log.xxxx.
By default, a log file can only be collected by one log topic. If you need multiple collection configurations for a file and the file resides in a Linux environment, add a soft link to the source file and add it to another set of collection configurations.
Configuring Collection Path Blocklist
After the collection path blocklist is enabled, the specified directory prefixes or complete file paths can be ignored during collection. Directory paths and file paths can be matched exactly or using wildcard patterns.
The collection blocklist supports two filter types, which can be used simultaneously:
File name: In the collection path, the complete file path for the collection needs to be ignored. The wildcard * or ? is supported, and ** path fuzzy matching is supported.
Directory: In the collection path, the directory prefix for the collection needs to be ignored. The wildcard * or ? is supported, and ** path fuzzy matching is supported.
Note:
LogListener 2.3.9 or later is required.
The collection blocklist excludes paths under the collection path. Therefore, in both file name mode and directory mode, the specified path should be a subset of the collection path.
Configuring the Collection Policy
All Collection: When LogListener collects a file, it reads from the beginning of the file.
New Collection: When LogListener collects a file, it collects only the newly added content in the file.
Configuring Backtracking Collection
Note:
Windows environments currently do not support backtracking collection.
When Collection Policy is set to New Collection, you can further specify whether to start collecting from the position offset by the specified number of bytes from the latest position when LogListener starts collecting.
Configuring the Encoding Mode
UTF-8: Select this option if your log file is encoded in UTF-8 mode.
GBK: Select this option if your log file is encoded in GBK mode.
Configuring the Extraction Mode
Select NGINX Log Template as the extraction mode and populate the log template in the NGINX configuration file (usually /etc/NGINX/NGINX.conf or /usr/local/NGINX/conf/NGINX.conf), which usually starts with log_format. Example:
log_format main '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent"';
CLS will automatically generate a regular expression for log extraction based on the above NGINX log template. Example:
Specify an actual NGINX log in the sample log to verify whether the above extraction mode configuration is correct. Example:
59.x.x.x - - [06/Aug/2019:12:12:19 +0800] "GET /nginx-logo.png HTTP/1.1" 200 368 "http://119.x.x.x/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36" "-"
After successful verification of the extraction mode, you can set a name for each field or directly use the default value:
Configuring Custom Metadata
Note:
Windows environments currently do not support custom metadata.
LogListener 2.8.7 or later is required.
You can configure custom metadata to distinguish logs. The following metadata configurations are supported. For details, see Custom Metadata.
Machine group metadata: Use the machine group metadata.
Collection Path: Extract values from the collection path using regular expressions and set them as metadata.
Custom: Define custom key-value pairs as metadata.
Configuring Filter Conditions
The purpose of the filter is to add log collection filter rules based on business needs, helping you filter out valuable log data.
To collect logs in full regular expression mode, you need to configure filter conditions based on the custom key-value pairs. The following filtering rules are supported:
Equal to: collects only logs with specified field values matching the specified characters. Both exact match and regular expression match are supported.
Not equal to: collects only logs whose specified field values do not match the specified characters. Both exact match and regular expression match are supported.
Field exists: collects only logs where the specified field exists.
Field does not exist: collects only logs where the specified field does not exist.
For example, if you want to collect all log data where the status field is set to 400 or 500 after the sample log is parsed in full regular expression mode, you need to configure key as status, the filtering rule as Equal to, and the value as 400|500.
Note:
Windows environments currently do not support custom metadata.
The filtering rules "Not equal to", "Field exists", and "Field does not exist" are only supported in LogListener 2.9.3 and later versions.
Multiple filtering conditions are in an AND relationship. If multiple filtering conditions are configured for the same key name, the rule will be overwritten.
Configuring Upload Parsing-Failed Logs
It is recommended to enable the upload of logs that failed to be parsed. After this feature is enabled, LogListener will upload various logs that failed to be parsed. If this feature is disabled, the logs that failed to be parsed will be discarded.
After this feature is enabled, you need to configure the key value for parsing failures (which is LogParseFailure by default). All logs that failed to be parsed are uploaded with the input content as the key name (Key) and the raw log content as the key value (Value).
Uploading Raw Logs
After this feature is enabled, LogListener will upload raw logs together with parsed logs. All raw logs are uploaded with the key name you specified and the raw log content as the value.
Advanced Configurations
Note:
Windows environments currently do not support backtracking collection.
Select the advanced configuration options you need by checking the corresponding items:
In single-line full-text mode, the following advanced configurations are supported:
Name
Description
Configuration Item
Timeout property
This configuration controls the timeout for log files. If a log file has no updates within the specified time, it is considered timed out. LogListener will stop collecting from that timed-out log file. If you have a large number of log files, it is recommended to reduce the timeout to avoid LogListener performance waste.
No time out: Log files never time out.
Custom: The timeout for log files can be customized.
Maximum directory levels
/**/ in the collection path represents searching through all subdirectories for files. However, if you do not want to search too deeply into subdirectories, you can use the Maximum Directory Depth configuration item to limit the search depth.
An integer greater than 0. 0 means no drilling down into subdirectories.
Data Processing
If you need to further process the collected CLS logs, such as structuring, masking, or filtering, before writing them into the log topic, you can click Data Processing at the bottom of the Collection Configuration page, add data processing, and then configure the index.
Note:
For data processing-related operations, see the Immediate Processing After Collection tab in Creating a Processing Task.
