Custom metadata can be used to distinguish logs. You can define metadata for logs when configuring collection rules in the Cloud Log Service (CLS) console. CLS currently supports the following types of custom metadata:
Machine group metadata
Collection path
Custom
Prerequisites
LogListener 2.8.7 or later
Machine group metadata
When you need to collect logs from multiple machine groups with the same target collection path, you can associate all the machine groups through a single collection configuration and use machine group metadata to clearly differentiate which machine group generated each collected log.
3. In the Collection Confirguration step, enable Custom metadata and set Metadata type to Machine group metadata.
4. Once the configuration is completed, when LogListener reports logs, it will upload the metadata attributes of the machine group where the logs are deployed to CLS in the format of __TAG__.{Machine group metadata key}:{Machine group metadata value}.
Collection path
When your target collection path is a fuzzy match, you can extract specified fields from the file path as custom metadata and use this metadata to clearly distinguish logs based on the file path from which they originate.
Specific Operations
1. In the Collection Confirguration, enable Custom metadata and set Metadata type to Collection Path.
2. Specify the file path regular expression. The regular expression must fully match the file path. Use "()" to identify the regular expression corresponding to the target key in the path.
3. When collecting logs, LogListener treats "()" as a capture group. LogListener supports the following two capture groups:
Named capture group
A named capture group identifies the name of the field extracted by the regular expression in brackets through ?<> and reports it together with logs in the form of __TAG__.{Field name}:{Extracted field}. For example, (?<name>.*?) signifies that the field extracted by .*? will be named "name". Up to 5 named capture groups are supported.
Non-named capture group
Non-named capture groups do not perform special naming for extracted fields. By default, they use the serial number of the capture group as the field name and report it together with logs in the form of __TAG__.{i}:{Extracted field}, where i indicates the serial number of the capture group. Up to 5 non-named capture groups are supported.
Example
Assume that the file directory structure is as follows:
/logs
| - /appA/userA
| - access.log
| - /appB/userB
| - access.log
| - /appC/userC
| - access.log
If the target collection path is configured with fuzzy matching, the directory prefix is set as /logs, and the file name is set as access.log, the three access.log files will be collected under the same log topic. This makes it impossible to clearly distinguish which specific log file the log entries originate from during search and analysis. However, we can use the following regular expression to extract values from the file path and upload them as metadata along with the logs.
Regular expression:
/logs/(.*?)/.*
After being extracted via the above regular expression, the following metadata will be reported:
# /logs/appA/userA/access.log will include a new key value.
__TAG__.1: appA
# /logs/appB/userB/access.log will include a new key value.
__TAG__.1: appB
# /logs/appC/userC/access.log will include a new key value.
__TAG__.1: appC
Furthermore, you can use the following regular expression to extract multiple values from the file path as metadata to be reported, by using multiple named capture groups, and assign meaningful names to the extracted values.
Regular expression:
/logs/(?<APP>.*?)/(?<USER>.*?)/access.log
After being extracted via the above regular expression, the following metadata will be reported:
# /logs/appA/userA/access.log will include a new key value.
__TAG__.APP: appA
__TAG__.USER: userA
# /logs/appB/userB/access.log will include a new key value.
__TAG__.APP: appB
__TAG__.USER: userB
# /logs/appC/userC/access.log will include a new key value.
__TAG__.APP: appC
__TAG__.USER: userC
Custom
You can also configure custom metadata when setting up collection rules. LogListener will include the specified custom metadata when reporting each log entry.
Specific Operations
1. In the Collection Confirguration, enable Custom metadata and set Metadata type to Custom.
2. Specify the key and value of custom metadata.
3. When reporting each log entry, LogListener includes the specified custom metadata.
