tencent cloud

masukan

Collect container logs to CLS

Terakhir diperbarui:2023-05-25 10:41:57
    This document introduces how to configure log collection rules in the TKE console and ship logs to Tencent Cloud Log Service (CLS).

    Directions

    Creating log collection rules

    1. Log in to the TKE console and choose Log Management > Log Rules in the left sidebar.
    2. At the top of the Log Rules page, select a region and a cluster for which you want to configure log collection rules and click Create, as shown below:
    
    3. On the Create Log Collecting Policy page, configure the consumer of logs: Set Type to CLS in the Consumer end area, as shown in the figure below:
    
    Rule name: You can customize the log collection rule name.
    Log region: CLS supports cross-region log shipping. You can click Modify to select the destination region for log shipping.
    Logset: Created logsets are displayed by log region. If the existing logsets are not suitable, you can create a new one in the CLS console. For operation details, see Creating logset.
    Log topic: Select the corresponding log topic under the logset. Two modes are supported: Auto-create log topic and Select existing log topic.
    Advanced settings:
    Default metadata: CLS sets the metadata pod_name, namespace, and container_name as indexes for log searches by default.
    Custom metadata: You can customize metadata and log indexes.
    Note
    CLS does not support cross-region log shipping (from the Chinese mainland to outside the Chinese mainland and vice versa). For regions where CLS is not activated, logs can be shipped only to the nearest regions. For example, the container logs collected from a Shenzhen cluster can be shipped only to Guangzhou, and the containers logs collected from a Tianjin cluster can be shipped only to Beijing. You can find more information in the console.
    Currently, a log topic supports the collection configuration of only one type of logs. That is, the log, audit, and event types cannot use the same topic. If they use the same topic, logs will be overwritten. Ensure that the selected log topic is not occupied by other collection configurations. A logset can contain up to 500 log topics.
    Custom metadata and metadata indexes cannot be modified once created. You can go to the CLS console to modify the configuration.
    4. Select a collection type and configure a log source. Supported collection types are Container standard output, Container file path, and Node file path.
    Container Standard Output Logs
    Container File Logs
    Node File Logs
    Log source supports All containers, Specify workload, and Specify Pod Labels, as shown below:
    
    
    
    Log source supports Specify workload and Specify Pod Labels.
    You can specify a file path or use wildcards for the collection path. For example, when the container file path is /opt/logs/*.log, you can specify the collection path as /opt/logs and the file name as *.log.
    
    
    Note
    For Container file path, the corresponding path cannot be a soft link or hard link. Otherwise, the actual path of the soft link will not exist in the collector's container, resulting in log collection failure.
    
    You can specify a file path or use wildcards. For example, when the container file paths for collection are /opt/logs/service1/*.log and /opt/logs/service2/*.log, you can specify the folder of the collection path as /opt/logs/service* and the file name as *.log.
    You can attach metadata in the key-value pair format to log records as needed.
    
    Note:
    For Node file path, the corresponding path cannot be a soft link or hard link. Otherwise, the actual path of the soft link will not exist in the collector, resulting in log collection failure.
    Each node log file can be collected to only one log topic.
    
    Note
    For Container standard output and Container file path (excluding Node file path/not mounted in hostPath), besides the original log content, the metadata related to the container or Kubernetes (such as the ID of the container that generated the logs) will also be reported to the CLS. Therefore, when viewing logs, users can trace the log source or search based on the container identifier or characteristics (such as container name and labels). The metadata related to the container or Kubernetes is shown in the table below:
    Field
    Description
    container_id
    ID of the container to which the log belongs
    container_name
    Name of the container to which the log belongs
    image_name
    Image name IP of the container to which the log belongs
    namespace
    Namespace of the Pod to which the log belongs
    pod_uid
    UID of the Pod to which the log belongs
    pod_name
    Name of the Pod to which the log belongs
    pod_lable_{label name}
    Labels of the Pod to which the log belongs (for example, if a Pod has two labels: app=nginx and env=prod, the reported log will have two metadata entries attached: pod_label_app:nginx and pod_label_env:prod)
    
    5. Configure the collection policy as Full or Incremental.
    Full: Collecting logs from the beginning of the log file.
    Incremental: Collecting logs 1 MB ahead of the end of the log file. For a log file less than 1 MB in size, incremental collection is equivalent to full collection.
    6. Click Next and select a log parsing method, as shown below:
    
    Encoding Mode: Supports UTF-8 and GBK.
    Extraction Mode: Supports multiple extraction modes, as described below:
    Parsing mode
    Description
    Reference
    Full text in a single line
    A log contains only one line of content, and the line break \\n marks the end of a log. Each log will be parsed into a complete string with CONTENT as the key. When log indexing is enabled, you can search for log content via full-text search. The time attribute of a log is determined by the collection time.
    Full text in multi lines
    A log with full text in multi lines spans multiple lines, and a first-line regular expression is used for matching. When a log in a line matches the preset regular expression, the line is considered as the beginning of a log, and the next matching line will be the end mark of the log. A default key, CONTENT, will be set as well. The time attribute of a log is determined by the collection time. The regular expression can be generated automatically.
    Single line - full regex
    The single-line - full regular expression mode is a log parsing mode where multiple key-value pairs can be extracted from a complete log. When configuring the single-line - full regular expression mode, you need to enter a sample log first and then customize your regular expression. After the configuration is completed, the system will extract the corresponding key-value pairs according to the capture group in the regular expression. The regular expression can be generated automatically.
    Multiple lines - full regex
    The multi-line - full regular expression mode is a log parsing mode where multiple key-value pairs can be extracted from a complete log that spans multiple lines in a log text file (such as Java program logs) based on a regular expression. When configuring the multi-line - full regular expression mode, you need to enter a sample log first and then customize your regular expression. After the configuration is completed, the system will extract the corresponding key-value pairs according to the capture group in the regular expression. The regular expression can be generated automatically.
    JSON
    A JSON log automatically extracts the key at the first layer as the field name and the value at the first layer as the field value to implement structured processing of the entire log. Each complete log ends with a line break \\n.
    Separator
    Logs in this format are structured with the specified separator, and each complete log ends with a line break \\n. You need to define a unique key for each separate field. Leave the field blank if you don’t want to collect it. At least one field is required.
    Filter: LogListener collects only logs that meet the filter rules. Key supports exact matching, and Filter Rule supports regular expression matching. For example, you can specify to collect only logs where ErrorCode is 404. You can enable the filter feature and configure rules as needed.
    Note
    Currently, one log topic supports only one collection configuration. Ensure that all container logs that adopt the log topic can accept the log parsing method that you choose. If you create different collection configurations under the same log topic, the earlier collection configurations will be overwritten.
    7. Click Done.

    Updating the log rules

    1. Log in to the TKE console and choose Log Management > Log Rules in the left sidebar.
    2. At the top of the Log Rules page, select the region and the cluster where you want to update the log collection rules and click Edit Collecting Rule at the right, as shown in the figure below:
    
    3. Update the configuration as needed and click Done.
    Note
    The logset and log topic of a created rule cannot be modified.

    References

    Besides using the TKE console, you can also configure log collection by using custom resource definition (CRD). For more information, see Using CRD to Configure Log Collection.
    Hubungi Kami

    Hubungi tim penjualan atau penasihat bisnis kami untuk membantu bisnis Anda.

    Dukungan Teknis

    Buka tiket jika Anda mencari bantuan lebih lanjut. Tiket kami tersedia 7x24.

    Dukungan Telepon 7x24