tencent cloud

Cloud Log Service

DocumentationCloud Log ServiceOperation GuideLog CollectionCollecting Logs in Self-Built Kubernetes ClusterConfiguring Log Collection in Self-Built Kubernetes Cluster via CRD

Configuring Log Collection in Self-Built Kubernetes Cluster via CRD

Download
Focus Mode
Font Size
Last updated: 2026-05-20 19:47:51

Overview

CLS supports log collection for self-built Kubernetes clusters. Before performing log collection on a self-built Kubernetes cluster, you need to use a custom resource definition (CRD) to define log collection configuration (LogConfig), and deploy Log-Provisioner, Log-Agent, and LogListener on the cluster. If you are a Tencent Kubernetes Engine (TKE) user, you can quickly access and use the CLS service by referring to Enabling log collection.

Prerequisites

You have created a cluster of Kubernetes 1.10 or above.
You have configured TencentCloud API permissions required for self-built Kubernetes log collection. For more information, see Examples of Custom Access Policies.
You have obtained the domain name (CLS_HOST) of the region of your log topic. For details of the CLS domain name list, see Available Regions.

Kubernetes Log Collection Principles

Log collection on a Kubernetes cluster requires the following:
LogConfig: CRD of log collection configuration, which defines where logs are collected, how they are parsed, and to which CLS log topic they are shipped after being parsed.
Log-Agent: component used to listen for changes in LogConfig and containers on nodes and dynamically calculate the actual positions of log files in containers on node hosts.
Log-Provisioner: component used to synchronize the log collection configuration defined in LogConfig to CLS.
LogListener: component used to collect log file content from node hosts, parses it, and uploads it to CLS.

Flowchart

1. Install LogListener in a self-built Kubernetes cluster
2. Define the LogConfig object
3. Create a LogConfig object
4. View LogConfig

Directions

Step 1. Install LogListener in a self-built Kubernetes cluster

First, you need to install the LogListener component in the self-built Kubernetes cluster as instructed here to collect logs to CLS.

Step 2. Define the LogConfig object

Define log collection configuration in the LogConfig object via CRD. Run the wget command to download the LogConfig.yaml CRD declaration file, using the master node path /usr/local/ as an example.
wget https://mirrors.tencent.com/install/cls/k8s/LogConfig.yaml
The LogConfig.yaml declaration file consists of the following two parts:
clsDetail: The configuration for shipping to CLS.
inputDetail: Log source configuration.
apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig                               ## Default value
metadata:
  name: test						      	## CRD resource name, which is unique in the cluster.
spec:
  clsDetail:							  	## The configuration for shipping to CLS
    ...
  inputDetail:                                ## Log source configuration
    ...

clsDetail (the configuration for shipping to CLS) field description

  clsDetail:
    # You need to specify the logset and topic names to automatically create a log topic, which cannot be modified after being defined.
    logsetName: test                      	## The name of the CLS logset. If there is no logset with this name, one will be created automatically. If there is such a logset, a log topic will be created under it.
    topicName: test             	   ## The name of the CLS log topic. If there is no log topic with this name, one will be created automatically.
                                              # Select an existing logset and log topic. If the logset is specified but the log topic is not, a log topic will be created automatically, which cannot be modified after being defined.
    logsetId: xxxxxx-xx-xx-xx-xxxxxxxx  	 ## The ID of the CLS logset. The logset needs to be created in advance in CLS.
    topicId: xxxxxx-xx-xx-xx-xxxxxxxx  	 ## The ID of the CLS log topic. The log topic needs to be created in advance in CLS and not occupied by other collection configurations.
    
    # Define the log topic configuration when a log topic is created automatically. The configuration cannot be modified after being defined.
    period: 30                                ## Optional. Lifetime, in days. Valid value range: 1~3600. For STANDARD_IA storage, the valid range is 7~3600 days. A value of 3640 indicates permanent retention.
    storageType: hot                          ## Optional. The storage type for the log topic. Valid values: hot (STANDARD storage), cold (STANDARD_IA storage). Default value: hot.
    hotPeriod: 7                              ## Optional. Transition period, in days. Valid value range: 7~3600. This parameter takes effect only when storageType is hot. The value of hotPeriod must be greater than that of period. The transition period is calculated as Period-hotPeriod.
    partitionCount: 1                         ## Optional. The number of partitions for the log topic. One partition is created by default. Up to 10 partitions can be created.
    autoSplit: "true"                         ## Optional. Whether to enable auto-splitting. Default value: true. The value must be enclosed in double quotation marks ("").
    maxSplitPartitions: 50                    ## Optional. The maximum number of splits after auto-splitting is enabled. Default value: 50.
    tags:                                     ## Optional. A list of Tag descriptions. By specifying this parameter, you can bind Tags to the corresponding log topic. Up to 9 Tag key-value pairs are supported. A resource can be bound to only one Tag key.
    - key: xxx                                ## Tag key
      value: xxx                              ## Tag value
    
    ### Define Collection Rules
    logType: json_log                         ## Log parsing format. `json_log` indicates json format, `delimiter_log` indicates delimiter format, `minimalist_log` indicates single-line full-text format, `multiline_log` indicates multi-line full-text format, `fullregex_log` indicates single-line full regex format, `multiline_fullregex_log` indicates multi-line full regex format, and `user_define_log` indicates Combined parsing. Default value: minimalist_log.
    excludePaths:                             ## Optional. A list of collection blocklist paths. This parameter is valid only when `inputDetail` is `containerFile` or `hostFile`.
    - type: File                              ## Type. Optional. Valid values: File or Path. 
      value: /xx/xx/xx/xx.log                 ## The value corresponding to `type`.
    userDefineRule: xxxxxx                    ## Optional. Custom collection rules, serialized Json string.
    
    ### Extraction and Filtering Rules
    extractRule:                              ## Optional. Extraction and filtering rules. If `ExtractRule` is set, `LogType` must also be set. For details, see the description of the `extractRule` object.
      timeKey: xxx                            ## The key name of the time field. `time_key` and `time_format` must be used together.
      timeFormat: xxx                         ## The format of the time field. Refer to the format specifiers for time in the c language's `strftime` function for output parameters.
      delimiter: xxx                          ## The delimiter for delimiter-type logs. This parameter is valid only when `log_type` is `delimiter_log`.
      logRegex: xxx                           ## The rule for matching the entire log. This parameter is valid only when `log_type` is `fullregex_log`.
      beginningRegex: xxx                     ## The line-start regular expression matching rule. This parameter is valid only when `log_type` is `multiline_log` or `fullregex_log`.
      keys:                                   ## The key name for each field. A key left empty indicates that the field is to be discarded. This parameter is valid only when `log_type` is `delimiter_log`.
      - a
      - b
      filterKeys:                             ## Must correspond to `filterRegex`. `filterKeys` is an array that specifies the keys of the logs to be filtered.
      - a
      - b
      filterRegex:                            ## Must correspond to `filterKeys`. `filterRegex` is an array of regular expression strings, representing the filtering rules regex corresponding to the keys.
      - aRegx
      - bRegx
      advancedFilters:                       ## Advanced filtering rules. This field is applicable only to collector component versions v1.1.15 and above. For versions below v1.1.15, use `filterKeys` and `filterRegex`.
      - key: level                            ## The key to be filtered
        rule: 0                               ## Filtering rule. Supported numeric enumerations are: 0 (equals), 1 (field exists), 2 (field does not exist), 3 (not equals).
        value: info                           ## The value to be filtered. When `rule` is 1 or 2, `value` does not need to be specified.
      unMatchUpload: "false"                  ## Whether to upload logs that fail to parse. `true` indicates upload, `false` indicates no upload. The default is no upload.
      unMatchedKey: "parseFailed"             ## The key name for uploading logs that fail to parse to CLS.
      backtracking: "-1"                      ## "-1" or "0". "-1" indicates full collection, "0" indicates incremental collection. Any number greater than 0 indicates the amount of data to be backtracked in incremental collection mode. Full collection is the default.
      isGBK: "true"                           ## "true" or "false". Default value: "false".
      jsonStandard: "true"                    ## "true" or "false". Default value: "true".
      
    ### Advanced Collection Configuration
    advancedConfig:                           ## Optional
      ClsAgentMaxDepth: 1                     ## The maximum directory depth for the collector to scan. The value must be an integer greater than or equal to 0. A value of 0 indicates the current directory.
      ClsAgentFileTimeout: 60                 ## The file timeout attribute. If a log file has not been modified for longer than the specified time, the collector stops monitoring it. This parameter must be an integer greater than or equal to 0, in seconds. A value of 0 indicates no timeout, meaning all files are collected.
      ClsAgentParseFailMerge: true            ## Merge logs that fail to parse. Valid values: true or false.

    ### Customize Full-Text Index When Creating a topic
    ### This feature is supported only when a log topic is created automatically. It does not support modifying the index of an existing log topic.
    fullTextIndex:                            ## Optional. Full-text index settings.  
       status: "on"                           ## The full-text index switch. If not set, the full-text index is enabled by default. If set to "on" (enabled) and other parameters are not set, caseSensitive and tokenizer will use their default values. If set to "off" (disabled), the full-text index is not enabled.
       tokenizer: "@&()='\\",;:<>[]{}/ \\n\\t\\r" ## The delimiter for full-text indexing. If the full-text index switch is enabled, you must set tokenizer. It can be set to "@&()='\\",;:<>[]{}/ \\n\\t\\r", which is the default setting.
       caseSensitive: false                   ## Whether case-sensitive
       containZH: false                       ## Whether to include Chinese
    
    ### Customize TAG and Key-Value Index Settings When Creating a topic
    indexStatus: "on"                          ## Optional. The switch for TAG and key-value indexing. If not set, the system creates three TAG indexes (pod_name/namespace/container_name) by default. A value of "off" indicates that key-value indexing is not configured.
    autoIndex: "off"                          ## Optional. A value of "off" indicates that automatic index configuration is not set. A value of "on" indicates that automatic index configuration is enabled.
    indexs:                                   ## Optional.
    - indexName: xxx                          ## The field for which you need to configure key-value indexing or TAG indexing. The eight metadata fields related to Pods (namespace/pod_name/pod_ip/pod_uid/container_id/container_name/image_name/cluster_id) and fields with the pod_label_ prefix are set as TAG indexes. Other fields are set as key-value indexes.
      indexType: "text"                       ## Optional. The field type. Currently supported types are: long, text, double. Optional.
      tokenizer: "@&()='\\",;:<>[]{}/ \\n\\t\\r"  ## Optional. The delimiter for the field. Each character in the string represents a delimiter. Only English symbols and \\n\\t\\r are supported. For fields of type long or double, this value must be left empty.
                                              ## For fields of type text, it is recommended to use @&?|#()='\\",;:<>[]{}/ \\n\\t\\r\\ as the delimiter.
      sqlFlag: true                           ## Optional. Whether to enable the analysis feature for the field. true/false indicates whether to enable the analysis feature. If the log topic is a STANDARD_IA log topic, the sqlFlag option cannot be enabled.
      containZH: false                        ## Optional. Whether to include Chinese.

Log Collection Rule Configuration Example

Single-Line Full-Text Format
Multi-line Full-Text Format
Single-Line Full Regular Expression Format
Multi-line Full Regular Expression Format
JSON Format
Delimiter format
Combined Parsing
A single-line full-text log means that the content of one line is a complete log. During log collection, CLS uses the line break \\n as the end identifier for a log. For unified structured management, each log contains a default key-value pair \\_\\_CONTENT\\_\\_. However, the log data itself is no longer processed for log structuring, and no log fields are extracted. The time attribute of the log is determined by the time of log collection.
Assume that the raw data of a log is:
Tue Jan 22 12:08:15 CST 2019 Installed: libjpeg-turbo-static-1.2.90-6.el7.x86_64
A sample LogConfig is as follows:
apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig                   
metadata:
  name: test                                  ## The name of the collection configuration.
spec:
  clsDetail:
    topicId: xxxxxx-xx-xx-xx-xxxxxxxx
    # Single-line Log
    logType: minimalist_log
The data collected into CLS is:
__CONTENT__:Tue Jan 22 12:08:15 CST 2019 Installed: libjpeg-turbo-static-1.2.90-6.el7.x86_64
A multi-line full-text log represents that the content of a complete log may span multiple lines, for example, Java stack trace. In this case, it is somewhat unreasonable to use the line break \\n as the end identifier of the log. In order to enable the logging system to clearly distinguish each log, the first-line regular expression method is adopted for matching. If a certain line of a log matches the preset regular expression, it is considered as the beginning of the log, and the beginning of the next line appears as the end identifier of the log.
A multi-line full-text log also contains a default key-value pair \\_\\_CONTENT\\_\\_. However, the log data itself is no longer processed for log structuring, and no log fields are extracted. The time attribute of the log is determined by the time of log collection.
Assume that the raw data of a multi-line log is:
2019-12-15 17:13:06,043 [main] ERROR com.test.logging.FooFactory:
java.lang.NullPointerException
    at com.test.logging.FooFactory.createFoo(FooFactory.java:15)
    at com.test.logging.FooFactoryTest.test(FooFactoryTest.java:11)
A sample of LogConfig is as follows:
apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig                   
metadata:
  name: test                                  ## The name of the collection configuration.
spec:
  clsDetail:
    topicId: xxxxxx-xx-xx-xx-xxxxxxxx
    # Multi-line Log
    logType: multiline_log
    extractRule:
      # Only a line that begins with a date and time is recognized as the start of a new log entry. Otherwise, a newline character \\n is appended to the current log entry.
      beginningRegex: \\d{4}-\\d{2}-\\d{2}\\s\\d{2}:\\d{2}:\\d{2},\\d{3}\\s.+
The data collected into CLS is:
__CONTENT__:2019-12-15 17:13:06,043 [main] ERROR com.test.logging.FooFactory:\\njava.lang.NullPointerException\\n    at com.test.logging.FooFactory.createFoo(FooFactory.java:15)\\n    at com.test.logging.FooFactoryTest.test(FooFactoryTest.java:11)
The single-line full regular expression format is usually used to process structured logs. This represents a log parsing mode in which multiple key-value pairs are extracted from a complete log entry using regular expressions.
Assume that the raw data of a log is:
10.135.46.111 - - [22/Jan/2019:19:19:30 +0800] "GET /my/course/1 HTTP/1.1" 127.0.0.1 200 782 9703 "http://127.0.0.1/course/explore?filter%5Btype%5D=all&filter%5Bprice%5D=all&filter%5BcurrentLevelId%5D=all&orderBy=studentNum" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0"  0.354 0.354
A sample of LogConfig is as follows:
apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig                   
metadata:
  name: test                                  ## The name of the collection configuration.
spec:
  clsDetail:
    topicId: xxxxxx-xx-xx-xx-xxxxxxxx
    # Full Regular Expression Format
    logType: fullregex_log
    extractRule:
      # Regular Expression: Values are extracted based on the () capture groups.
      logRegex: (\\S+)[^\\[]+(\\[[^:]+:\\d+:\\d+:\\d+\\s\\S+)\\s"(\\w+)\\s(\\S+)\\s([^"]+)"\\s(\\S+)\\s(\\d+)\\s(\\d+)\\s(\\d+)\\s"([^"]+)"\\s"([^"]+)"\\s+(\\S+)\\s(\\S+).*
      beginningRegex: (\\S+)[^\\[]+(\\[[^:]+:\\d+:\\d+:\\d+\\s\\S+)\\s"(\\w+)\\s(\\S+)\\s([^"]+)"\\s(\\S+)\\s(\\d+)\\s(\\d+)\\s(\\d+)\\s"([^"]+)"\\s"([^"]+)"\\s+(\\S+)\\s(\\S+).*
      # List of extracted keys, which correspond one-to-one with the extracted values.
      keys:  ['remote_addr','time_local','request_method','request_url','http_protocol','http_host','status','request_length','body_bytes_sent','http_referer','http_user_agent','request_time','upstream_response_time']
The data collected into CLS is:
body_bytes_sent: 9703
http_host: 127.0.0.1
http_protocol: HTTP/1.1
http_referer: http://127.0.0.1/course/explore?filter%5Btype%5D=all&filter%5Bprice%5D=all&filter%5BcurrentLevelId%5D=all&orderBy=studentNum
http_user_agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0
remote_addr: 10.135.46.111
request_length: 782
request_method: GET
request_time: 0.354
request_url: /my/course/1
status: 200
time_local: [22/Jan/2019:19:19:30 +0800]
upstream_response_time: 0.354
The Multi-line Full Regular Expression mode is a log parsing mode suitable for log text where a complete log entry spans multiple lines (for example, Java program logs). It extracts multiple key-value pairs using regular expressions. If you do not need to extract key-value pairs, refer to the Multi-line Full Text Format for configuration.
Assume that the raw data of a log is:
[2018-10-01T10:30:01,000] [INFO] java.lang.Exception: exception happened
   at TestPrintStackTrace.f(TestPrintStackTrace.java:3)
   at TestPrintStackTrace.g(TestPrintStackTrace.java:7)
   at TestPrintStackTrace.main(TestPrintStackTrace.java:16)
A sample of LogConfig is as follows:
apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig                   
metadata:
  name: test                                  ## The name of the collection configuration.
spec: 
  clsDetail: 
    topicId: xxxxxx-xx-xx-xx-xxxxxxxx
        # Multi-line Full Regular Expression Format
        logType: multiline_fullregex_log
        extractRule: 
          # Full Regular Expression at Line Start: Only a line that begins with a date and time is recognized as the start of a new log entry. Otherwise, a newline character \\n is appended to the current log entry.
          beginningRegex: \\[\\d+-\\d+-\\w+:\\d+:\\d+,\\d+\\]\\s\\[\\w+\\]\\s.*
          # Regular Expression: Values are extracted based on the () capture groups.
          logRegex: \\[(\\d+-\\d+-\\w+:\\d+:\\d+,\\d+)\\]\\s\\[(\\w+)\\]\\s(.*)
          # List of extracted keys, which correspond one-to-one with the extracted values.
          keys: ['time','level','msg']
According to the extracted key, the collected data of CLS is:
time: 2018-10-01T10:30:01,000`
level: INFO`
msg:java.lang.Exception: exception happened
   at TestPrintStackTrace.f(TestPrintStackTrace.java:3)
   at TestPrintStackTrace.g(TestPrintStackTrace.java:7)
   at TestPrintStackTrace.main(TestPrintStackTrace.java:16)
JSON format logs automatically extract the first-level key as the field name and the first-level value as the field value. The entire log will be structured in this way, and each complete log will end with a line break character \\n.
Assume that the raw data of a JSON log is:
{"remote_ip":"10.135.46.111","time_local":"22/Jan/2019:19:19:34 +0800","body_sent":23,"responsetime":0.232,"upstreamtime":"0.232","upstreamhost":"unix:/tmp/php-cgi.sock","http_host":"127.0.0.1","method":"POST","url":"/event/dispatch","request":"POST /event/dispatch HTTP/1.1","xff":"-","referer":"http://127.0.0.1/my/course/4","agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0","response_code":"200"}
A sample of LogConfig is as follows:
apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig                   
metadata:
  name: test                                  ## The name of the collection configuration.
spec:
  clsDetail:
    topicId: xxxxxx-xx-xx-xx-xxxxxxxx
    # JSON Format Logs
    logType: json_log
The data collected into CLS is:
agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0
body_sent: 23
http_host: 127.0.0.1
method: POST
referer: http://127.0.0.1/my/course/4
remote_ip: 10.135.46.111
request: POST /event/dispatch HTTP/1.1
response_code: 200
responsetime: 0.232
time_local: 22/Jan/2019:19:19:34 +0800
upstreamhost: unix:/tmp/php-cgi.sock
upstreamtime: 0.232
url: /event/dispatch
xff: -
Delimiter-based logs refer to log data that can be structured by using a specified delimiter to process the entire log entry, with each complete log ending with the newline character \\n. When CLS processes logs in delimiter format, you must define a unique key for each separated field.
Assume that the raw data of a log is:
10.20.20.10 ::: [Tue Jan 22 14:49:45 CST 2019 +0800] ::: GET /online/sample HTTP/1.1 ::: 127.0.0.1 ::: 200 ::: 647 ::: 35 ::: http://127.0.0.1/
A sample of LogConfig is as follows:
apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig                   
metadata:
  name: test                                  ## The name of the collection configuration.
spec:
  clsDetail:
    topicId: xxxxxx-xx-xx-xx-xxxxxxxx
    # Separator log
    logType: delimiter_log
    extractRule:
      # Separator
      delimiter: ':::'
      # List of extracted keys, which correspond one-to-one with the split fields.
      keys: ['IP','time','request','host','status','length','bytes','referer']
The data collected into CLS is:
IP: 10.20.20.10
bytes: 35
host: 127.0.0.1
length: 647
referer: http://127.0.0.1/
request: GET /online/sample HTTP/1.1
status: 200
time: [Tue Jan 22 14:49:45 CST 2019 +0800]
Assume that the raw data of a log is as follows:
1571394459,http://127.0.0.1/my/course/4|10.135.46.111|200,status:DEAD,
The custom extension content is as follows:
{
 "processors": [
   {
     "type": "processor_split_delimiter",
     "detail": {
       "Delimiter": ",",
       "ExtractKeys": [ "time", "msg1","msg2"]
     },
     "processors": [
       {
         "type": "processor_timeformat",
         "detail": {
           "KeepSource": true,
           "TimeFormat": "%s",
           "SourceKey": "time"
          }
       },
       {
         "type": "processor_split_delimiter",
         "detail": {
           "KeepSource": false,
           "Delimiter": "|",
           "SourceKey": "msg1",
           "ExtractKeys": [ "submsg1","submsg2","submsg3"]
          },
          "processors": []
       },
       {
         "type": "processor_split_key_value",
         "detail": {
           "KeepSource": false,
           "Delimiter": ":",
           "SourceKey": "msg2"
         }
       }
     ]
   }
 ]
}
A sample of LogConfig is as follows:
apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig                   
metadata:
  name: test                                  ## The name of the collection configuration.
spec:
  clsDetail:
    topicId: xxxxxx-xx-xx-xx-xxxxxxxx
    # Separator log
    logType: delimiter_log
    userDefineRule:"{ \\"processors\\": [ { \\"type\\": \\"processor_split_delimiter\\", \\"detail\\": { \\"Delimiter\\": \\",\\", \\"ExtractKeys\\": [ \\"time\\", \\"msg1\\",\\"msg2\\"] }, \\"processors\\": [ { \\"type\\": \\"processor_timeformat\\", \\"detail\\": { \\"KeepSource\\": true, \\"TimeFormat\\": \\"%s\\", \\"SourceKey\\": \\"time\\" } }, { \\"type\\": \\"processor_split_delimiter\\", \\"detail\\": { \\"KeepSource\\": false, \\"Delimiter\\": \\"|\\", \\"SourceKey\\": \\"msg1\\", \\"ExtractKeys\\": [ \\"submsg1\\",\\"submsg2\\",\\"submsg3\\"] }, \\"processors\\": [] }, { \\"type\\": \\"processor_split_key_value\\", \\"detail\\": { \\"KeepSource\\": false, \\"Delimiter\\": \\":\\", \\"SourceKey\\": \\"msg2\\" } } ] } ] }"
The data collected to CLS is as follows:
time: [Tue Jan 22 14:49:45 CST 2019 +0800]
referer: http://127.0.0.1/my/course/4
remote_ip: 10.135.46.111
response_code: 200
status: 200

inputDetail (log source) field description

  inputDetail:
    type: container_stdout   			    	## Log collection type. Valid values: `container_stdout` (container standard output); `container_file` (container file); `host_file` (host file).
    containerStdout:        				 	## Container standard output configuration, which is valid only if `type` is `container_stdout`.
      namespace: default   			 	 	## The Kubernetes namespace of the container to be collected. Separate multiple namespaces by `,`, for example, `default,namespace`. If this field is not specified, it indicates all namespaces. Note that this field cannot be specified if `excludeNamespace` is specified.
      excludeNamespace: nm1,nm2   		   	## The Kubernetes namespace of the container to be excluded. Separate multiple namespaces by `,`, for example, `nm1,nm2`. If this field is not specified, it indicates all namespaces. Note that this field cannot be specified if `namespace` is specified.
      nsLabelSelector: environment in (production),tier in (frontend) ## The namespace label for filtering namespaces
      allContainers: false      			 	## Whether to collect the standard output of all containers in the specified namespace. Note that if `allContainers=true`, you cannot specify `workload`, `includeLabels`, and `excludeLabels` at the same time.
      containerOperator: in                      ## Container selection method. Valid values: `in` (include); `not in` (exclude).
      container: xxx             				## The name of the container to be or not to be collected
      includeLabels:                             ## Pods with the specified labels will be collected. This field cannot be specified if `workload` is specified.
        key: value1   					   	## Pods with multiple values of the same key can be matched. For example, if you enter `environment = production,qa`, Pods with the `production` or `qa` value of the `environment` key will be matched. Separate multiple values by comma. If `excludeLabels` is also specified, Pods in the intersection will be matched.
      excludeLabels:  					   	## The labels of the Pods to be excluded. This field cannot be specified if `workload`, `namespace`, and `excludeNamespace` are specified.
        key2: value2 							## Pods with multiple values of the same key can be matched. For example, if you enter `environment = production,qa`, Pods with the `production` or `qa` value of the `environment` key will be excluded. Separate multiple values by comma. If `includeLabels` is also specified, Pods in the intersection will be matched.
      metadataLabels:        		        	## The Pod labels to be collected as metadata. If this field is not specified, all Pod labels will be collected as metadata.
      - label1
      metadataContainer:				     	## The container environments of the metadata to be collected. If this field is not specified, metadata (`namespace`, `pod_name`, `pod_ip`, `pod_uid`, `container_id`, `container_name`, and `image_name`) of all container environments will be collected.
      - namespace
      customLabels:          					## Custom metadata
        label: l1
      workloads:						 		## The workloads of the specified workload types in the specified namespaces of the containers of the logs to be collected
      - container: xxx    			   		## The name of the container to be collected. If this field is not specified, all containers in the workload Pod will be collected.
        containerOperator: in                      ## Container selection method. Valid values: `in` (include); `not in` (exclude).
        kind: deployment  				   	## Workload type. Valid values: `deployment`, `daemonset`, `statefulset`, `job`, `cronjob`.
        name: sample-app  					   ## Workload name
        namespace: prod					  	## Workload namespace
        
    containerFile:  				 			## Container file configuration, which is valid only if `type` is `container_file`.
      namespace: default      			   	## The Kubernetes namespace of the container to be collected. You must specify a namespace.      
      excludeNamespace: nm1,nm2   	       	## The Kubernetes namespace of the container to be excluded. Separate multiple namespaces by `,`, for example, `nm1,nm2`. If this field is not specified, it indicates all namespaces. Note that this field cannot be specified if `namespace` is specified.
      nsLabelSelector: environment in (production),tier in (frontend) ## The namespace label for filtering namespaces
      containerOperator: in                      ## container selection method. Valid values: `in` (include); `not in` (exclude).
      container: xxx                             ## The name of the container to be or not to be collected. Multiple names are supported and must be separated by commas. If `containerOperator` is `in`, you can enter `*` to collect logs from all containers. If a workload type and name are specified, you can define the container name within that specific workload here.
      logPath: /var/logs      			   	## Log folder. Wildcards are not supported.
      filePattern: app_*.log 					## Log filename. Wildcards `*` and `?` are supported. `*` indicates to match any number of characters, while `?` indicates to match any single character.
      
      filePaths: /var/log/app1.log,/var/log/app2.log  ## Specifies multiple log paths and files within the container. This parameter is incompatible with `logPath` and `filePattern`. You can specify only `filePaths` or (`logPath` and `filePattern`).
      metadataLabels:        		        	## The Pod labels to be collected as metadata. If this field is not specified, all Pod labels will be collected as metadata.
      filePaths:                                
      - file: /var/logs                          ## Log folder. Wildcards `*` and `?` are supported. `*` indicates to match any number of characters, while `?` indicates to match any single character.
        path: app_*.log                          ## Log filename. Wildcards `*` and `?` are supported. `*` indicates to match any number of characters, while `?` indicates to match any single character.
      - file: /var/logs1
        path: app_*.log
      
      includeLabels:                             ## Pods with the specified labels will be collected. This field cannot be specified if `workload` is specified.
        key: value1                              ## Logs collected by the collection rule will be attached with metadata and reported to the consumer. Pods with multiple values of the same key can be matched. For example, if you enter `environment = production,qa`, Pods with the `production` or `qa` value of the `environment` key will be matched. Separate multiple values by comma. If `excludeLabels` is also specified, Pods in the intersection will be matched.
      excludeLabels:  					   	## Pods with the specified labels will be excluded. This field cannot be specified if `workload` is specified.
        key2: value2 							## Pods with multiple values of the same key can be matched. For example, if you enter `environment = production,qa`, Pods with the `production` or `qa` value of the `environment` key will be excluded. Separate multiple values by comma. If `includeLabels` is also specified, Pods in the intersection will be matched.
      metadataLabels:        		        	## The Pod labels to be collected as metadata. If this field is not specified, all Pod labels will be collected as metadata.
      - namespace
      metadataContainer:				     	## The container environments of the metadata to be collected. If this field is not specified, metadata (`namespace`, `pod_name`, `pod_ip`, `pod_uid`, `container_id`, `container_name`, and `image_name`) of all container environments will be collected.
      customLabels:          					## Custom metadata
        key: value
      workload:
        kind: deployment  				   	## Workload type. Valid values: `deployment`, `daemonset`, `statefulset`, `job`, `cronjob`.
        name: sample-app  					   ## Workload name
        namespace: prod					  	## Workload namespace

    hostFile:                					## Node file path, which is valid only if `type` is `host_file`.
      filePattern: '*.log'   					## Log filename. Wildcards `*` and `?` are supported. `*` indicates to match any number of characters, while `?` indicates to match any single character.
      logPath: /tmp/logs     					## Log folder. Wildcards are not supported.
      
      filePaths: /var/log/app1.log,/var/log/app2.log  ## Specifies multiple log paths and files within the container. This parameter is incompatible with `logPath` and `filePattern`. You can specify only `filePaths` or (`logPath` and `filePattern`).
      metadataLabels:        		        	## The Pod labels to be collected as metadata. If this field is not specified, all Pod labels will be collected as metadata.
      filePaths:                                
      - file: /var/logs                          ## Log folder. Wildcards `*` and `?` are supported. `*` indicates to match any number of characters, while `?` indicates to match any single character.
        path: app_*.log                          ## Log filename. Wildcards `*` and `?` are supported. `*` indicates to match any number of characters, while `?` indicates to match any single character.
      - file: /var/logs1
        path: app_*.log
      
      customLabels:          					## Custom metadata
        label1: v1

Log source configuration sample

Standard container output
Container file
Host file
Sample 1: collecting the standard output of all containers in the default namespace
apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig                   
metadata:
  name: test                                  ## The name of the collection configuration.
spec:
  inputDetail:
    type: container_file
    containerFile:
      namespace: default
      allContainers: true
 ...
Sample 2: collecting the container standard output in the Pod that belongs to ingress-gateway deployment in the production namespace
apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec:
  inputDetail:
    type: container_file
    containerFile:
      allContainers: false
      workloads:
      - namespace: production
        name: ingress-gateway
        kind: deployment
  ...
Sample 3: collecting the container standard output in the Pod whose Pod labels contain "k8s-app=nginx" in the production namespace
apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec:
  inputDetail:
    type: container_file
    containerFile:
      namespace: production
      allContainers: false
      includeLabels:
        k8s-app: nginx
  ...
Sample 4: collecting the standard output of the container named `ingress` in the pod that belongs to the `ingress-gateway` workload in the `production` namespace.
apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec:
  inputDetail:
    type: container_file
    containerFile:
      allContainers: false
      containerOperator: in
      container: xxx    			   		## The name of the container to be collected. If this field is not specified, all containers in the workload Pod will be collected.
      workloads:
      - namespace: production
        name: ingress-gateway
        kind: deployment
  ...
Sample 1: collecting the access.log file in the /data/nginx/log/ path in the NGINX container in the Pod that belongs to ingress-gateway deployment in the production namespace
apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig                   
metadata:
  name: test                                  ## The name of the collection configuration.
spec:
  inputDetail:
    type: container_file
    containerFile:
      namespace: production
      workload:
        name: ingress-gateway
        kind: deployment
      container: nginx
      filePaths:                                
      filePattern: access.log
        logPath: /data/nginx/log
  ...
Sample 2: collecting the access.log file in the /data/nginx/log/ path in the NGINX container in the Pod whose pod labels contain "k8s-app=ingress-gateway" in the production namespace
apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec:
  inputDetail:
    type: container_file
    containerFile:
      namespace: production
      includeLabels:
        k8s-app: ingress-gateway
      container: nginx
      filePaths:                                
      filePattern: access.log
        logPath: /data/nginx/log
  ...
Sample: collecting all .log files in the host path /data/
apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig                   
metadata:
  name: test                                  ## The name of the collection configuration.
spec:
  inputDetail:
    type: host_file
    hostFile:
      logPath: /data
      filePattern: *.log
  ...

Step 3. Create a LogConfig object

As the LogConfig.yaml declaration file is defined in Step 2. Define the LogConfig object, you can run the kubectl command to create a LogConfig object based on the file.
kubectl create -f /usr/local/LogConfig.yaml

Step 4: Viewing LogConfig 

kubectl get logconfig

Related Operations

After the deployment for cluster log collection is completed, you can go to CLS console > Search and Analysis to view collected logs.
Previous Topic: Configuring Log Collection in Self-Built Kubernetes Cluster in the ConsoleNext Topic: Collecting TKE Kubernetes Cluster Logs

Help and Support

Was this page helpful?

Help us improve! Rate your documentation experience in 5 mins.

Feedback