- Release Notes and Announcements
- Release Notes
- Announcements
- qGPU Service Adjustment
- Version Upgrade of Master Add-On of TKE Managed Cluster
- Upgrading tke-monitor-agent
- Discontinuing TKE API 2.0
- Instructions on Cluster Resource Quota Adjustment
- Discontinuing Kubernetes v1.14 and Earlier Versions
- Deactivation of Scaling Group Feature
- Notice on TPS Discontinuation on May 16, 2022 at 10:00 (UTC +8)
- Basic Monitoring Architecture Upgrade
- Starting Charging on Managed Clusters
- Instructions on Stopping Delivering the Kubeconfig File to Nodes
- Security Vulnerability Fix Description
- Release Notes
- Product Introduction
- Purchase Guide
- Quick Start
- TKE General Cluster Guide
- TKE General Cluster Overview
- Purchase a TKE General Cluster
- High-risk Operations of Container Service
- Deploying Containerized Applications in the Cloud
- Kubernetes API Operation Guide
- Open Source Components
- Permission Management
- Cluster Management
- Cluster Overview
- Cluster Hosting Modes Introduction
- Cluster Lifecycle
- Creating a Cluster
- Deleting a Cluster
- Cluster Scaling
- Changing the Cluster Operating System
- Connecting to a Cluster
- Upgrading a Cluster
- Enabling IPVS for a Cluster
- Enabling GPU Scheduling for a Cluster
- Custom Kubernetes Component Launch Parameters
- Using KMS for Kubernetes Data Source Encryption
- Images
- Worker node introduction
- Normal Node Management
- Native Node Management
- Overview
- Purchasing Native Nodes
- Lifecycle of a Native Node
- Native Node Parameters
- Creating Native Nodes
- Deleting Native Nodes
- Self-Heal Rules
- Declarative Operation Practice
- Native Node Scaling
- In-place Pod Configuration Adjustment
- Enabling SSH Key Login for a Native Node
- Management Parameters
- Enabling Public Network Access for a Native Node
- Supernode management
- Registered Node Management
- GPU Share
- Kubernetes Object Management
- Overview
- Namespace
- Workload
- Deployment Management
- StatefulSet Management
- DaemonSet Management
- Job Management
- CronJob Management
- Setting the Resource Limit of Workload
- Setting the Scheduling Rule for a Workload
- Setting the Health Check for a Workload
- Setting the Run Command and Parameter for a Workload
- Using a Container Image in a TCR Enterprise Instance to Create a Workload
- Auto Scaling
- Configuration
- Register node management
- Service Management
- Ingress Management
- Storage Management
- Application and Add-On Feature Management Description
- Add-On Management
- Add-on Overview
- Add-On Lifecycle Management
- CBS-CSI Description
- UserGroupAccessControl
- COS-CSI
- CFS-CSI
- P2P
- OOMGuard
- TCR Introduction
- TCR Hosts Updater
- DNSAutoscaler
- NodeProblemDetectorPlus Add-on
- NodeLocalDNSCache
- Network Policy
- DynamicScheduler
- DeScheduler
- Nginx-ingress
- HPC
- Description of tke-monitor-agent
- GPU-Manager Add-on
- CFSTURBO-CSI
- tke-log-agent
- Helm Application
- Application Market
- Network Management
- Container Network Overview
- GlobalRouter Mode
- VPC-CNI Mode
- VPC-CNI Mode
- Multiple Pods with Shared ENI Mode
- Pods with Exclusive ENI Mode
- Static IP Address Mode Instructions
- Non-static IP Address Mode Instructions
- Interconnection Between VPC-CNI and Other Cloud Resources/IDC Resources
- Security Group of VPC-CNI Mode
- Instructions on Binding an EIP to a Pod
- VPC-CNI Component Description
- Limits on the Number of Pods in VPC-CNI Mode
- Cilium-Overlay Mode
- OPS Center
- Log Management
- Backup Center
- Cloud Native Monitoring
- Remote Terminals
- TKE Serverless Cluster Guide
- TKE Edge Cluster Guide
- TKE Registered Cluster Guide
- TKE Container Instance Guide
- Cloud Native Service Guide
- Best Practices
- Cluster
- Cluster Migration
- Serverless Cluster
- Edge Cluster
- Security
- Service Deployment
- Hybrid Cloud
- Network
- DNS
- Using Network Policy for Network Access Control
- Deploying NGINX Ingress on TKE
- Nginx Ingress High-Concurrency Practices
- Nginx Ingress Best Practices
- Limiting the bandwidth on pods in TKE
- Directly connecting TKE to the CLB of pods based on the ENI
- Use CLB-Pod Direct Connection on TKE
- Obtaining the Real Client Source IP in TKE
- Using Traefik Ingress in TKE
- Release
- Logs
- Monitoring
- OPS
- Removing and Re-adding Nodes from and to Cluster
- Using Ansible to Batch Operate TKE Nodes
- Using Cluster Audit for Troubleshooting
- Renewing a TKE Ingress Certificate
- Using cert-manager to Issue Free Certificates
- Using cert-manager to Issue Free Certificate for DNSPod Domain Name
- Using the TKE NPDPlus Plug-In to Enhance the Self-Healing Capability of Nodes
- Using kubecm to Manage Multiple Clusters kubeconfig
- Quick Troubleshooting Using TKE Audit and Event Services
- Customizing RBAC Authorization in TKE
- Clearing De-registered Tencent Cloud Account Resources
- Terraform
- DevOps
- Auto Scaling
- Cluster Auto Scaling Practices
- Using tke-autoscaling-placeholder to Implement Auto Scaling in Seconds
- Installing metrics-server on TKE
- Using Custom Metrics for Auto Scaling in TKE
- Utilizing HPA to Auto Scale Businesses on TKE
- Using VPA to Realize Pod Scaling up and Scaling down in TKE
- Adjusting HPA Scaling Sensitivity Based on Different Business Scenarios
- Storage
- Containerization
- Microservice
- Cost Management
- Fault Handling
- Disk Full
- High Workload
- Memory Fragmentation
- Cluster DNS Troubleshooting
- Cluster kube-proxy Troubleshooting
- Cluster API Server Inaccessibility Troubleshooting
- Service and Ingress Inaccessibility Troubleshooting
- Troubleshooting for Pod Network Inaccessibility
- Pod Status Exception and Handling
- Authorizing Tencent Cloud OPS Team for Troubleshooting
- Engel Ingres appears in Connechtin Reverside
- CLB Loopback
- CLB Ingress Creation Error
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- Cluster APIs
- DescribeEncryptionStatus
- DisableEncryptionProtection
- EnableEncryptionProtection
- AcquireClusterAdminRole
- CreateClusterEndpoint
- CreateClusterEndpointVip
- DeleteCluster
- DeleteClusterEndpoint
- DeleteClusterEndpointVip
- DescribeAvailableClusterVersion
- DescribeClusterAuthenticationOptions
- DescribeClusterCommonNames
- DescribeClusterEndpointStatus
- DescribeClusterEndpointVipStatus
- DescribeClusterEndpoints
- DescribeClusterKubeconfig
- DescribeClusterLevelAttribute
- DescribeClusterLevelChangeRecords
- DescribeClusterSecurity
- DescribeClusterStatus
- DescribeClusters
- DescribeEdgeAvailableExtraArgs
- DescribeEdgeClusterExtraArgs
- DescribeResourceUsage
- DisableClusterDeletionProtection
- EnableClusterDeletionProtection
- GetClusterLevelPrice
- GetUpgradeInstanceProgress
- ModifyClusterAttribute
- ModifyClusterAuthenticationOptions
- ModifyClusterEndpointSP
- UpgradeClusterInstances
- CreateCluster
- UpdateClusterVersion
- UpdateClusterKubeconfig
- DescribeBackupStorageLocations
- DeleteBackupStorageLocation
- CreateBackupStorageLocation
- Add-on APIs
- Network APIs
- Node APIs
- Node Pool APIs
- TKE Edge Cluster APIs
- DescribeTKEEdgeScript
- DescribeTKEEdgeExternalKubeconfig
- DescribeTKEEdgeClusters
- DescribeTKEEdgeClusterStatus
- DescribeTKEEdgeClusterCredential
- DescribeEdgeClusterInstances
- DescribeEdgeCVMInstances
- DescribeECMInstances
- DescribeAvailableTKEEdgeVersion
- DeleteTKEEdgeCluster
- DeleteEdgeClusterInstances
- DeleteEdgeCVMInstances
- DeleteECMInstances
- CreateTKEEdgeCluster
- CreateECMInstances
- CheckEdgeClusterCIDR
- ForwardTKEEdgeApplicationRequestV3
- UninstallEdgeLogAgent
- InstallEdgeLogAgent
- DescribeEdgeLogSwitches
- CreateEdgeLogConfig
- CreateEdgeCVMInstances
- UpdateEdgeClusterVersion
- DescribeEdgeClusterUpgradeInfo
- Cloud Native Monitoring APIs
- Virtual node APIs
- Other APIs
- Scaling group APIs
- Data Types
- Error Codes
- API Mapping Guide
- TKE Insight
- TKE Scheduling
- FAQs
- Service Agreement
- Contact Us
- Purchase Channels
- Glossary
- User Guide(Old)
- Release Notes and Announcements
- Release Notes
- Announcements
- qGPU Service Adjustment
- Version Upgrade of Master Add-On of TKE Managed Cluster
- Upgrading tke-monitor-agent
- Discontinuing TKE API 2.0
- Instructions on Cluster Resource Quota Adjustment
- Discontinuing Kubernetes v1.14 and Earlier Versions
- Deactivation of Scaling Group Feature
- Notice on TPS Discontinuation on May 16, 2022 at 10:00 (UTC +8)
- Basic Monitoring Architecture Upgrade
- Starting Charging on Managed Clusters
- Instructions on Stopping Delivering the Kubeconfig File to Nodes
- Security Vulnerability Fix Description
- Release Notes
- Product Introduction
- Purchase Guide
- Quick Start
- TKE General Cluster Guide
- TKE General Cluster Overview
- Purchase a TKE General Cluster
- High-risk Operations of Container Service
- Deploying Containerized Applications in the Cloud
- Kubernetes API Operation Guide
- Open Source Components
- Permission Management
- Cluster Management
- Cluster Overview
- Cluster Hosting Modes Introduction
- Cluster Lifecycle
- Creating a Cluster
- Deleting a Cluster
- Cluster Scaling
- Changing the Cluster Operating System
- Connecting to a Cluster
- Upgrading a Cluster
- Enabling IPVS for a Cluster
- Enabling GPU Scheduling for a Cluster
- Custom Kubernetes Component Launch Parameters
- Using KMS for Kubernetes Data Source Encryption
- Images
- Worker node introduction
- Normal Node Management
- Native Node Management
- Overview
- Purchasing Native Nodes
- Lifecycle of a Native Node
- Native Node Parameters
- Creating Native Nodes
- Deleting Native Nodes
- Self-Heal Rules
- Declarative Operation Practice
- Native Node Scaling
- In-place Pod Configuration Adjustment
- Enabling SSH Key Login for a Native Node
- Management Parameters
- Enabling Public Network Access for a Native Node
- Supernode management
- Registered Node Management
- GPU Share
- Kubernetes Object Management
- Overview
- Namespace
- Workload
- Deployment Management
- StatefulSet Management
- DaemonSet Management
- Job Management
- CronJob Management
- Setting the Resource Limit of Workload
- Setting the Scheduling Rule for a Workload
- Setting the Health Check for a Workload
- Setting the Run Command and Parameter for a Workload
- Using a Container Image in a TCR Enterprise Instance to Create a Workload
- Auto Scaling
- Configuration
- Register node management
- Service Management
- Ingress Management
- Storage Management
- Application and Add-On Feature Management Description
- Add-On Management
- Add-on Overview
- Add-On Lifecycle Management
- CBS-CSI Description
- UserGroupAccessControl
- COS-CSI
- CFS-CSI
- P2P
- OOMGuard
- TCR Introduction
- TCR Hosts Updater
- DNSAutoscaler
- NodeProblemDetectorPlus Add-on
- NodeLocalDNSCache
- Network Policy
- DynamicScheduler
- DeScheduler
- Nginx-ingress
- HPC
- Description of tke-monitor-agent
- GPU-Manager Add-on
- CFSTURBO-CSI
- tke-log-agent
- Helm Application
- Application Market
- Network Management
- Container Network Overview
- GlobalRouter Mode
- VPC-CNI Mode
- VPC-CNI Mode
- Multiple Pods with Shared ENI Mode
- Pods with Exclusive ENI Mode
- Static IP Address Mode Instructions
- Non-static IP Address Mode Instructions
- Interconnection Between VPC-CNI and Other Cloud Resources/IDC Resources
- Security Group of VPC-CNI Mode
- Instructions on Binding an EIP to a Pod
- VPC-CNI Component Description
- Limits on the Number of Pods in VPC-CNI Mode
- Cilium-Overlay Mode
- OPS Center
- Log Management
- Backup Center
- Cloud Native Monitoring
- Remote Terminals
- TKE Serverless Cluster Guide
- TKE Edge Cluster Guide
- TKE Registered Cluster Guide
- TKE Container Instance Guide
- Cloud Native Service Guide
- Best Practices
- Cluster
- Cluster Migration
- Serverless Cluster
- Edge Cluster
- Security
- Service Deployment
- Hybrid Cloud
- Network
- DNS
- Using Network Policy for Network Access Control
- Deploying NGINX Ingress on TKE
- Nginx Ingress High-Concurrency Practices
- Nginx Ingress Best Practices
- Limiting the bandwidth on pods in TKE
- Directly connecting TKE to the CLB of pods based on the ENI
- Use CLB-Pod Direct Connection on TKE
- Obtaining the Real Client Source IP in TKE
- Using Traefik Ingress in TKE
- Release
- Logs
- Monitoring
- OPS
- Removing and Re-adding Nodes from and to Cluster
- Using Ansible to Batch Operate TKE Nodes
- Using Cluster Audit for Troubleshooting
- Renewing a TKE Ingress Certificate
- Using cert-manager to Issue Free Certificates
- Using cert-manager to Issue Free Certificate for DNSPod Domain Name
- Using the TKE NPDPlus Plug-In to Enhance the Self-Healing Capability of Nodes
- Using kubecm to Manage Multiple Clusters kubeconfig
- Quick Troubleshooting Using TKE Audit and Event Services
- Customizing RBAC Authorization in TKE
- Clearing De-registered Tencent Cloud Account Resources
- Terraform
- DevOps
- Auto Scaling
- Cluster Auto Scaling Practices
- Using tke-autoscaling-placeholder to Implement Auto Scaling in Seconds
- Installing metrics-server on TKE
- Using Custom Metrics for Auto Scaling in TKE
- Utilizing HPA to Auto Scale Businesses on TKE
- Using VPA to Realize Pod Scaling up and Scaling down in TKE
- Adjusting HPA Scaling Sensitivity Based on Different Business Scenarios
- Storage
- Containerization
- Microservice
- Cost Management
- Fault Handling
- Disk Full
- High Workload
- Memory Fragmentation
- Cluster DNS Troubleshooting
- Cluster kube-proxy Troubleshooting
- Cluster API Server Inaccessibility Troubleshooting
- Service and Ingress Inaccessibility Troubleshooting
- Troubleshooting for Pod Network Inaccessibility
- Pod Status Exception and Handling
- Authorizing Tencent Cloud OPS Team for Troubleshooting
- Engel Ingres appears in Connechtin Reverside
- CLB Loopback
- CLB Ingress Creation Error
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- Cluster APIs
- DescribeEncryptionStatus
- DisableEncryptionProtection
- EnableEncryptionProtection
- AcquireClusterAdminRole
- CreateClusterEndpoint
- CreateClusterEndpointVip
- DeleteCluster
- DeleteClusterEndpoint
- DeleteClusterEndpointVip
- DescribeAvailableClusterVersion
- DescribeClusterAuthenticationOptions
- DescribeClusterCommonNames
- DescribeClusterEndpointStatus
- DescribeClusterEndpointVipStatus
- DescribeClusterEndpoints
- DescribeClusterKubeconfig
- DescribeClusterLevelAttribute
- DescribeClusterLevelChangeRecords
- DescribeClusterSecurity
- DescribeClusterStatus
- DescribeClusters
- DescribeEdgeAvailableExtraArgs
- DescribeEdgeClusterExtraArgs
- DescribeResourceUsage
- DisableClusterDeletionProtection
- EnableClusterDeletionProtection
- GetClusterLevelPrice
- GetUpgradeInstanceProgress
- ModifyClusterAttribute
- ModifyClusterAuthenticationOptions
- ModifyClusterEndpointSP
- UpgradeClusterInstances
- CreateCluster
- UpdateClusterVersion
- UpdateClusterKubeconfig
- DescribeBackupStorageLocations
- DeleteBackupStorageLocation
- CreateBackupStorageLocation
- Add-on APIs
- Network APIs
- Node APIs
- Node Pool APIs
- TKE Edge Cluster APIs
- DescribeTKEEdgeScript
- DescribeTKEEdgeExternalKubeconfig
- DescribeTKEEdgeClusters
- DescribeTKEEdgeClusterStatus
- DescribeTKEEdgeClusterCredential
- DescribeEdgeClusterInstances
- DescribeEdgeCVMInstances
- DescribeECMInstances
- DescribeAvailableTKEEdgeVersion
- DeleteTKEEdgeCluster
- DeleteEdgeClusterInstances
- DeleteEdgeCVMInstances
- DeleteECMInstances
- CreateTKEEdgeCluster
- CreateECMInstances
- CheckEdgeClusterCIDR
- ForwardTKEEdgeApplicationRequestV3
- UninstallEdgeLogAgent
- InstallEdgeLogAgent
- DescribeEdgeLogSwitches
- CreateEdgeLogConfig
- CreateEdgeCVMInstances
- UpdateEdgeClusterVersion
- DescribeEdgeClusterUpgradeInfo
- Cloud Native Monitoring APIs
- Virtual node APIs
- Other APIs
- Scaling group APIs
- Data Types
- Error Codes
- API Mapping Guide
- TKE Insight
- TKE Scheduling
- FAQs
- Service Agreement
- Contact Us
- Purchase Channels
- Glossary
- User Guide(Old)
Overview
Tencent Cloud upgraded the basic monitoring architecture to improve the stability of the TKE basic monitoring and alarming feature. After the upgrade, a DaemonSet named
tke-monitor-agent is deployed under the kube-system namespace in the cluster, and the K8s resource objects of authentication and authorization are created, including ClusterRole, ServiceAccount, and ClusterRoleBinding. These resource objects are all named tke-monitor-agent.Strengths
This add-on collects the monitoring data of containers, Pods, nodes, and community add-ons. The collected data is used for basic monitoring metrics display, metrics alarming, and metric-based HPA service in the console. By deploying this add-on, you can fix the problem that the monitoring data can't be obtained due to the instability of the basic monitoring service, thereby enjoying more stable monitoring, alarming, and HPA services.
Impact
Deploying this add-on does not affect the normal running of the cluster.
If your node resources are allocated unreasonably, node load is too heavy, or node resources are not enough, deploying the basic monitoring add-on may cause the problem where the Pod corresponding to the
tke-monitor-agent DaemonSet is in the status of Pending, Evicted, OOMKilled or CrashLoopBackOff. The details of the status are as follows:Pending: The resources on the cluster node are not enough to schedule a Pod. You can schedule the Pod to the node by setting the quantity of requested resources for the
tke-monitor-agent DaemonSet to 0. For more information, see Pod Remains in Pending.Evicted: This status may be caused by insufficient node resources or a heavy load on the node. You can find out the cause and solve the problem in the following ways:
Run
kubectl describe pod -n kube-system <podName> to check the cause according to the description in the Message field.Run
kubectl describe pod -n kube-system <podName> to check the cause according to the description in the Events field.CrashLoopBackOff or OOMKilled: Run
kubectl describe pod -n kube-system <podName> to check whether an OOM error occurs. If yes, you can increase the value of memory limits, which can't exceed 100 MB. If the error still occurs after the value is set to 100 MB, submit a ticket for assistance.ContainerCreating: Run
kubectl describe pod -n kube-system <podName> to check the Events field. If Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create a sandbox for pod "<pod name >": Error response from daemon: Failed to set projid for /data/docker/overlay2/xxx-init: no space left on device is displayed, the container data disk is full, and you can clear the data disk to restore it.Note:
Quantity of resources consumed in each Pod managed by the DaemonSet (named
tke-monitor-agent) is positively correlated with the number of Pods and containers running on the node. Below is a sample stress test with low MEM and CPU usage:
Data volume
220 Pods are deployed on a node, and each Pod contains three containers.
Resources consumedMEM (peak) | CPU (peak) |
About 40 MiB | 0.01C |
The stress test result of the CPU usage is as shown below:
The stress test result of the memory usage is as shown below:
Component Permission Description
Permission Description
The permission of this component is the minimal dependency required for the current feature to operate.
Permission Scenarios
Feature | Involved Object | Involved Operation Permission |
It is required to gather the number of Pods and related information in the cluster. | ReplicaSets, Deployments, and Pods | list/watch |
Obtaining the metric information of cadvisor by visiting the /metrics port on the Kubelet of the node. | nodes, nodes/proxy, and nodes/metrics | list/watch/get |
Delivering metric data with cluster-monitor | services | list/watch |
Reporting metrics to HPA-Metrics-Server | custommetrics | update |
Permission Definition
apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata:name: tke-monitor-agentrules:- apiGroups: ["apps"]resources: ["replicasets"]verbs: ["list", "watch"]- apiGroups: ["apps"]resources: ["deployments"]verbs: ["list", "watch"]- apiGroups: [""]resources: ["nodes", "nodes/proxy", "nodes/metrics"]verbs: ["list", "watch", "get"]- apiGroups: [""]resources: ["services"]verbs: ["list", "watch"]- apiGroups: [""]resources: ["pods"]verbs: ["list", "watch"]- apiGroups: ["monitor.tencent.io"]resources: ["custommetrics"]verbs: ["update"]
Yes
No
Was this page helpful?