tencent cloud

Tencent Kubernetes Engine

Release Notes and Announcements
Release Notes
Announcements
Release Notes
Product Introduction
Overview
Strengths
Architecture
Scenarios
Features
Concepts
Native Kubernetes Terms
Common High-Risk Operations
Regions and Availability Zones
Service Regions and Service Providers
Open Source Components
Purchase Guide
Purchase Instructions
Purchase a TKE General Cluster
Purchasing Native Nodes
Purchasing a Super Node
Getting Started
Beginner’s Guide
Quickly Creating a Standard Cluster
Examples
Container Application Deployment Check List
Cluster Configuration
General Cluster Overview
Cluster Management
Network Management
Storage Management
Node Management
GPU Resource Management
Remote Terminals
Application Configuration
Workload Management
Service and Configuration Management
Component and Application Management
Auto Scaling
Container Login Methods
Observability Configuration
Ops Observability
Cost Insights and Optimization
Scheduler Configuration
Scheduling Component Overview
Resource Utilization Optimization Scheduling
Business Priority Assurance Scheduling
QoS Awareness Scheduling
Security and Stability
TKE Security Group Settings
Identity Authentication and Authorization
Application Security
Multi-cluster Management
Planned Upgrade
Backup Center
Cloud Native Service Guide
Cloud Service for etcd
TMP
TKE Serverless Cluster Guide
TKE Registered Cluster Guide
Use Cases
Cluster
Serverless Cluster
Scheduling
Security
Service Deployment
Network
Release
Logs
Monitoring
OPS
Terraform
DevOps
Auto Scaling
Containerization
Microservice
Cost Management
Hybrid Cloud
AI
Troubleshooting
Disk Full
High Workload
Memory Fragmentation
Cluster DNS Troubleshooting
Cluster kube-proxy Troubleshooting
Cluster API Server Inaccessibility Troubleshooting
Service and Ingress Inaccessibility Troubleshooting
Common Service & Ingress Errors and Solutions
Engel Ingres appears in Connechtin Reverside
CLB Ingress Creation Error
Troubleshooting for Pod Network Inaccessibility
Pod Status Exception and Handling
Authorizing Tencent Cloud OPS Team for Troubleshooting
CLB Loopback
API Documentation
History
Introduction
API Category
Making API Requests
Elastic Cluster APIs
Resource Reserved Coupon APIs
Cluster APIs
Third-party Node APIs
Relevant APIs for Addon
Network APIs
Node APIs
Node Pool APIs
TKE Edge Cluster APIs
Cloud Native Monitoring APIs
Scaling group APIs
Super Node APIs
Other APIs
Data Types
Error Codes
TKE API 2022-05-01
FAQs
TKE General Cluster
TKE Serverless Cluster
About OPS
Hidden Danger Handling
About Services
Image Repositories
About Remote Terminals
Event FAQs
Resource Management
Service Agreement
TKE Service Level Agreement
TKE Serverless Service Level Agreement
Contact Us
Glossary
DocumentaçãoTencent Kubernetes EngineRelease Notes and AnnouncementsAnnouncementsSecurity Vulnerability Fix DescriptionOOM Causes docker to Stop and Not Restart for Repair

OOM Causes docker to Stop and Not Restart for Repair

PDF
Modo Foco
Tamanho da Fonte
Última atualização: 2024-05-27 16:08:53

Problem Description

In Docker 19 and later versions, when excessive system memory usage causes containerd to encounter an Out of Memory (OOM) situation, it may result in Docker stopping and not restarting automatically. This issue can be reproduced by executing the pkill -9 containerd; systemctl is-active dockerd containerd command. At this point, dockerd will be stopped by systemd.
The most severe impact could be general nodes becoming NotReady after OOM, and issues with the primary node in an independent cluster could trigger an avalanche effect.

Problem Analysis

Initially, the Docker community set the relationship between docker and containerd as dockerd.service BindsTo containerd.service. This causes systemd to actively stop dockerd when containerd is forcibly terminated by the kill -9 command. Even if Restart is set in Docker, recovery is not possible. For more information, see:
https://github.com/moby/moby/commit/a985655ac4eb6c5b60b5eab8d8d09a487e353e1d
https://github.com/moby/moby/commit/009639e0f46d270309821998836fb664046b3e07

Fixing Incremental Nodes

Incremental nodes were fixed on April 20, 2023.

Fixing Legacy Nodes

For legacy nodes, you can fix the problem with the following script:
#!/bin/bash
insert_if_absent() {
    line="${1}"
    lead="$(echo "${line}" | cut -f1 -d=)""="
    if  ! grep "^${lead}" /usr/lib/systemd/system/containerd.service  > /dev/null 2>&1; then
        sed -i "/^ExecStart=/a${line}" /usr/lib/systemd/system/containerd.service
    fi
}

insert_if_absent OOMScoreAdjust=-999
insert_if_absent RestartSec=5
insert_if_absent Restart=always

sed -i '/BindsTo/d' /usr/lib/systemd/system/dockerd.service
sed -i 's/^Wants.*/Wants\\=network-online.target containerd.service/' /usr/lib/systemd/system/dockerd.service

systemctl daemon-reload
You can verify whether the issue of Docker not being able to restart after containerd is forcibly terminated has been successfully fixed by executing the command below. Additionally, you can further verify by executing the docker run command.
pkill -9 containerd;systemctl is-active dockerd containerd

Anterior: CVE-2024-21626 Vulnerability Fix DescriptionPróximo: Runc Vulnerability (CVE-2021-30465) Fix Description

Ajuda e Suporte

Esta página foi útil?

comentários