Dear Tencent Cloud User,
Tencent Cloud CVM and CWPP have recently detected a critical kernel performance issue affecting some Ubuntu systems that may cause system instability. To ensure the security and stability of your business systems under your Tencent Cloud account, we are issuing this risk warning notification and recommend that affected users take action as soon as possible.
I. Risk Overview
When applications or system tools read process memory information from `/proc/[pid]/stat`, there is a risk of triggering soft lockup, significantly impacting system performance and stability.
1. Affected Scope:
● Affected kernel versions: 5.15.0-144-generic (inclusive) through 5.15.0-153.163 (exclusive)
● Primary impact: Ubuntu 22.04 LTS and related derivatives
2. Technical Details:
● Trigger condition: Reading process status file `/proc/[pid]/stat`
● Symptoms: Process unresponsive, system performance degradation
● Kernel call stack:
proc_tgid_stat+0x14/0x20
proc_single_show+0x52/0xc0
seq_read_iter+0x124/0x4b0
vfs_read+0x9f/0x1a0
● Log characteristics: Kernel dmesg output will show lockup warning messages
3. Official Bug Report: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2118407
II. Detection Methods
1) Check current kernel version
Execute command: `uname -r`
2) Determine if affected
If the output of `uname -r` shows a kernel version between 5.15.0-144 and 5.15.0-153.163, your system is at risk.
3) Check system logs
Execute commands:
sudo dmesg | grep -i "soft lockup"
sudo journalctl -k | grep -i "proc_tgid_stat"
If these commands produce output, your system may have already encountered this issue.
III. Solution
Please upgrade to the latest kernel following these steps:
1) Route traffic away from the CVM instance
Route traffic away from the CVM instance by modifying CLB/DNS configurations.
2) Update package lists
Execute command: sudo apt update
3) Upgrade kernel package
Execute command: sudo apt upgrade linux-image-generic
4) Reboot the system
Execute command: sudo reboot
5) Verify the fix
After reboot, execute: `uname -r`
Confirm that the version number is greater than or equal to 5.15.0-153.163 to verify successful upgrade.
Important Notes:
● It is recommended to perform the upgrade during low-traffic periods
● Please back up important data before upgrading
● For production environments, verify in a test environment first
● After upgrading, monitor system stability to confirm the issue is resolved
IV. Temporary Mitigation Measures
If immediate kernel upgrade is not possible, consider the following temporary measures:
1. Monitor system status: Regularly check system logs for soft lockup-related warnings
2. Avoid frequent reads of `/proc/[pid]/stat`: Reduce frequent queries of process status information
3. Prepare contingency plan: Develop a kernel upgrade plan and execute during low-traffic periods
If you have any questions in the process of using cloud products, you can submit a ticket , We will answer your questions as soon as we can !
Thank you!
Tencent Cloud Team
