In scenarios where a Pod is connected directly at the access layer, when the backend performs a rolling update, or the backend Pod is deleted, if you delete the Pod directly from the CLB backend, unprocessed requests that have been received by it cannot be processed.
Particularly, in persistent connection scenarios, such as meeting business, if the Pod of the workload is updated or deleted directly, the meeting will be interrupted.
This is only effective in the direct access mode. Check whether your cluster supports direct access.
The following is an example of using an annotation to indicate the use of graceful shutdown. For detailed Service annotations, see Service Annotation.
Step 2 involves using
terminationGracePeriodSeconds in the workload that requires graceful shutdown.
The following describes the container termination process in a Kubernetes environment:
DeletionTimestampand is in Terminating status, the weight of the Pod on the CLB backend is adjusted to
preStophook is configured for the Pod, it will be executed.
terminationGracePeriodSeconds(30s by default) elapses, a SIGKILL signal will be sent to forcibly stop it.
To implement graceful termination, you must process the SIGTERM signal in your business code. The main logic is to stop accepting new traffic, continue to process existing traffic, and quit after all connections are closed. For more information, see Go by Example: Signals.
If the SIGTERM signal is not processed in your business code, or if you cannot control the used third-party library or system to add the logic of graceful termination, you can also try configuring
preStop for the Pod to implement such logic as shown below:
In certain extreme cases, new connections may still be forwarded within a short period of time after the Pod is deleted. This is because kubelet and kube-proxy watch that the Pod is deleted at the same time, and kubelet may have stopped the containers before kube-proxy syncs the rules. Normally, an application will no longer accept new connections after it receives
SIGTERM, and it will only keep the existing connections for processing, which may cause some requests to fail at the moment when the Pod is deleted.
In view of the above, you can use
preStop to make the Pod sleep for a short while first and then start to stop the container processes after kube-proxy completes the rule sync as shown below:
terminationGracePeriodSeconds to adjust the termination grace period
If you need a long termination grace period (
preStop and the business process termination may take more than 30s in total), you can customize
terminationGracePeriodSeconds as shown below based on the actual situation so as to avoid being stopped by SIGKILL prematurely:
Graceful shutdown sets the weight on the CLB backend to
0 only when a Pod is deleted. If a running Pod becomes unhealthy, setting the weight to
0 on the backend can reduce the risk of service unavailability.
You can use the
service.cloud.tencent.com/enable-grace-shutdown-tkex: "true" annotation to implement graceful shutdown.
The annotation will check whether an endpoint in the Endpoint object is
not-ready, and if so, the annotation will set the weight on the CLB backend to