This article describes the causes that lead to a Pod remaining in the ImagePullBackOff status and how to troubleshoot these issues. Refer to the following instructions for troubleshooting.
Possible Causes
HTTP registry addresses are not added under insecure-registry
The self-signed registry CA for HTTPS traffic is not added to the node
The private image registry failed to authenticate the request
Damaged image file
Image pull timed out
Image not found
Troubleshooting
Checking if HTTP registries are added under insecure-registry
dockerd pulls images from HTTPS registries by default. If you want to use HTTP registries, you need to add them under insecure-registry and restart or reload dockerd to apply the change.
Checking if the self-signed registry CA is added to the node
If your HTTPS registry uses a self-signed CA, dockerd will authenticate the certificate. You can only use the registry if the certificate is successfully authenticated.
To make sure the process succeeds, place the certificate file under:
/etc/docker/certs.d/<Registry:port>/ca.crt
Checking for private registry configuration issues
If you use a private image registry and the Pod is not configured with an imagePullSecret or uses the wrong imagePullSecret, the registry will refuse the pull requests and the Pod will remain in the ImagePullBackOff status.
Checking for damaged image files
If the image file is damaged before or when it is pushed to the registry, the downloaded image is also damaged. In this case, you need to push the image again.
Checking for image pull timeout
Error description
When multiple Pods are launched at the same time from the same node, all containers pull images and these images are stored in a download queue. If the images in the front of the queue are large in size and take a long time to pull, the images behind them may fail to be pulled due to timeout.
By default, kubelet pulls images one at a time.
--serialize-image-pulls Pull images one at a time. We recommend *not* changing the default value on nodes that run docker daemon with version < 1.9 or an Aufs storage backend. Issue #10959 has more details. (default true)
Solution
If necessary, you can enable concurrent image pulling and set a concurrency limit. The following is an example:
--Registry-qps int32 If > 0, limit Registry pull QPS to this value. If 0, unlimited. (default 5)
--Registry-burst int32 Maximum size of a bursty pulls, temporarily allows pulls to burst to this number, while still not exceeding Registry-qps. Only used if --Registry-qps > 0 (default 10)
Checking if the image exists
If the image does not exist, the Pod may remain in the ImagePullBackOff status. You can identify the issue using kublet logs, as shown by the following:
PullImage "imroc/test:v0.2" from image service failed: rpc error: code = Unknown desc = Error response from daemon: manifest for imroc/test:v0.2 not found