--- name: containers-ts-app-02-imgpull title: Why do images fail to pull from registry with `ImagePullBackOff` or authorization errors? description: If your pod configuration has an image pull secret that uses an API key, check that the API key credentials are set up correctly. last-updated: 2026-06-08 --- > ## Documentation Index > The table of contents for this documentation set is at https://cloud.ibm.com/docs/containers?format=markdown > The index for all IBM Cloud docs is at: https://cloud.ibm.com/docs/llms.txt > Use these files to discover more information as needed. # Why do images fail to pull from registry with `ImagePullBackOff` or authorization errors? {: #ts-app-image-pull} {: support} [Virtual Private Cloud]{: tag-vpc} [Classic infrastructure]{: tag-classic-inf} When you deploy a workload that pulls an image from IBM Cloud Container Registry, your pods fail with an **`ImagePullBackOff`** status. {: tsSymptoms} ```sh kubectl get pods ``` {: pre} ```sh NAME READY STATUS RESTARTS AGE 0/1 ImagePullBackOff 0 2m ``` {: screen} When you describe the pod, you see authentication errors similar to the following. ```sh kubectl describe pod ``` {: pre} ```sh Failed to pull image ".icr.io//:" ... unauthorized: authentication required Failed to pull image ".icr.io//:" ... 401 Unauthorized ``` {: screen} ```sh ... Failed to pull image ":" ... Manifest for : not found ``` {: screen} Your cluster uses an API key that is stored in an [image pull secret](https://cloud.ibm.com/docs/containers?topic=containers-registry&format=markdown#cluster_registry_auth) to authorize the cluster to pull images from IBM Cloud Container Registry, or the image with the specific tag does not exist in the repository. {: tsCauses} By default, new clusters have image pull secrets that use API keys so that the cluster can pull images from any regional `icr.io` registry for containers that are deployed to the `default` Kubernetes namespace. 1. Verify that you use the correct name and tag of the image in your deployment YAML file. {: tsResolve} ```sh ibmcloud cr images ``` {: pre} 2. Check your [pull traffic and storage quota](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_quota&format=markdown). If the limit is reached, free up used storage or ask your registry administrator to increase the quota. ```sh ibmcloud cr quota ``` {: pre} 3. Get the pod configuration file of a failing pod, and look for the `imagePullSecrets` section. ```sh kubectl get pod -o yaml ``` {: pre} Example output ```sh ... imagePullSecrets: - name: all-icr-io ... ``` {: screen} 4. If no image pull secrets are listed, set up the image pull secret in your namespace. 1. Verify that the `default` namespace has `icr-io` image pull secrets for each regional registry that you want to use. If no `icr-io` secrets are listed in the namespace, [use the `ibmcloud ks cluster pull-secret apply --cluster ` command](https://cloud.ibm.com/docs/containers?topic=containers-registry&format=markdown#imagePullSecret_migrate_api_key) to create the image pull secrets in the `default` namespace. ```sh kubectl get secrets -n default | grep "icr-io" ``` {: pre} 2. [Copy the `all-icr-io` image pull secret from the `default` Kubernetes namespace to the namespace where you want to deploy your workload](https://cloud.ibm.com/docs/containers?topic=containers-registry&format=markdown#copy_imagePullSecret). 3. [Add the image pull secret to the service account for this Kubernetes namespace](https://cloud.ibm.com/docs/containers?topic=containers-registry&format=markdown#store_imagePullSecret) so that all pods in the namespace can use the image pull secret credentials. 5. If image pull secrets are listed in the pod, determine what type of credentials you use to access IBM Cloud Container Registry. * If the secret has `icr` in the name, you use an API key to authenticate with the `icr.io` domain names. Continue with [Troubleshooting image pull secrets that use API keys](#img-pull-api-key). * If you have both types of secrets, then you use both authentication methods. Going forward, use the `icr.io` domain names in your deployment YAMLs for the container image. Continue with [Troubleshooting image pull secrets that use API keys](#img-pull-api-key). ## Troubleshooting image pull secrets that use API keys {: #img-pull-api-key} If your pod configuration has an image pull secret that uses an API key, check that the API key credentials are set up correctly. {: shortdesc} The following steps assume that the API key stores the credentials of a service ID. If you set up your image pull secret to use an API key of an individual user, you must verify that user's IBM Cloud IAM permissions and credentials. {: note} 1. Find the service ID that API key uses for the image pull secret by reviewing the **Description**. The service ID that is created with the cluster is named `cluster-` and is used in the `default` Kubernetes namespace. If you created another service ID such as to access a different Kubernetes namespace or to modify IBM Cloud IAM permissions, you customized the description. ```sh ibmcloud iam service-ids ``` {: pre} Example output ```sh UUID Name Created At Last Updated Description Locked ServiceId-aa11... 2019-02-01T19:01+0000 2019-02-01T19:01+0000 ID for false ServiceId-bb22... 2019-02-01T19:01+0000 2019-02-01T19:01+0000 Service ID for IBM Cloud Container Registry in Kubernetes cluster namespace false ``` {: screen} 2. Verify that the service ID is assigned at least an IBM Cloud IAM **Reader** [service access role policy for IBM Cloud Container Registry](https://cloud.ibm.com/docs/Registry?topic=Registry-user&format=markdown#create). If the service ID does not have the **Reader** service access role, [edit the IAM policies](https://cloud.ibm.com/docs/iam?topic=iam-serviceids&interface=ui&format=markdown#update_serviceid). If the policies are correct, continue with the next step to see if the credentials are valid. ```sh ibmcloud iam service-policies ``` {: pre} Example output ```sh Policy ID: a111a111-b22b-333c-d4dd-e555555555e5 Roles: Reader Resources: Service Name container-registry Service Instance Region Resource Type namespace Resource ``` {: screen} 3. Check if the image pull secret credentials are valid. 1. Get the image pull secret configuration. If the pod is not in the `default` namespace, include the `-n` option. ```sh kubectl get secret -o yaml [-n ] ``` {: pre} 2. In the output, copy the base64 encoded value of the `.dockerconfigjson` field. ```yaml apiVersion: v1 kind: Secret data: .dockerconfigjson: eyJyZWdp...== ... ``` {: screen} 3. Decode the base64 string. For example, on OS X you can run the following command. ```sh echo -n "" | base64 --decode ``` {: pre} Example output ```sh {"auths":{".icr.io":{"username":"iamapikey","password":"","email":"","auth":""}}} ``` {: screen} 4. Compare the image pull secret regional registry domain name with the domain name that you specified in the container image. By default, new clusters have image pull secrets for each regional registry domain name for containers that run in the `default` Kubernetes namespace. However, if you modified the default settings or are using a different Kubernetes namespace, you might not have an image pull secret for the regional registry. [Copy an image pull secret](https://cloud.ibm.com/docs/containers?topic=containers-registry&format=markdown#copy_imagePullSecret) for the regional registry domain name. 5. Log in to the registry from your local machine by using the `username` and `password` from your image pull secret. If you can't log in, you might need to fix the service ID. ```sh docker login -u iamapikey -p .icr.io ``` {: pre} 1. Re-create the cluster service ID, IBM Cloud IAM policies, API key, and image pull secrets for containers that run in the `default` Kubernetes namespace. ```sh ibmcloud ks cluster pull-secret apply --cluster ``` {: pre} 2. Re-create your deployment in the `default` Kubernetes namespace. If you still see an authorization error message, repeat Steps 1-5 with the new image pull secrets. If you still can't log in, [open an IBM Cloud Support case](https://cloud.ibm.com/docs/containers?topic=containers-get-help&format=markdown). 6. If the login succeeds, pull an image locally. If the command fails with an `access denied` error, the registry account is in a different IBM Cloud account than the one your cluster is in. [Create an image pull secret to access images in the other account](https://cloud.ibm.com/docs/containers?topic=containers-registry&format=markdown#other_registry_accounts). If you can pull an image to your local machine, then your API key has correct permissions, but the API setup in your cluster is not correct. ```sh docker pull icr.io//: ``` {: pre} 7. Check that the pull secret is either referenced directly from the deployment or from the service account that the deployment uses. If you still can't resolve the issue, [contact support](https://cloud.ibm.com/docs/containers?topic=containers-get-help&format=markdown).