Setting up an image registry

Default image pull secret setup

Generally, your IBM Cloud Kubernetes Service cluster is set up to pull images from all IBM Cloud Container Registry icr.io domains from the default Kubernetes namespace only. Review the following FAQs to learn more about how to pull images in other Kubernetes namespaces or accounts, restrict pull access, or why your cluster might not have the default image pull secrets.

How is my cluster set up to pull images from the default Kubernetes namespace?

When you create a cluster, the cluster has an IBM Cloud IAM service ID that is given an IAM Reader service access role policy to IBM Cloud Container Registry. The service ID credentials are impersonated in a non-expiring API key that is stored in image pull secrets in your cluster. The image pull secrets are added to the default Kubernetes namespace and the list of secrets in the default service account for this Kubernetes namespace. By using image pull secrets, your deployments can pull images (read-only access) from the global and regional IBM Cloud Container Registry to deploy containers in the default Kubernetes namespace.

• The global registry securely stores public images that are provided by IBM. You can refer to these public images across your deployments instead of having different references for images that are stored in each regional registry.
• The regional registry securely stores your own private Docker images.

What if I don't have image pull secrets in the default Kubernetes namespace?

You can check the image pull secrets by logging in to your cluster and running kubectl get secrets -n default | grep "icr-io". If no icr secrets are listed, the person who created the cluster might not have had the required permissions to IBM Cloud Container Registry in IAM. See Updating existing clusters to use the API key image pull secret.

Can I restrict pull access to a certain regional registry?

Yes, you can edit the existing IAM policy of the service ID that restricts the Reader service access role to that regional registry or a registry resource such as a namespace. Before you can customize registry IAM policies, you must enable IBM Cloud IAM policies for IBM Cloud Container Registry.

Want to make your registry credentials even more secured Ask your cluster admin to enable a key management service provider in your cluster to encrypt Kubernetes secrets in your cluster, such as the image pull secret that stores your registry credentials.

Can I pull images in a Kubernetes namespace other than default?

Not by default. By using the default cluster setup, you can deploy containers from any image that is stored in your IBM Cloud Container Registry namespace into the default Kubernetes namespace of your cluster. To use these images in any other Kubernetes namespaces or other IBM Cloud accounts, you have the option to copy or create your own image pull secrets.

Can I pull images from a different IBM Cloud account?

Yes, create an API key in the IBM Cloud account that you want to use. Then, in each namespace of each cluster that you want to pull images from the IBM Cloud account, create a secret that holds the API key. For more information, follow along with this example that uses an authorized service ID API key.

To use a non-IBM Cloud registry such as Docker, see Accessing images that are stored in other private registries.

Does the API key need to be for a service ID? What happens if I reach the limit of service IDs for my account?

The default cluster setup creates a service ID to store IBM Cloud IAM API key credentials in the image pull secret. However, you can also create an API key for an individual user and store those credentials in an image pull secret. If you reach the IAM limit for service IDs, your cluster is created without the service ID and image pull secret and can't pull images from the icr.io registry domains by default. You must create your own image pull secret, but by using an API key for an individual user such as a functional ID, not an IBM Cloud IAM service ID.

I see image pull secrets for the regional registry domains and all registry domains. Which one do I use?

Previously, IBM Cloud Kubernetes Service created separate image pull secrets for each regional, public icr.io registry domain. Now, all the public and private icr.io registry domains for all regions are stored in a single all-icr-io image pull secret that is automatically created in the default Kubernetes namespace of your cluster.

For workloads in other Kubernetes namespaces in the cluster to pull container images from a private registry, you can now copy only the all-icr-io image pull secret to that Kubernetes namespace. Then, specify the all-icr-io secret in your service account or deployment. You don't need to copy the image pull secret that matches the regional registry of your image anymore. Also, keep in mind that you don't need image pull secrets for public registries, which don't require authentication.

My cluster image pull secret uses a registry token. Does a token still work?

From 19 August 2021, using IBM Cloud Container Registry tokens is discontinued and no longer works. For more information, see IBM Cloud Container Registry Deprecates Registry Tokens for Authentication.

After I copy or create an image pull secret in another Kubernetes namespace, am I done?

Not quite. Your containers must be authorized to pull images by using the secret that you created. You can add the image pull secret to the service account for the namespace, or refer to the secret in each deployment. For instructions, see Using the image pull secret to deploy containers.

Private network connection to icr.io registries

When you use the private network to pull images, your image pull traffic is not charged as public bandwidth, because the traffic is on the private network. For more information, see the IBM Cloud Container Registry private network documentation.

When you set up your IBM Cloud account to use service endpoints, you can use a private network connection to push images to and to pull images from IBM Cloud Container Registry.

What do I need to do to set up my cluster to use the private connection to icr.io registries?

1. Enable a Virtual Router Function (VRF) for your IBM Cloud infrastructure account so that you can use the IBM Cloud Container Registry private cloud service endpoint. To enable VRF, see Enabling VRF. To check whether a VRF is already enabled, use the ibmcloud account show command.
2. Enable your IBM Cloud account to use service endpoints.

Now, IBM Cloud Container Registry automatically uses the private cloud service endpoint. You don't need to enable the private cloud service endpoint for your IBM Cloud Kubernetes Service clusters.

I have a private-only cluster. How do I enforce that my image traffic remains on the private network?

The image push and pull traffic is automatically on the private network. IBM Cloud Kubernetes Service can use the public icr.io registry domains in existing image pull secrets to authenticate requests to IBM Cloud Container Registry. These requests are automatically redirected to the private icr.io registry domains. You don't need to modify the image pull secrets or configure additional settings.

Do I have to use the private icr.io registry addresses for anything else?

Yes, if you sign your images for trusted content, the signatures contain the registry domain name. If you want to use the private icr.io domain for your signed images, resign your images with the private icr.io domains.

Updating your image pull secret

To update your cluster image pull secret in the default Kubernetes namespace.

1. Get your cluster ID.

   ibmcloud ks cluster ls
   

2. Run the following command to create a service ID for the cluster and assign the service ID an IAM Reader service access role for IBM Cloud Container Registry. The command also creates an API key to impersonate the service ID credentials and stores the API key in a Kubernetes image pull secret in the cluster. The image pull secret is in the default Kubernetes namespace.

   ibmcloud ks cluster pull-secret apply --cluster <cluster_name_or_ID>
   

   When you run this command, the creation of IAM credentials and image pull secrets is initiated and can take some time to complete. You can't deploy containers that pull an image from the IBM Cloud Container Registry icr.io domains until the image pull secrets are created.

3. Verify that the image pull secrets are created in your cluster.

   kubectl get secrets | grep icr-io
   

   Example output

   all-icr-io           kubernetes.io/dockerconfigjson        1         16d
   

4. Update your container deployments to pull images from the icr.io domain name.

5. Optional: If you have a firewall, make sure you allow outbound network traffic to the registry subnets for the domains that you use.

6. Complete your setup by using one of the following options.

   • To pull images in Kubernetes namespaces other than default or from other IBM Cloud accounts, copy or create another image pull secret.
   • To restrict the image pull secret access to particular registry resources such as namespaces or regions:
     1. Make sure that IBM Cloud IAM policies for IBM Cloud Container Registry are enabled.
     2. Edit the IBM Cloud IAM policies for the service ID, or create another image pull secret.

Copying an existing image pull secret

You can copy an image pull secret, such as the one that is automatically created for the default Kubernetes namespace, to other namespaces in your cluster. If you want to use different IBM Cloud IAM API key credentials for this namespace such as to restrict access to specific namespaces, or to pull images from other IBM Cloud accounts, create an image pull secret instead.

1. List available Kubernetes namespaces in your cluster, or create a namespace to use.

   kubectl get namespaces
   

   Example output

   default          Active    79d
   ibm-cert-store   Active    79d
   ibm-system       Active    79d
   kube-public      Active    79d
   kube-system      Active    79d
   

   To create a namespace

   kubectl create namespace <namespace_name>
   

2. List the existing image pull secrets in the default Kubernetes namespace for IBM Cloud Container Registry.

   kubectl get secrets -n default | grep icr-io
   

   Example output

   all-icr-io          kubernetes.io/dockerconfigjson        1         16d
   

3. Copy the all-icr-io image pull secret from the default namespace to the namespace of your choice. The new image pull secrets are named <namespace_name>-icr-<region>-io.

   kubectl get secret all-icr-io -n default -o yaml | sed 's/default/<new-namespace>/g' | kubectl create -n <new-namespace> -f -   
   

4. Verify that the secrets are created successfully.

   kubectl get secrets -n <namespace_name> | grep icr-io
   

5. To deploy containers, add the image pull secret to each deployment or to the service account of the namespace so that any deployment in the namespace can pull images from the registry.

Creating an image pull secret with different IAM API key credentials

You can assign IBM Cloud IAM access policies to users or a service ID to restrict permissions to specific registry image namespaces or actions (such as push or pull). Then, create an API key and store these registry credentials in an image pull secret for your cluster.

For example, to access images in other IBM Cloud accounts, create an API key that stores the IBM Cloud Container Registry credentials of a user or service ID in that account. Then, in your cluster's account, save the API key credentials in an image pull secret for each cluster and cluster namespace.

The following steps create an API key that stores the credentials of an IBM Cloud IAM service ID. Instead of using a service ID, you might want to create an API key for a user ID that has an IBM Cloud IAM service access policy to IBM Cloud Container Registry. However, make sure that the user is a functional ID or have a plan in case the user leaves so that the cluster can still access the registry.

1. List available Kubernetes namespaces in your cluster, or create a namespace to use where you want to deploy containers from your registry images.

   kubectl get namespaces
   

   Example output

   default          Active    79d
   ibm-cert-store   Active    79d
   ibm-system       Active    79d
   kube-public      Active    79d
   kube-system      Active    79d
   

   To create a namespace

   kubectl create namespace <namespace_name>
   

2. Create an IBM Cloud IAM service ID for your cluster that is used for the IAM policies and API key credentials in the image pull secret. Be sure to give the service ID a description that helps you retrieve the service ID later, such as including both the cluster and namespace name.

   ibmcloud iam service-id-create <cluster_name>-<namespace>-id --description "Service ID for IBM Cloud Container Registry in Kubernetes cluster <cluster_name> namespace <namespace>"
   

3. Create a custom IBM Cloud IAM policy for your cluster service ID that grants access to IBM Cloud Container Registry.

   ibmcloud iam service-policy-create <cluster_service_ID> --roles <service_access_role> --service-name container-registry [--region <IAM_region>] [--resource-type namespace --resource <registry_namespace>]
   

   cluster_service_ID

   Required. Replace with the <cluster_name>-<kube_namespace>-id service ID that you previously created for your Kubernetes cluster.

   --service-name container-registry

   Required. Enter container-registry so that the IAM policy is for IBM Cloud Container Registry.

   --roles <service_access_role>

   Required. Enter the service access role for IBM Cloud Container Registry that you want to scope the service ID access to. Possible values are Reader, Writer, and Manager.

   --region <IAM_region>

   Optional. If you want to scope the access policy to certain IAM regions, enter the regions in a comma-separated list. Possible values are global and the local registry regions.

   --resource-type namespace --resource <registry_namespace>

   Optional. If you want to limit access to only images in certain IBM Cloud Container Registry namespaces, enter namespace for the resource type and specify the <registry_namespace>. To list registry namespaces, run ibmcloud cr namespaces.

4. Create an API key for the service ID. Name the API key similar to your service ID, and include the service ID that you previously created, <cluster_name>-<kube_namespace>-id. Be sure to give the API key a description that helps you retrieve the key later.

   ibmcloud iam service-api-key-create <cluster_name>-<namespace>-key <cluster_name>-<namespace>-id --description "API key for service ID <service_id> in Kubernetes cluster <cluster_name> namespace <namespace>"
   

5. Retrieve your API Key value from the output of the previous command.

   Please preserve the API key! It can't be retrieved after it's created.
   
   Name          <cluster_name>-<kube_namespace>-key   
   Description   key_for_registry_for_serviceid_for_kubernetes_cluster_multizone_namespace_test   
   Bound To      crn:v1:bluemix:public:iam-identity::a/1bb222bb2b33333ddd3d3333ee4ee444::serviceid:ServiceId-ff55555f-5fff-6666-g6g6-777777h7h7hh   
   Created At    2019-02-01T19:06+0000   
   API Key       i-8i88ii8jjjj9jjj99kkkkkkkkk_k9-llllll11mmm1   
   Locked        false   
   UUID          ApiKey-222nn2n2-o3o3-3o3o-4p44-oo444o44o4o4   
   

6. Create an image pull secret to store the API key credentials in the cluster namespace. Repeat this step for each namespace of each cluster for each icr.io domain that you want to pull images from.

   kubectl --namespace <namespace> create secret docker-registry <secret_name> --docker-server=<registry_URL> --docker-username=iamapikey --docker-password=<api_key_value> --docker-email=<docker_email>
   

   --namespace <namespace>

   Required. Specify the Kubernetes namespace of your cluster that you used for the service ID name.

   <secret_name>

   Required. Enter a name for your image pull secret.

   --docker-server <registry_URL>

   Required. Set the URL to the image registry where your registry namespace is set up. For available domains, see Local regions.

   --docker-username iamapikey

   Required. Enter the username to log in to your private registry. If you use IBM Cloud Container Registry, enter iamapikey.

   --docker-password <token_value>

   Required. Enter the value of your API Key that you previously retrieved.

   --docker-email <docker-email>

   Required. If you have one, enter your Docker email address. If you don't, enter a fictional email address, such as a@b.c. This email is required to create a Kubernetes secret, but is not used after creation.

7. Verify that the secret was created successfully. Replace <namespace> with the namespace where you created the image pull secret.

   kubectl get secrets --namespace <namespace>
   

8. Add the image pull secret to a Kubernetes service account so that any pod in the namespace can use the image pull secret when you deploy a container.

Accessing images that are stored in other private registries

If you already have a private registry, you must store the registry credentials in a Kubernetes image pull secret and reference this secret from your configuration file.

Before you begin:

1. Create a cluster.
2. Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

To create an image pull secret:

1. Create the Kubernetes secret to store your private registry credentials.

   kubectl --namespace <namespace> create secret docker-registry <secret_name>  --docker-server=<registry_URL> --docker-username=<docker_username> --docker-password=<docker_password> --docker-email=<docker_email>
   

   --namespace <namespace>

   Required. The Kubernetes namespace of your cluster where you want to use the secret and deploy containers to. To list available namespaces in your cluster, run kubectl get namespaces.

   <secret_name>

   Required. The name that you want to use for your image pull secret.

   --docker-server <registry_URL>

   Required. The URL to the registry where your private images are stored.

   --docker-username <docker_username>

   Required. The username to log in to your private registry.

   --docker-password <token_value>

   Required. The password to log in to your private registry, such as a token value.

   --docker-email <docker-email>

   Required. If you have one, enter your Docker email address. If you don't have one, enter a fictional email address, such as a@b.c. This email is required to create a Kubernetes secret, but is not used after creation.

2. Verify that the secret was created successfully. Replace <namespace> with the name of the namespace where you created the image pull secret.

   kubectl get secrets --namespace <namespace>
   

3. Create a pod that references the image pull secret.

Storing the image pull secret in the Kubernetes service account for the selected namespace

Every Kubernetes namespace has a Kubernetes service account that is named default. Within the namespace, you can add the image pull secret to this service account to grant access for pods to pull images from your registry. Deployments that don't specify a service account automatically use the default service account for this Kubernetes namespace.

1. Check if an image pull secret already exists for your default service account.

   kubectl describe serviceaccount default -n <namespace_name>
   

   When <none> is displayed in the Image pull secrets entry, no image pull secret exists.

2. Add the image pull secret to your default service account.

   • Example command to add the image pull secret when no image pull secret is defined.

     kubectl patch -n <namespace_name> serviceaccount/default -p '{"imagePullSecrets":[{"name": "<image_pull_secret_name>"}]}'
     

   • Example command to add the image pull secret when an image pull secret is already defined.

     kubectl patch -n <namespace_name> serviceaccount/default --type='json' -p='[{"op":"add","path":"/imagePullSecrets/-","value":{"name":"<image_pull_secret_name>"}}]'
     

3. Verify that your image pull secret was added to your default service account.

   kubectl describe serviceaccount default -n <namespace_name>
   

   Example output

   Name:                default
   Namespace:           <namespace_name>
   Labels:              <none>
   Annotations:         <none>
   Image pull secrets:  <image_pull_secret_name>
   Mountable secrets:   default-token-sh2dx
   Tokens:              default-token-sh2dx
   Events:              <none>
   

   If the Image pull secrets says <secret> (not found), verify that the image pull secret exists in the same namespace as your service account by running kubectl get secrets -n namespace.

4. Create a pod configuration file that is named mypod.yaml to deploy a container from an image in your registry.

   apiVersion: v1
   kind: Pod
   metadata:
     name: mypod
   spec:
     containers:
       - name: mypod-container
         image: <region>.icr.io/<namespace>/<image>:<tag>
   

5. Create the pod in the cluster by applying the mypod.yaml configuration file.

   kubectl apply -f mypod.yaml
   

Example daemonset to update a containerd custom registry configuration

Use the example YAML file to define a daemonset that runs on all worker nodes to set or update a containerd registry host configuration and mount to the corresponding containerd registry path.

The example sets the following registry host configuration for dockerhub. This registry host configuration is already provided and automatically configured during the worker provisioning phase. The init container initializes hosts.toml on every worker node after deployment and after worker nodes reload or restart.

server = "https://docker.io"
[host."https://registry-1.docker.io"]
capabilities = ["pull", "resolve"]

Example YAML file:

apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
    name: containerd-dockerhub-registry-config
name: containerd-dockerhub-registry-config
namespace: kube-system
spec:
selector:
    matchLabels:
    name: containerd-dockerhub-registry-config
template:
    metadata:
    labels:
        name: containerd-dockerhub-registry-config
    spec:
    initContainers:
    - image: alpine:3.13.6
        name: containerd-dockerhub-registry-config
        command:
        - /bin/sh
        - -c
        - |
            #!/bin/sh
            set -uo pipefail
            cat << EOF > /etc/containerd/certs.d/docker.io/hosts.toml
            server = "https://docker.io"
            [host."https://registry-1.docker.io"]
            capabilities = ["pull", "resolve"]
            EOF
        volumeMounts:
        - mountPath: /etc/containerd/certs.d/docker.io/
        name: dockerhub-registry-config
    containers:
    - name: pause
        image: "us.icr.io/armada-master/pause:3.5"
        imagePullPolicy: IfNotPresent
    volumes:
    - name: dockerhub-registry-config
        hostPath:
        path: /etc/containerd/certs.d/docker.io/

For more information on updating a containerd registry host configuration, see the containerd documentation.