Setting up an image registry

Choosing an image registry solution

Your container images must be stored in a container registry that your cluster can access to deploy apps into your cluster. You can choose to use the built-in registry of your Red Hat OpenShift cluster, a private registry with access restricted to select users, or a public registry. Review the following table to choose the best option for your use case.

Internal Red Hat OpenShift Container Registry (OCR)

Your cluster is set up with the internal Red Hat OpenShift Container Registry so that Red Hat OpenShift can automatically build, deploy, and manage your application lifecycle from within the cluster. Images are stored in a backing IBM Cloud classic file storage device that is provisioned at cluster creation time. If you need more storage, you can resize the device. Use cases:

• Red Hat OpenShift-native image stream, build, and app deployment process on a per cluster basis.
• Images can be shared across all projects in the cluster, with access that is controlled through RBAC roles.
• Integrating the internal registry with other Red Hat products like CloudForms for extended features such as vulnerability scanning.
• Option to expose the internal registry with a route so that users can pull images from the registry over the public network.
• Option to set up the internal registry to pull images from or push images to a private registry such as IBM Cloud Container Registry.

For more information, see Using the internal registry.

Private registry

Private registries are a good choice to protect your images from unauthorized users. Private registries must be set up by the cluster administrator to make sure that access, storage quotas, image trust and other features work as intended. By default, your Red Hat OpenShift clusters are integrated with the private IBM Cloud Container Registry through image pull secrets in the default project. IBM Cloud Container Registry is a highly available, multi-tenant private registry to store your own images. You can also pull IBM-provided images from the global icr.io registry, and licensed software from the entitled registry. With IBM Cloud Container Registry, you can manage images for multiple clusters with integration with IBM Cloud IAM and billing. Advantages of using IBM Cloud Container Registry with the internal registry:

• Local image caching for faster builds via the internal registry.
• Deployments in other projects can see the image stream so that you don't have to copy pull secrets to each project.
• Sharing images across multiple clusters without needing to push images to multiple registries.
• Automatically scanning the vulnerability of images.
• Controlling access through IBM Cloud IAM policies and separate regional registries.
• Retaining images without requiring storage space in your cluster or an attached storage device. You can also set policies to manage the quantity of images to prevent them from taking up too much space.
• VPC infrastructure: Using the private registry service endpoint so that clusters that use only a private cloud service endpoint can still access the registry.
• Setting storage and image pull traffic quotas to better control image storage, usage, and billing.
• Pulling licensed IBM content from the entitled registry.

To get started, see the following topics:

• Getting started with IBM Cloud Container Registry.
• Importing images from IBM Cloud Container Registry into the internal registry image stream.
• Using IBM Cloud Container Registry.

Public registry

Public registries such as Docker Hub are a way to share images across teams, companies, clusters, or cloud providers. Some public registries might also offer a private registry component. Use cases:

• Pushing and pulling images on the public network.
• Quick testing of a container across cloud providers.
• Don't need enterprise-grade features such as vulnerability scanning or access management.

For more information, see the public registry's documentation, such as Quay or Docker Hub.

Storing images in the internal registries

Red Hat OpenShift clusters have an internal registry by default. The images in the internal registry are backed up, but vary depending on the infrastructure provider of your Red Hat OpenShift on IBM Cloud cluster.

Classic clusters

Your Red Hat OpenShift cluster is set up by default with an internal registry that uses File Storage for Classic as the backing storage. When you delete the cluster, the internal registry and its images are also deleted. If you want to persist your images, consider using a private registry such as IBM Cloud Container Registry, backing up your images to persistent storage such as Object Storage, or creating a separate, stand-alone Red Hat OpenShift container registry (OCR) cluster. For more information, see the Red Hat OpenShift docs.

VPC clusters

The internal registry of your Red Hat OpenShift cluster backs up your images to a bucket that is automatically created in an IBM Cloud Object Storage instance in your account. Any data that is stored in the object storage bucket remains even if you delete the cluster.

Classic, VPC, or Satellite clusters

You can optionally choose to set up the internal registry to store data in the emptyDir of the worker node where the internal registry pod runs. Keep in mind that this data is not persistent, and if the pod or worker node is restarted, the stored data is deleted and unrecoverable. You might store the images locally in the emptyDir to increase performance if you build containers from large images regularly.

oc describe pvc -n openshift-image-registry image-registry-storage

Removing the internal image registry

Virtual Private Cloud

If you don't want to use the internal image registry, you can complete the following steps to remove it.

1. Save a copy of your internal registry configurations.

   oc get configs.imageregistry.operator.openshift.io cluster -o yaml > configs.yaml
   

2. Run the following patch command to change the management state of the image registry to Removed.

   kubectl patch configs.imageregistry.operator.openshift.io cluster -p '{"spec":{"managementState":"Removed"}}' --type='merge'
   

3. After changing the management state, the image registry service and deployment is removed from the openshift-image-registry namespace in your cluster. You can run the following commands to verify they were removed. Note that only the image registry deployment and service are removed. The image registry operator deployment and service remain.

   oc get deployment -n openshift-image-registry
   

   oc get svc -n openshift-image-registry
   

Setting up a secure external route for the internal registry

By default, your Red Hat OpenShift cluster has an internal registry that is available through a service with an internal IP address. If you want to make the internal registry available on the public network, you can set up a secure re-encrypt route. For example, you might set up your cluster's internal registry to act as a public registry for deployments in other projects or clusters.

Before you begin:

• Confirm that you have the Manager IBM Cloud IAM service access role for the cluster.
• Make sure that your cluster has public network connectivity to expose the internal registry with a public route.
• Install Docker on your local machine.
• Access your Red Hat OpenShift cluster.

To use the internal registry, set up a public route to access the registry. Then, create an image pull secret that includes the credentials to access the registry so that deployments in other projects can pull images from this registry.

1. From the openshift-image-registry project, make sure that the image-registry service exists for the internal registry.

   oc get svc -n openshift-image-registry
   

   Example output

   NAME                      TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                      AGE
   image-registry            ClusterIP      172.21.xxx.xxx    <none>          5000/TCP                     36d
   image-registry-operator   ClusterIP      None             <none>          60000/TCP                     36d
   

2. Create a secured route for the image-registry service that uses reencrypt TLS termination. With re-encryption, the router terminates the TLS connection with a certificate, and then re-encrypts the connection to the internal registry with a different certificate. With this approach, the full path of the connection between the user and the internal registry is encrypted. To provide your own custom domain name, include the --hostname option.

   oc create route reencrypt --service=image-registry -n openshift-image-registry
   

3. Retrieve the hostname (HOST/PORT) and the PORT that were assigned to the image-registry route.

   oc get route image-registry -n openshift-image-registry
   

   Example output

   NAME              HOST/PORT                                                                                                  PATH      SERVICES          PORT       TERMINATION   WILDCARD
   image-registry   image-registry-openshift-image-registry.<cluster_name>-<ID_string>.<region>.containers.appdomain.cloud             image-registry   5000-tcp   reencrypt     None
   

4. Edit the route to set the load balancing strategy to source so that the same client IP address reaches the same server, as in a passthrough route setup. You can set the strategy by adding an annotation in the metadata.annotations section: haproxy.router.openshift.io/balance: source. You can edit the configuration file from the Red Hat OpenShift Application Console or in your command line by running the following command.

   oc edit route image-registry -n openshift-image-registry
   

   Add the annotation.

   apiVersion: route.openshift.io/v1
   kind: Route
   metadata:
   annotations:
       haproxy.router.openshift.io/balance: source
   ...
   

5. If corporate network policies prevent access from your local system to public endpoints via proxies or firewalls, allow access to the route subdomain that you create for the internal registry in the following steps.

6. Log in to the internal registry by using the route as the hostname.

   docker login -u $(oc whoami) -p $(oc whoami -t) image-registry-openshift-image-registry.<cluster_name>-<ID_string>.<region>.containers.appdomain.cloud
   

7. Now that you are logged in, try pushing a sample hello-world app to the internal registry.

   1. Pull the hello-world image from DockerHub, or build an image on your local machine.

      docker pull hello-world
      

   2. Tag the local image with the hostname of your internal registry, the project that you want to deploy the image to, and the image name and tag.

      docker tag hello-world:latest image-registry-openshift-image-registry.<cluster_name>-<ID_string>.<region>.containers.appdomain.cloud/<project>/<image_name>:<tag>
      

   3. Push the image to your cluster's internal registry.

      docker push image-registry-openshift-image-registry.<cluster_name>-<ID_string>.<region>.containers.appdomain.cloud/<project>/<image_name>:<tag>
      

   4. Verify that the image is added to the Red Hat OpenShift image stream.

      oc get imagestream
      

      Example output

      NAME          DOCKER REPO                                                            TAGS      UPDATED
      hello-world   image-registry-openshift-image-registry.svc:5000/default/hello-world   latest    7 hours ago
      

8. To enable deployments in your project to pull images from the internal registry, create an image pull secret in your project that holds the credentials to access your internal registry. Then, add the image pull secret to the default service account for each project.

   1. List the image pull secrets that the default service account uses, and note the secret that begins with default-dockercfg.

      oc describe sa default
      

      Example output

      ...
      Image pull secrets:
      all-icr-io
      default-dockercfg-mpcn4
      ...
      

   2. Get the encoded secret information from the data field of the configuration file.

      oc get secret <default-dockercfg-name> -o yaml
      

      Example output

      apiVersion: v1
      data:
        .dockercfg: ey...=
      

   3. Decode the value of the data field.

      echo "<ey...=>" | base64 -D
      

      Example output

      {"172.21.xxx.xxx:5000":{"username":"serviceaccount","password":"eyJ...
      

   4. Create a new image pull secret for the internal registry.

      • secret_name: Give your image pull secret a name, such as internal-registry.
      • --namespace: Enter the project to create the image pull secret in, such as default.
      • --docker-server: Instead of the internal service IP address (172.21.xxx.xxx:5000), enter the hostname of the image-registry route with the port (image-registry-openshift-image-registry.<cluster_name>-<ID_string>.<region>.containers.appdomain.cloud:5000).
      • --docker-username: Copy the "username" from the previous image pull secret, such as serviceaccount.
      • --docker-password: Copy the "password" from the previous image pull secret.
      • --docker-email: If you have one, enter your Docker email address. If you don't, enter a fictional email address, such as a@b.c. This email is required to create a Kubernetes secret, but is not used after creation.

      oc create secret docker-registry internal-registry --namespace default --docker-server image-registry-openshift-image-registry.<cluster_name>-<ID_string>.<region>.containers.appdomain.cloud:5000 --docker-username serviceaccount --docker-password <eyJ...> --docker-email a@b.c
      

   5. Add the image pull secret to the default service account of your project.

      oc patch -n <namespace_name> serviceaccount/default --type='json' -p='[{"op":"add","path":"/imagePullSecrets/-","value":{"name":"<image_pull_secret_name>"}}]'
      

   6. Repeat these steps for each project that you want to pull images from the internal registry.

Now that you set up the internal registry with an accessible route, you can log in, push, and pull images to the registry. For more information, see the Red Hat OpenShift documentation.

Importing images from IBM Cloud Container Registry into the internal registry image stream

By default, your Red Hat OpenShift on IBM Cloud cluster is set up to pull images from the remote, private IBM Cloud Container Registry icr.io domains in the default project. You can import an image from IBM Cloud Container Registry into the internal registry of your Red Hat OpenShift cluster by tagging the image as an image stream. With this setup, you can deploy apps from the image by using the local cache of the internal registry, which can make your app deployments build faster. Also, deployments in other projects can refer to the image stream so that you don't have to create image pull secret credentials to IBM Cloud Container Registry in each project.

If you update your image in IBM Cloud Container Registry, the image is not pulled automatically into the internal registry of your Red Hat OpenShift cluster. Instead, configure periodic importing, or repeat these steps to tag the image. Depending on the image pull policy that you use in your deployment, you might also need to restart your deployment.

Want to learn more about how builds, image streams, and the internal registry work together? Read the Red Hat OpenShift docs, or check out this blog on managing container images.

1. Access your Red Hat OpenShift cluster.

2. Switch to the default project to pull your image into the image stream. The default project is already set up with credentials to access the icr.io registries.

   oc project default
   

3. List the available images in your IBM Cloud Container Registry. Note the Repository and Tag of the image that you want to pull into the internal registry of your Red Hat OpenShift cluster.

   ibmcloud cr images
   

4. Tag the image to pull it from your IBM Cloud Container Registry namespace into the internal registry as an image stream. For more information, see the Red Hat OpenShift documentation or run oc tag --help.

   oc tag <region>.icr.io/<namespace>/<image>:<tag> default/<image>:<tag> --reference-policy=local [--scheduled]
   

   • <region>.icr.io/<namespace>/<image>:<tag>: Use the Repository and Tag information that you previously retrieved to fill out the IBM Cloud Container Registry region, namespace, image, and tag name of the image that you want to pull.
   • default/<image>:<tag>: Enter the information for the internal image stream that you create from the IBM Cloud Container Registry tagged image. You create this image stream in the default project, which is also the project where the image stream is created if you don't specify a project. The values for <image>:<tag> typically match the values that you previously retrieved.
   • --reference-policy=local: Set this value to local so that a copy of the image from IBM Cloud Container Registry is imported into the local cache of the internal registry and made available to the cluster's projects as an image stream. If you don't include this value, the image stream refers back to IBM Cloud Container Registry when you use it in your deployments and therefore requires credentials in the project.
   • --scheduled: Set this optional option to set up periodic importing of the image from IBM Cloud Container Registry into the internal registry. The default frequency is 15 minutes. For more information, see the Red Hat OpenShift documentation.

5. Verify that your image stream is created.

   oc get imagestreams
   

6. Verify that the image stream successfully pulled the image from IBM Cloud Container Registry. In the output, check that the latest tagged from image matches your * <region>.icr.io/<namespace>/<image>@<digest> image.

   oc describe is/<imagestream>
   

   Example output

   NAME:            <imagestream>
   Namespace:        default
   Created:        2 days ago
   Labels:            <none>
   Annotations:        openshift.io/image.dockerRepositoryCheck=2020-03-31T09:41:36Z
   Image Repository:    image-registry.openshift-image-registry.svc:5000/default/ant1
   Image Lookup:        local=false
   Unique Images:        1
   Tags:                1
   
   latest
       tagged from <region>.icr.io/<namespace>/<image>:<tag>
   
       * <region>.icr.io/<namespace>/<image>@<digest>
           2 days ago
   

Now, your developers can use the image stream in an app deployment. The image successfully builds from the locally pulled image in the internal registry. You don't need to set up an image pull secret in the project to IBM Cloud Container Registry, because the image stream is local to the cluster.

Setting up builds in the internal registry to push images to IBM Cloud Container Registry

When you create a build in your Red Hat OpenShift on IBM Cloud cluster, you can set up the internal registry to push the image to your external repository in IBM Cloud Container Registry. By default, the image pull secret in the default project of your cluster only has read access to pull images from IBM Cloud Container Registry. To push images, you must add a secret with write access.

1. Access your Red Hat OpenShift cluster.

2. Switch to the default project.

   oc project default
   

3. Follow the steps to set up an IBM Cloud IAM API key with the Reader and Writer service access roles to pull from and push images to your icr.io registries.

   Keep in mind that any user with access to the project can use this secret to push images to your private registry. You might want to set up logging and monitoring tools so that you can observe who does what actions in your cluster.

4. Repeat the previous step for each icr.io region that you want to push images to.

5. Add the secret to the build service account and refer to the secrets in the build configuration file. For more information, see the Red Hat OpenShift documentation.

   1. Add the secret to the build service account by linking the secret that you just created to the builder role that all builds in the cluster use.

      oc secrets link builder <secret_name>
      

   2. List the build configurations and note the ones that you want to give push and pull access to IBM Cloud Container Registry.

      oc get bc
      

   3. Set the image push secret for the build configuration to use the secret that you just created with Writer service access to IBM Cloud Container Registry.

      oc set build-secret --push bc/<build_config_name> <secret_name>
      

   4. Set the image pull secret for the build configuration to pull from the registry that you want to pull the initial build image from. For example, you can use the secret that you just created with Reader service access to IBM Cloud Container Registry if the source image is in an IBM Cloud Container Registry repository.

      oc set build-secret --pull bc/<build_config_name> <secret_name>
      

6. After you update the build service account and build configuration file to push to IBM Cloud Container Registry, restart your build.

   oc start-build <build_name>
   

7. Get the name of your build pod, such as <build>-2-build.

   oc get pods
   

8. Check the logs of the build and note where the image was pushed.

   oc logs <build_pod>
   

   Example of a successful image push log.

   ...
   Successfully pushed <region>.icr.io/<namespace>/<build_name>@sha256:<hash>
   Push successful
   

9. Check your images in your private registry to confirm that the image is created.

   ibmcloud cr image list
   

   Example output

   Repository                                Tag       Digest     Namespace     Created         Size     Security status   
   <region>.icr.io/<namespace>/<build_name>  latest    <digest>   <namespace>   2 minutes ago   182 MB   33 Issues  
   

Your Red Hat OpenShift build can now pull images from and push images to IBM Cloud Container Registry.

Using IBM Cloud Container Registry

By default, your Red Hat OpenShift on IBM Cloud cluster is set up to pull images from the remote, private IBM Cloud Container Registry icr.io domains in the default project. If you want to use images that are stored in IBM Cloud Container Registry for other projects, you can pull the image to the internal registry in an image stream, or create image pull secrets for each global and regional registry in each project.

To import images into the internal registry: See Importing images from IBM Cloud Container Registry into the internal registry image stream.

To pull images directly from the external IBM Cloud Container Registry: See the following topics.

• Understanding how to authorize your cluster to pull images from a registry.
• Copying the all-icr-io secret from the default project to the project that you want to pull images from.
• Creating your own image pull secret.
• Adding the image pull secret to your deployment configuration or to the project service account.

Understanding how to authorize your cluster to pull images from a private registry

To pull images from a registry, your Red Hat OpenShift on IBM Cloud cluster uses a special type of Kubernetes secret, an imagePullSecret. This image pull secret stores the credentials to access a container registry.

The container registry can be:

• A private namespace in your own IBM Cloud Container Registry.
• A private namespace in IBM Cloud Container Registry that belongs to a different IBM Cloud account.
• Any other private registry such as Docker.

However, by default, your cluster is set up to pull images from only your account's namespaces in IBM Cloud Container Registry, and deploy containers from these images to the default Red Hat OpenShift project in your cluster. If you need to pull images in other projects of the cluster or from other container registries, then you must set up your own image pull secrets.

Updating existing clusters to use the API key image pull secret

New Red Hat OpenShift on IBM Cloud clusters store an API key in image pull secrets to authorize access to IBM Cloud Container Registry. With these image pull secrets, you can deploy containers from images that are stored in the icr.io registry domains. You can add the image pull secrets to your cluster if your cluster was not created with the secrets.

Before you begin

1. Access your Red Hat OpenShift cluster.

2. Make sure that you have the following permissions: IBM Cloud IAM Operator or Administrator platform access role for Red Hat OpenShift on IBM Cloud. The account owner can give you the role by running the following command.

   ibmcloud iam user-policy-create EMAIL --service-name containers-kubernetes --roles "Administrator,Operator"
   

3. IBM Cloud IAM Administrator platform access role for IBM Cloud Container Registry, across all regions and resource groups. The policy can't be scoped to a particular region or resource group. The account owner can give you the role by running the following command.

   Verify that the secret was created successfully

   ibmcloud iam user-policy-create <your_user_email> --service-name container-registry --roles Administrator
   

4. If your account restricts service ID creation, add the Service ID creator role to Identity and Access Management in the console (iam-identity in the API or CLI).

5. If your account restricts API key creation, add the User API key creator role to Identity and Access Management in the console (iam-identity in the API or CLI).

Using an image pull secret to access images in external private registries

Set up your own image pull secret in your cluster to deploy containers to Red Hat OpenShift projects other than default, use images that are stored in other IBM Cloud accounts, or use images that are stored in external private registries. Further, you might create your own image pull secret to apply IAM access policies that restrict permissions to specific registry image namespaces, or actions (such as push or pull).

After you create the image pull secret, your containers must use the secret to be authorized to pull an image from the registry. You can add the image pull secret to the service account for the project, or refer to the secret in each deployment. For instructions, see Using the image pull secret to deploy containers.

Image pull secrets are valid only for the Red Hat OpenShift projects that they were created for. Repeat these steps for every namespace where you want to deploy containers.

Before you begin:

1. Set up a namespace in IBM Cloud Container Registry and push images to this namespace.
2. Create a cluster.
3. Access your Red Hat OpenShift cluster.

To use your own image pull secret, choose among the following options:

• Copy the image pull secret from the default Red Hat OpenShift project to other projects in your cluster.
• Create new IAM API key credentials and store them in an image pull secret to access images in other IBM Cloud accounts or to apply IAM policies that restrict access to certain registry domains or namespaces.
• Create an image pull secret to access images in external private registries.

If you already created an image pull secret in your project that you want to use in your deployment, see Deploying containers by using the created imagePullSecret.

Using the image pull secret to deploy containers

You can define an image pull secret in your pod deployment or store the image pull secret in your Kubernetes service account so that it is available for all deployments that don't specify a Kubernetes service account in the project.

To plan how image pull secrets are used in your cluster, choose between the following options.

• Referring to the image pull secret in your pod deployment: Use this option if you don't want to grant access to your registry for all pods in your project by default. Developers can include the image pull secret in each pod deployment that must access your registry.
• Storing the image pull secret in the Kubernetes service account: Use this option to grant access to images in your registry for all deployments in the selected Red Hat OpenShift projects. To store an image pull secret in the Kubernetes service account, use the following steps.

Setting up a cluster to pull entitled software

You can set up your Red Hat OpenShift on IBM Cloud cluster to pull entitled software, which is a collection of protected container images that are packaged in Helm charts that you are licensed to use by IBM. Entitled software is stored in a special IBM Cloud Container Registry cp.icr.io domain. To access this domain, you must create an image pull secret with an entitlement key for your cluster and add this image pull secret to the Kubernetes service account of each project where you want to deploy this entitled software.

Before you begin: Access your Red Hat OpenShift cluster.

1. Get the entitlement key for your entitled software library.

   1. Log in to MyIBM.com and scroll to the Container software library section. Click View library.
   2. From the Access your container software > Entitlement keys page, click Copy key. This key authorizes access to all the entitled software in your container software library.

2. In the project that you want to deploy your entitled containers, create an image pull secret so that you can access the cp.icr.io entitled registry. Use the entitlement key that you previously retrieved as the --docker-password value. For more information, see Accessing images that are stored in other private registries.

   oc create secret docker-registry entitled-cp-icr-io --docker-server=cp.icr.io --docker-username=cp --docker-password=<entitlement_key> --docker-email=<docker_email> -n <project>
   

3. Add the image pull secret to the service account of the namespace so that any container in the project can use the entitlement key to pull entitled images. For more information, see Using the image pull secret to deploy containers.

   oc patch -n <project> serviceaccount/default --type='json' -p='[{"op":"add","path":"/imagePullSecrets/-","value":{"name":"entitled-cp-icr-io"}}]'
   

4. Create a pod in the project that builds a container from an image in the entitled registry.

   oc run <pod_name> --image=cp.icr.io/<image_name> -n <project> --generator=run-pod/v1
   

5. Check that your container was able to successfully build from the entitled image by verifying that the pod is in a Running status.

   oc get pod <pod_name> -n <project>
   

Wondering what to do next? You can set up the entitled Helm chart repository, where Helm charts that incorporate entitled software are stored. If you already have Helm installed in your cluster, run helm repo add entitled https://raw.githubusercontent.com/IBM/charts/master/repo/entitled.

Adding a private registry to the global pull secret

With OpenShift Container Platform, you can set up a global image pull secret that each worker node in the cluster can use to pull images from a private registry.

By default, your Red Hat OpenShift on IBM Cloud cluster has a global image pull secret for the following registries, so that default Red Hat OpenShift components can be deployed.

• cloud.openshift.com
• quay.io
• registry.connect.redhat.com
• registry.redhat.io

Do not replace the global pull secret with a pull secret that does not have credentials to the default Red Hat registries. If you do, the default Red Hat OpenShift components that are installed in your cluster, such as the OperatorHub, might fail because they can't pull images from these registries.

Before you begin:

• Download the jq JSON processor command line package. You use jq to combine the JSON value of the default global pull secret with the private registry pull secret that you want to add.
• Access your Red Hat OpenShift cluster.

To add private registries, edit the global pull-secret in the openshift-config project.

1. Create a secret value that holds the credentials to access your private registry and store the decoded secret value in a JSON file. When you create the secret value, the credentials are automatically encoded to base64. By using the --dry-run option, the secret value is created only and no secret object is created in your cluster. The decoded secret value is then stored in a JSON file to later use in your global pull secret.

   oc create secret docker-registry <secret_name> --docker-server=<registry_URL> --docker-username=<docker_username> --docker-password=<docker_password> --docker-email=<docker_email> --dry-run=true --output="jsonpath={.data.\.dockerconfigjson}" | base64 --decode > myregistryconfigjson
   

   --namespace <project>

   Required. The Red Hat OpenShift project of your cluster where you want to use the secret and deploy containers to. To list available projects in your cluster, run oc get ns.

   <secret_name>

   Required. The name that you want to use for your image pull secret.

   --docker-server <registry_URL>

   Required. The URL to the registry where your private images are stored.

   --docker-username <docker_username>

   Required. The username to log in to your private registry.

   --docker-password <token_value>

   Required. The password to log in to your private registry, such as a token value.

   --docker-email <docker-email>

   Required. If you have one, enter your Docker email address. If you don't have one, enter a fictional email address, such as a@b.c. This email is required to create a Kubernetes secret, but is not used after creation.

   --dry-run=true

   Include this option to create the secret value only, and not create and store the secret object in your cluster.

   --output="jsonpath={.data.\.dockerconfigjson}"

   Get only the .dockerconfigjson value from the data section of the Kubernetes secret.

   | base64 --decode > myregistryconfigjson

   Download the decoded secret data to a local myregistryconfigjson file.

2. Retrieve the decoded secret value of the default global pull secret and store the value in a dockerconfigjson file.

   oc get secret pull-secret -n openshift-config --output="jsonpath={.data.\.dockerconfigjson}" | base64 --decode > dockerconfigjson
   

3. Combine the downloaded private registry pull secret myregistryconfigjson file with the default global pull secret dockerconfigjson file.

   jq -s '.[0] * .[1]' dockerconfigjson myregistryconfigjson > dockerconfigjson-merged
   

4. Update the global pull secret with the combined dockerconfigjson-merged file.

   oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=dockerconfigjson-merged
   

5. Verify that the global pull secret is updated. Check that your private registry and each of the default Red Hat registries are in the output of the following command.

   oc get secret pull-secret -n openshift-config --output="jsonpath={.data.\.dockerconfigjson}" | base64 --decode
   

   Example output

   {
       "auths": {
           "cloud.openshift.com": {
               "auth": "<encoded_string>",
               "email": "email@example.com"
           },
           "quay.io": {
               "auth": "<encoded_string>",
               "email": "email@example.com"
           },
           "registry.connect.redhat.com": {
               "auth": "<encoded_string>",
               "email": "email@example.com"
           },
           "registry.redhat.io": {
               "auth": "<encoded_string>",
               "email": "email@example.com"
           },
           "<private_registry>": {
               "username": "iamapikey",
               "password": "<encoded_string>",
               "email": "email@example.com",
               "auth": "<encoded_string>"
           }
       }
   }
   

6. To pick up the global configuration changes, reload all the worker nodes in your cluster.

   1. Note the ID of the worker nodes in your cluster.

      ibmcloud oc worker ls -c <cluster_name_or_ID>
      

   2. For Classic clusters Reload each worker node. You can reload multiple worker nodes by including multiple -w options, but make sure to leave enough worker nodes running at the same time for your apps to avoid an outage.

      ibmcloud oc worker reload -c <cluster_name_or_ID> -w <workerID_1> -w <workerID_2>
      

   3. For VPC clusters Replace each worker node. Before you begin, make sure that your cluster has enough other worker nodes so that your pods can be rescheduled and continue to run.

      ibmcloud oc worker replace --cluster <cluster_name_or_ID> --worker <worker_node_ID>
      

7. After the worker nodes are back in a healthy state, verify that the global pull secret is updated on a worker node.

   1. Start a debugging pod to log in to a worker node. Use the Private IP that you retrieved earlier for the <node_name>.

      oc debug node/<node_name>
      

   2. Change the root directory to the host so that you can view files on the worker node.

      chroot /host
      

   3. Verify that the Docker configuration file has the registry credentials that match the global pull secret that you set.

      vi /.docker/config.json
      

Updating the global pull secret

Complete the following steps to update the global pull secret in your Red Hat OpenShift on IBM Cloud cluster.

1. Create a secret that has the credentials for the registry you want to use.

   oc create secret docker-registry docker-auth-secret \
   --docker-server=REGISTRY \
   --docker-username=USERNAME \
   --docker-password=PASSWORD \
   --namespace kube-system
   

   Example create secret command.

   oc create secret docker-registry docker-auth-secret \
   --docker-server=REGISTRY \
   --docker-username=cp \
   --docker-password=ENTITLEMENT-KEY \
   --namespace kube-system
   

2. Create a DaemonSet to apply the secret across all worker nodes.

   cat << EOF | oc create -f -
   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: update-docker-config
     namespace: kube-system
     labels:
       app: update-docker-config
   spec:
     selector:
       matchLabels:
         name: update-docker-config
     template:
       metadata:
         labels:
           name: update-docker-config
       spec:
         initContainers:
           - command: ["/bin/sh", "-c"]
             args:
               - >
                 echo "Checking if RHEL or RHCOS host";
                 [[ -s /docker-config/.docker/config.json  ]] && CONFIG_PATH=/docker-config/.docker || CONFIG_PATH=/docker-config/root/.docker;
                 echo "Backing up or restoring config.json";
                 [[ -s \$CONFIG_PATH/config.json ]] && cp \$CONFIG_PATH/config.json \$CONFIG_PATH/config.json.bak || cp \$CONFIG_PATH/config.json.bak \$CONFIG_PATH/config.json;
                 echo "Merging secret with config.json";
                 /host/usr/bin/jq -s '.[0] * .[1]' \$CONFIG_PATH/config.json /auth/.dockerconfigjson > \$CONFIG_PATH/config.tmp;
                 mv \$CONFIG_PATH/config.tmp \$CONFIG_PATH/config.json;
                 echo "Sending signal to reload crio config";
                 pidof crio;
                 kill -1 \$(pidof crio)
             image: icr.io/ibm/alpine:latest
             imagePullPolicy: IfNotPresent
             name: updater
             resources: {}
             securityContext:
               privileged: true
             volumeMounts:
               - name: docker-auth-secret
                 mountPath: /auth
               - name: docker
                 mountPath: /docker-config
               - name: bin
                 mountPath: /host/usr/bin
               - name: lib64
                 mountPath: /lib64
         containers:
           - resources:
               requests:
                 cpu: 0.01
             image: icr.io/ibm/alpine:latest
             name: sleepforever
             command: ["/bin/sh", "-c"]
             args:
               - >
                 while true; do
                   sleep 100000;
                 done
         hostPID: true
         volumes:
           - name: docker-auth-secret
             secret:
               secretName: docker-auth-secret
           - name: docker
             hostPath:
               path: /
           - name: bin
             hostPath:
               path: /usr/bin
           - name: lib64
             hostPath:
               path: /lib64
               hostPathType: Directory
   EOF
   

3. Verify the pods are running.

   oc get daemonset -n kube-system update-docker-config
   

Updating an IBM Cloud Kubernetes Service containerd custom registry configuration

With Kubernetes version 1.22 or later, you can use containerd configuration files on worker nodes to configure pulling from a container registry. You can use a daemonset to update the configurations across all nodes in a cluster, which prevents configurations from being wiped when worker nodes reload or when new workers are added.

server = "https://docker.io"
[host."https://registry-1.docker.io"]
capabilities = ["pull", "resolve"]

apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
    name: containerd-dockerhub-registry-config
name: containerd-dockerhub-registry-config
namespace: kube-system
spec:
selector:
    matchLabels:
    name: containerd-dockerhub-registry-config
template:
    metadata:
    labels:
        name: containerd-dockerhub-registry-config
    spec:
    initContainers:
    - image: alpine:3.13.6
        name: containerd-dockerhub-registry-config
        command:
        - /bin/sh
        - -c
        - |
            #!/bin/sh
            set -uo pipefail
            cat << EOF > /etc/containerd/certs.d/docker.io/hosts.toml
            server = "https://docker.io"
            [host."https://registry-1.docker.io"]
            capabilities = ["pull", "resolve"]
            EOF
        volumeMounts:
        - mountPath: /etc/containerd/certs.d/docker.io/
        name: dockerhub-registry-config
    containers:
    - name: pause
        image: "us.icr.io/armada-master/pause:3.5"
        imagePullPolicy: IfNotPresent
    volumes:
    - name: dockerhub-registry-config
        hostPath:
        path: /etc/containerd/certs.d/docker.io/