--- name: openshift-registry title: Setting up an image registry description: Red Hat® OpenShift® on IBM Cloud® clusters include an internal registry to build, deploy, and manage container images locally. For a private registry to manage and control access to images across your enterprise, you can also set up your cluster to use IBM Cloud® Container Registry. last-updated: 2026-08-13 --- > ## Documentation Index > The table of contents for this documentation set is at https://cloud.ibm.com/docs/openshift?format=markdown > The index for all IBM Cloud docs is at: https://cloud.ibm.com/docs/llms.txt > Use these files to discover more information as needed. # Setting up an image registry {: #registry} Red Hat® OpenShift® on IBM Cloud® clusters include an internal registry to build, deploy, and manage container images locally. For a private registry to manage and control access to images across your enterprise, you can also set up your cluster to use IBM Cloud® Container Registry. {: shortdesc} ## Choosing an image registry solution {: #openshift_registry_options} Your container images must be stored in a container registry that your cluster can access to deploy apps into your cluster. You can choose to use the built-in registry of your Red Hat OpenShift cluster, a private registry with access restricted to select users, or a public registry. Review the following table to choose the best option for your use case. {: shortdesc} Internal Red Hat OpenShift Container Registry (OCR) : Your cluster is set up with the internal Red Hat OpenShift Container Registry so that Red Hat OpenShift can automatically build, deploy, and manage your application lifecycle from within the cluster. Images are stored in a backing IBM Cloud classic file storage device that is provisioned at cluster creation time. If you need more storage, you can resize the device. Use cases: - Red Hat OpenShift-native image stream, build, and app deployment process on a per cluster basis. - Images can be shared across all projects in the cluster, with access that is controlled through RBAC roles. - Integrating the internal registry with other Red Hat products like CloudForms for extended features such as vulnerability scanning. - Option to expose the internal registry with a route so that users can pull images from the registry over the public network. - Option to set up the internal registry to [pull images](#imagestream_registry) from or [push images](#builds_registry) to a private registry such as IBM Cloud Container Registry. : For more information, see [Using the internal registry](#openshift_internal_registry). Private registry : Private registries are a good choice to protect your images from unauthorized users. Private registries must be set up by the cluster administrator to make sure that access, storage quotas, image trust and other features work as intended. By default, your [Red Hat OpenShift clusters](#openshift_iccr) are integrated with the private IBM Cloud Container Registry through image pull secrets in the `default` project. IBM Cloud Container Registry is a highly available, multi-tenant private registry to store your own images. You can also pull IBM-provided images from the global `icr.io` registry, and licensed software from the entitled registry. With IBM Cloud Container Registry, you can manage images for multiple clusters with integration with IBM Cloud IAM and billing. Advantages of using IBM Cloud Container Registry with the internal registry: - Local image caching for faster builds via the internal registry. - Deployments in other projects can see the image stream so that you don't have to copy pull secrets to each project. - Sharing images across multiple clusters without needing to push images to multiple registries. - [Automatically scanning](https://cloud.ibm.com/docs/Registry?topic=Registry-va_index&format=markdown) the vulnerability of images. - Controlling access through [IBM Cloud IAM policies](https://cloud.ibm.com/docs/Registry?topic=Registry-user&format=markdown) and [separate regional registries](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overview&format=markdown#registry_regions). - [Retaining images](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_retention&format=markdown) without requiring storage space in your cluster or an attached storage device. You can also set policies to manage the quantity of images to prevent them from taking up too much space. - VPC infrastructure: Using the private registry service endpoint so that clusters that use only a private cloud service endpoint can still access the registry. - [Setting storage and image pull traffic quotas](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_quota&format=markdown) to better control image storage, usage, and billing. - Pulling licensed IBM content from the [entitled registry](https://cloud.ibm.com/docs/openshift?topic=openshift-registry&format=markdown#secret_entitled_software). : To get started, see the following topics: - [Getting started with IBM Cloud Container Registry](https://cloud.ibm.com/docs/Registry?topic=Registry-getting-started&format=markdown). - [Importing images from IBM Cloud Container Registry](#imagestream_registry) into the internal registry image stream. - [Using IBM Cloud Container Registry](#openshift_iccr). Public registry : Public registries such as Docker Hub are a way to share images across teams, companies, clusters, or cloud providers. Some public registries might also offer a private registry component. Use cases: - Pushing and pulling images on the public network. - Quick testing of a container across cloud providers. - Don't need enterprise-grade features such as vulnerability scanning or access management. : For more information, see the public registry's documentation, such as [Quay](https://quay.io/){: external} or [Docker Hub](https://hub.docker.com/){: external}. ## Storing images in the internal registries {: #openshift_internal_registry} Red Hat OpenShift clusters have an internal registry by default. The images in the internal registry are backed up, but vary depending on the infrastructure provider of your Red Hat OpenShift on IBM Cloud cluster. {: shortdesc} Classic clusters : Your Red Hat OpenShift cluster is set up by default with an internal registry that uses File Storage for Classic as the backing storage. When you delete the cluster, the internal registry and its images are also deleted. If you want to persist your images, consider using a private registry such as IBM Cloud Container Registry, backing up your images to persistent storage such as Object Storage, or creating a separate, stand-alone Red Hat OpenShift container registry (OCR) cluster. For more information, see the [Red Hat OpenShift docs](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/registry/registry-overview){: external}. VPC clusters : The internal registry of your Red Hat OpenShift cluster backs up your images to a bucket that is automatically created in an IBM Cloud Object Storage instance in your account. Any data that is stored in the object storage bucket remains even if you delete the cluster. After the cluster is created, you cannot remove or detach the IBM Cloud Object Storage bucket from the cluster. If you are planning to create a cluster without a COS-backed internal registry, such as for [IBM Financial Services Cloud](https://cloud.ibm.com/docs/framework-financial-services?format=markdown){: external} environments, you must omit the COS instance at cluster creation time. For more information, see [Creating a VPC cluster without an Object Storage bucket](https://cloud.ibm.com/docs/openshift?topic=openshift-tutorial-no-registry-cos&format=markdown). Classic, VPC, or Satellite clusters : You can optionally choose to set up the internal registry to store data in the `emptyDir` of the worker node where the internal registry pod runs. Keep in mind that this data is not persistent, and if the pod or worker node is restarted, the stored data is deleted and unrecoverable. You might store the images locally in the `emptyDir` to increase performance if you build containers from large images regularly. ### Backing up your internal image registry to IBM Cloud Object Storage {: #cos_image_registry} [Virtual Private Cloud]{: tag-vpc} Your images in your Red Hat OpenShift cluster internal registry are automatically backed up to an IBM Cloud Object Storage bucket. Any data that is stored in the object storage bucket remains even if you delete the cluster. {: shortdesc} However, if the bucket fails to create when you create your cluster, you must manually create a bucket and set up your cluster to use the bucket. In the meantime, the internal registry uses an `emptyDir` Kubernetes volume that stores your container images on the secondary disk of your worker node. The `emptyDir` volumes are not considered persistent highly available storage, and if you delete the pods that use the image, the image is automatically deleted. To manually create a bucket for your internal registry, see [Cluster create error about cloud object storage bucket](https://cloud.ibm.com/docs/openshift?topic=openshift-ts_cos_bucket_cluster_create&format=markdown). ### Configuring a COS bucket for your cluster registry {: #configure-cos-registry} You can manually configure an IBM Cloud Object Storage bucket for your cluster's internal image registry. This is useful when you need to set up or reconfigure the registry storage backend. {: shortdesc} #### Before you begin {: #cos-registry-prereqs} * You must have an existing Red Hat OpenShift on IBM Cloud cluster and an IBM Cloud Object Storage instance. * [Access your Red Hat OpenShift cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-access_cluster&format=markdown). #### Configuring the COS bucket {: #cos-registry-steps} 1. Set environment variables for your cluster and Object Storage instance. ```sh export CLUSTER_NAME= export COS_INSTANCE_ID= export CLUSTER_ID=$(ibmcloud ks cluster get --cluster $CLUSTER_NAME --json | jq -r .id) ``` {: pre} 1. Check the current image registry operator configuration. ```sh oc get configs.imageregistry.operator.openshift.io/cluster -o yaml ``` {: pre} 1. Temporarily set the operator to `Removed` mode. If the registry is already active, disable it to avoid conflicts. ```sh oc patch configs.imageregistry.operator.openshift.io/cluster \ --type=merge -p '{"spec":{"managementState":"Removed"}}' ``` {: pre} 1. Check if the registry uses `emptyDir`, which means no COS bucket is configured. ```yaml storage: emptyDir: {} managementState: Managed ``` {: codeblock} 1. Create an Object Storage bucket for your cluster. Replace `` with your preferred region, such as `us-south` or `eu-de`. ```sh RANDOM_SUFFIX=$(LC_ALL=C tr -dc 'a-z0-9' ``` {: pre} 1. Remove the `emptyDir` field from the registry configuration. ```sh oc patch configs.imageregistry.operator.openshift.io/cluster \ --type=json \ -p '[{"op": "remove", "path": "/spec/storage/emptyDir"}]' ``` {: pre} 1. Configure the registry to use S3 storage (Object Storage). Update the `region` and `regionEndpoint` values to match your bucket's region. ```sh oc patch configs.imageregistry.operator.openshift.io/cluster --type=merge -p "$(cat <-standard", "regionEndpoint": "https://s3.direct..cloud-object-storage.appdomain.cloud", "trustedCA": { "name": "" }, "virtualHostedStyle": false } } } } EOF )" ``` {: pre} For example, if your bucket is in `eu-de`, use `region: "eu-de-standard"` and `regionEndpoint: "https://s3.direct.eu-de.cloud-object-storage.appdomain.cloud"`. {: tip} 1. Verify the image registry operator configuration. ```sh oc get configs.imageregistry.operator.openshift.io/cluster -o yaml ``` {: pre} 1. Create a service credential key for the COS instance and extract the HMAC credentials. ```sh ibmcloud resource service-key-create roks-$CLUSTER_ID-key \ --instance-id "$COS_INSTANCE_ID" \ -p '{"HMAC": true}' \ -o json > service-key.json # Extract values with jq ACCESS_KEY=$(jq -r '.credentials.cos_hmac_keys.access_key_id' service-key.json) SECRET_KEY=$(jq -r '.credentials.cos_hmac_keys.secret_access_key' service-key.json) # Create credentials file cat < creds.txt [default] aws_access_key_id = $ACCESS_KEY aws_secret_access_key = $SECRET_KEY EOF ``` {: pre} This creates a `creds.txt` file with your Object Storage credentials. {: note} Example credentials file ```sh [default] aws_access_key_id = f1ab5fcf7XXXXXXXXXb2a14e046 aws_secret_access_key = 4c78ddfa763XXXXXXX23d1cc9350d1 ``` {: screen} 1. Create the `image-registry-private-configuration-user` secret if it does not exist. ```sh oc create secret generic image-registry-private-configuration-user -n openshift-image-registry ``` {: pre} 1. Update the secret with the access credentials. ```sh oc patch secret image-registry-private-configuration-user \ -n openshift-image-registry \ --type merge \ -p "{ \"data\": { \"REGISTRY_STORAGE_S3_ACCESSKEY\": \"$(echo -n "$ACCESS_KEY" | base64)\", \"REGISTRY_STORAGE_S3_SECRETKEY\": \"$(echo -n "$SECRET_KEY" | base64)\" } }" ``` {: pre} 1. Create the `image-registry-private-configuration` secret if it does not exist. ```sh oc create secret generic image-registry-private-configuration -n openshift-image-registry ``` {: pre} 1. Update the main image registry secret with the full Object Storage credentials file. ```sh CREDENTIALS_B64=$(base64 -i creds.txt | tr -d '\n') oc patch secret image-registry-private-configuration \ -n openshift-image-registry \ --type merge \ -p "{\"data\": {\"credentials\": \"${CREDENTIALS_B64}\"}}" ``` {: pre} 1. Set the image registry operator to `Managed` mode. ```sh oc patch configs.imageregistry.operator.openshift.io/cluster \ --type=merge -p '{"spec":{"managementState":"Managed"}}' ``` {: pre} 1. Restart the image registry deployment. ```sh oc rollout restart deployment image-registry -n openshift-image-registry ``` {: pre} 1. Verify that the registry is in a healthy state. ```sh oc get clusteroperator image-registry ``` {: pre} ### Troubleshooting COS registry configuration {: #cos-registry-troubleshooting} If you encounter issues with your Object Storage registry configuration, try the following troubleshooting steps. 1. Check that the `image-registry` pod is running. ```sh oc project openshift-image-registry oc get pods ``` {: pre} Example output ```sh NAME READY STATUS RESTARTS AGE image-registry-7d8b57c98f-m92t7 1/1 Running 0 3h53m node-ca-62p88 1/1 Running 0 5h12m node-ca-6s8vx 1/1 Running 0 5h17m node-ca-b5jwl 1/1 Running 0 4h56m node-ca-chl7k 1/1 Running 0 5h17m node-ca-l6tlb 1/1 Running 0 4h55m node-ca-q4vw8 1/1 Running 0 4h55m ``` {: screen} 1. Check the cluster operator `image-registry` status. ```sh oc describe clusteroperator image-registry ``` {: pre} 1. If needed, temporarily remove the S3 backend configuration and revert to `emptyDir` to unlock the operator. ```sh oc patch configs.imageregistry.operator.openshift.io/cluster --type=merge -p '{ "spec": { "managementState": "Managed", "storage": { "emptyDir": {} } } }' ``` {: pre} 1. Check the pods in the `openshift-image-registry` namespace. ```sh oc get pods -n openshift-image-registry ``` {: pre} 1. Check the deployed configuration environment variables. ```sh oc get pod -n openshift-image-registry -l docker-registry=default -o jsonpath='{.items[0].spec.containers[0].env}' | jq ``` {: pre} For more information about configuring the registry, see [Setting up and configuring the registry](https://docs.redhat.com/en/documentation/openshift_container_platform/4.7/html/registry/setting-up-and-configuring-the-registry#configuring-registry-storage-aws-user-infrastructure){: external}. ### Storing images in the internal registry in Classic clusters {: #storage_internal_registry} By default, your Red Hat OpenShift cluster's internal registry uses an [IBM Cloud File Storage for Classic](https://cloud.ibm.com/docs/openshift?topic=openshift-file_storage&format=markdown) volume to store the registry images. You can review the default size of the storage volume, or update the volume size. {: shortdesc} #### Viewing volume details {: #view_volume_details_registry} To view volume details including the storage class and size, you can describe the persistent volume claim. ```sh oc describe pvc -n openshift-image-registry image-registry-storage ``` {: pre} #### Changing volume details {: #change_volume_details_registry} If your registry needs additional gigabytes of storage for your images, you can resize the file storage volume. For more information, see [Changing the size and IOPS of your existing storage device](https://cloud.ibm.com/docs/openshift?topic=openshift-file_storage&format=markdown#file_change_storage_configuration). When you resize the volume in your IBM Cloud infrastructure account, the attached PVC description is not updated. Instead, you can log in to the `openshift-image-registry` pod that uses the `registry-backing` PVC to verify that the volume is resized. ### Storing images in the worker node empty directory {: #emptydir_internal_registry} You might store the internal registry images locally in the `emptyDir` of the worker node, such as a bare metal worker node, to increase performance if you build containers from large images regularly. {: shortdesc} Keep in mind that this data is not persistent, and if the pod or worker node is restarted, the stored data is deleted and unrecoverable. {: important} 1. [Access your Red Hat OpenShift cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-access_cluster&format=markdown). 1. [Update the image registry operator configmap](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/storage/persistent-storage-using-local-storage){: external} to set the storage to use the `emptyDir` of the worker node. Note that updating the configmap to use `emptyDir` does **not** remove the image registry's original PVC. ```sh oc patch configs.imageregistry.operator.openshift.io/cluster --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}' ``` {: pre} 1. If the [image registry operator management state](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/registry/configuring-registry-operator#registry-operator-configuration-resource-overview_configuring-registry-operator){: external} is set to `Unmanaged`, such as in Satellite clusters, update the management state to `Managed`. Now, the operator updates the internal registry pod. ```sh oc patch configs.imageregistry.operator.openshift.io/cluster --type merge -p '{"spec":{"managementState":"Managed"}}' ``` {: pre} 4. Get the details of the internal registry pod so that you can verify your updates. 1. Check that the `image-registry` pod is running and that a pod runs per worker node in the cluster. ```sh oc get pods -n openshift-image-registry ``` {: pre} Example output ```sh NAME READY STATUS RESTARTS AGE cluster-image-registry-operator-695bf78ffc-zvkhd 2/2 Running 0 33m image-registry-6774598589-65cnx 1/1 Running 0 112s node-ca-gg66r 1/1 Running 0 113s node-ca-n8jpq 1/1 Running 0 113s node-ca-p2d7j 1/1 Running 0 113s ``` {: screen} 1. Get the public IP address of the **Node** that the `image-registry` pod runs on. ```sh oc describe pod -n openshift-image-registry | grep Node ``` {: pre} Example output ```sh Node: 169.xx.xxx.xxx/169.xx.xxx.xxx ``` {: screen} If the worker node IP address is private, run `ibmcloud oc worker ls -c | grep ` and note the corresponding public IP address. {: tip} 1. Get the **UID** of the `image-registry` pod in the `metadata.uid` section in the pod YAML (not the UID of the replica set in the `metadata.ownerReferences.uid` section). ```sh oc get pod -n openshift-image-registry -o yaml ``` {: pre} Example output ```yaml apiVersion: v1 kind: Pod metadata: uid: e8d7718d-b0bd-47e2-9aaa-05f3a608fd9b ... ``` {: screen} 5. Verify that the internal registry stores data to the `emptyDir` of the worker node. 1. [Access the registry directly from the cluster](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/registry/accessing-the-registry#registry-accessing-directly_accessing-the-registry){: external}, using the worker node that you previously retrieved. Follow the steps to push a test image to the internal registry. To complete these steps in the Red Hat OpenShift documentation, you need the `podman` CLI tool. Your worker nodes might not have this CLI tool by default. See the [Podman installation guide](https://podman.io/get-started){: external} for the available RHEL versions. {: tip} 1. Navigate to the internal registry pod folder that saves to the `emptyDir`. For ``, use the pod **UID** that you retrieved earlier. ```sh cd var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/registry-storage/docker/registry/v2/repositories/openshift ``` {: pre} 1. Verify that your image is in repository directory. ```sh ls ``` {: pre} Example output ```sh nginx ... ``` {: screen} ## Removing the internal image registry {: #remove-image-registry} [Virtual Private Cloud]{: tag-vpc} The image registry operator deployments are present only in an Red Hat OpenShift on IBM Cloud Toolkit managed clusters. HyperShift managed clusters run the image registry operator in the control plane. {: note} If you don't want to use the internal image registry, you can complete the following steps to remove it. 1. Save a copy of your internal registry configurations. ```sh oc get configs.imageregistry.operator.openshift.io cluster -o yaml > configs.yaml ``` {: pre} 1. Run the following patch command to change the management state of the image registry to `Removed`. ```sh kubectl patch configs.imageregistry.operator.openshift.io cluster -p '{"spec":{"managementState":"Removed"}}' --type='merge' ``` {: pre} 1. After changing the management state, the image registry service and deployment is removed from the `openshift-image-registry` namespace in your cluster. You can run the following commands to verify they were removed. Note that only the image registry deployment and service are removed. The image registry operator deployment and service remain. ```sh oc get deployment -n openshift-image-registry ``` {: pre} ```sh oc get svc -n openshift-image-registry ``` {: pre} ## Setting up a secure external route for the internal registry {: #route_internal_registry} By default, your Red Hat OpenShift cluster has an internal registry that is available through a service with an internal IP address. If you want to make the internal registry available on the public network, you can set up a secure re-encrypt route. For example, you might set up your cluster's internal registry to act as a public registry for deployments in other projects or clusters. {: shortdesc} Before you begin: * Confirm that you have the [**Manager** IBM Cloud IAM service access role](https://cloud.ibm.com/docs/openshift?topic=openshift-iam-platform-access-roles&format=markdown) for the cluster. * Make sure that your cluster has public network connectivity to expose the internal registry with a public route. * Install Docker on your local machine. * [Access your Red Hat OpenShift cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-access_cluster&format=markdown). To use the internal registry, set up a public route to access the registry. Then, create an image pull secret that includes the credentials to access the registry so that deployments in other projects can pull images from this registry. 1. From the `openshift-image-registry` project, make sure that the `image-registry` service exists for the internal registry. ```sh oc get svc -n openshift-image-registry ``` {: pre} Example output ```sh NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE image-registry ClusterIP 172.21.xxx.xxx 5000/TCP 36d image-registry-operator ClusterIP None 60000/TCP 36d ``` {: screen} 1. Create a secured route for the `image-registry` service that uses `reencrypt` TLS termination. With re-encryption, the router terminates the TLS connection with a certificate, and then re-encrypts the connection to the internal registry with a different certificate. With this approach, the full path of the connection between the user and the internal registry is encrypted. To provide your own custom domain name, include the `--hostname` option. ```sh oc create route reencrypt --service=image-registry -n openshift-image-registry ``` {: pre} 1. Retrieve the hostname (**HOST/PORT**) and the **PORT** that were assigned to the `image-registry` route. ```sh oc get route image-registry -n openshift-image-registry ``` {: pre} Example output ```sh NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD image-registry image-registry-openshift-image-registry.-..containers.appdomain.cloud image-registry 5000-tcp reencrypt None ``` {: screen} 4. Edit the route to set the [load balancing strategy](https://docs.redhat.com/en/documentation/openshift_container_platform/3.11/html/architecture/networking#architecture-core-concepts-routes){: external} to `source` so that the same client IP address reaches the same server, as in a passthrough route setup. You can set the strategy by adding an annotation in the `metadata.annotations` section: `haproxy.router.openshift.io/balance: source`. You can edit the configuration file from the **Red Hat OpenShift Application Console** or in your command line by running the following command. ```sh oc edit route image-registry -n openshift-image-registry ``` {: pre} Add the annotation. ```yaml apiVersion: route.openshift.io/v1 kind: Route metadata: annotations: haproxy.router.openshift.io/balance: source ... ``` {: screen} 5. If corporate network policies prevent access from your local system to public endpoints via proxies or firewalls, [allow access to the route subdomain](https://cloud.ibm.com/docs/openshift?topic=openshift-vpc-firewall&format=markdown#openshift-registry) that you create for the internal registry in the following steps. 6. Log in to the internal registry by using the route as the hostname. ```sh docker login -u $(oc whoami) -p $(oc whoami -t) image-registry-openshift-image-registry.-..containers.appdomain.cloud ``` {: pre} 7. Now that you are logged in, try pushing a sample `hello-world` app to the internal registry. 1. Pull the `hello-world` image from DockerHub, or build an image on your local machine. ```sh docker pull hello-world ``` {: pre} 1. Tag the local image with the hostname of your internal registry, the project that you want to deploy the image to, and the image name and tag. ```sh docker tag hello-world:latest image-registry-openshift-image-registry.-..containers.appdomain.cloud//: ``` {: pre} 1. Push the image to your cluster's internal registry. ```sh docker push image-registry-openshift-image-registry.-..containers.appdomain.cloud//: ``` {: pre} 4. Verify that the image is added to the Red Hat OpenShift image stream. ```sh oc get imagestream ``` {: pre} Example output ```sh NAME DOCKER REPO TAGS UPDATED hello-world image-registry-openshift-image-registry.svc:5000/default/hello-world latest 7 hours ago ``` {: screen} 8. To enable deployments in your project to pull images from the internal registry, create an image pull secret in your project that holds the credentials to access your internal registry. Then, add the image pull secret to the default service account for each project. 1. List the image pull secrets that the default service account uses, and note the secret that begins with `default-dockercfg`. ```sh oc describe sa default ``` {: pre} Example output ```sh ... Image pull secrets: all-icr-io default-dockercfg-mpcn4 ... ``` {: screen} 1. Get the encoded secret information from the `data` field of the configuration file. ```sh oc get secret -o yaml ``` {: pre} Example output ```yaml apiVersion: v1 data: .dockercfg: ey...= ``` {: screen} 1. Decode the value of the `data` field. ```sh echo "" | base64 -D ``` {: pre} Example output ```sh {"172.21.xxx.xxx:5000":{"username":"serviceaccount","password":"eyJ... ``` {: screen} 4. Create a new image pull secret for the internal registry. - `secret_name`: Give your image pull secret a name, such as `internal-registry`. - `--namespace`: Enter the project to create the image pull secret in, such as `default`. - `--docker-server`: Instead of the internal service IP address (`172.21.xxx.xxx:5000`), enter the hostname of the `image-registry` route with the port (`image-registry-openshift-image-registry.-..containers.appdomain.cloud:5000`). - `--docker-username`: Copy the `"username"` from the previous image pull secret, such as `serviceaccount`. - `--docker-password`: Copy the `"password"` from the previous image pull secret. - `--docker-email`: If you have one, enter your Docker email address. If you don't, enter a fictional email address, such as `a@b.c`. This email is required to create a Kubernetes secret, but is not used after creation. ```sh oc create secret docker-registry internal-registry --namespace default --docker-server image-registry-openshift-image-registry.-..containers.appdomain.cloud:5000 --docker-username serviceaccount --docker-password --docker-email a@b.c ``` {: pre} 5. Add the image pull secret to the default service account of your project. ```sh oc patch -n serviceaccount/default --type='json' -p='[{"op":"add","path":"/imagePullSecrets/-","value":{"name":""}}]' ``` {: pre} 6. Repeat these steps for each project that you want to pull images from the internal registry. Now that you set up the internal registry with an accessible route, you can log in, push, and pull images to the registry. For more information, see the [Red Hat OpenShift documentation](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/registry/accessing-the-registry){: external}. ## Importing images from IBM Cloud Container Registry into the internal registry image stream {: #imagestream_registry} By default, your Red Hat OpenShift on IBM Cloud cluster is set up to pull images from the remote, private IBM Cloud Container Registry `icr.io` domains in the `default` project. You can import an image from IBM Cloud Container Registry into the internal registry of your Red Hat OpenShift cluster by tagging the image as an [image stream](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/images/managing-image-streams){: external}. With this setup, you can deploy apps from the image by using the local cache of the internal registry, which can make your app deployments build faster. Also, deployments in other projects can refer to the image stream so that you don't have to create image pull secret credentials to IBM Cloud Container Registry in each project. {: shortdesc} If you update your image in IBM Cloud Container Registry, the image is not pulled automatically into the internal registry of your Red Hat OpenShift cluster. Instead, [configure periodic importing](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/images/managing-image-streams#images-imagestreams-import_image-streams-managing){: external}, or repeat these steps to tag the image. Depending on the image pull policy that you use in your deployment, you might also need to restart your deployment. {: note} Want to learn more about how builds, image streams, and the internal registry work together? Read the [Red Hat OpenShift docs](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/registry/registry-overview){: external}, or check out [this blog on managing container images](https://blog.cloudowski.com/articles/why-managing-container-images-on-openshift-is-better-than-on-kubernetes){: external}. {: tip} 1. [Access your Red Hat OpenShift cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-access_cluster&format=markdown). 1. Switch to the `default` project to pull your image into the image stream. The `default` project is already set up with credentials to access the `icr.io` registries. ```sh oc project default ``` {: pre} 1. List the available images in your IBM Cloud Container Registry. Note the **Repository** and **Tag** of the image that you want to pull into the internal registry of your Red Hat OpenShift cluster. ```sh ibmcloud cr images ``` {: pre} 4. Tag the image to pull it from your IBM Cloud Container Registry namespace into the internal registry as an image stream. For more information, see the [Red Hat OpenShift documentation](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/images/managing-image-streams){: external} or run `oc tag --help`. ```sh oc tag .icr.io//: default/: --reference-policy=local [--scheduled] ``` {: pre} - `.icr.io//:`: Use the **Repository** and **Tag** information that you previously retrieved to complete the IBM Cloud Container Registry region, namespace, image, and tag name of the image that you want to pull. - `default/:`: Enter the information for the internal image stream that you create from the IBM Cloud Container Registry tagged image. You create this image stream in the `default` project, which is also the project where the image stream is created if you don't specify a project. The values for `:` typically match the values that you previously retrieved. - `--reference-policy=local`: Set this value to `local` so that a copy of the image from IBM Cloud Container Registry is imported into the local cache of the internal registry and made available to the cluster's projects as an image stream. If you don't include this value, the image stream refers back to IBM Cloud Container Registry when you use it in your deployments and therefore requires credentials in the project. - `--scheduled`: Set this optional option to set up periodic importing of the image from IBM Cloud Container Registry into the internal registry. The default frequency is 15 minutes. For more information, see the [Red Hat OpenShift documentation](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/images/managing-image-streams#images-imagestreams-import_image-streams-managing){: external}. 5. Verify that your image stream is created. ```sh oc get imagestreams ``` {: pre} 6. Verify that the image stream successfully pulled the image from IBM Cloud Container Registry. In the output, check that the **latest tagged from** image matches your `* .icr.io//@` image. ```sh oc describe is/ ``` {: pre} Example output ```sh NAME: Namespace: default Created: 2 days ago Labels: Annotations: openshift.io/image.dockerRepositoryCheck=2020-03-31T09:41:36Z Image Repository: image-registry.openshift-image-registry.svc:5000/default/ant1 Image Lookup: local=false Unique Images: 1 Tags: 1 latest tagged from .icr.io//: * .icr.io//@ 2 days ago ``` {: screen} Now, your developers can [use the image stream in an app deployment](https://cloud.ibm.com/docs/openshift?topic=openshift-images&format=markdown#oc_imagestream_deploy). The image successfully builds from the locally pulled image in the internal registry. You don't need to set up an image pull secret in the project to IBM Cloud Container Registry, because the image stream is local to the cluster. ## Setting up builds in the internal registry to push images to IBM Cloud Container Registry {: #builds_registry} When you create a [build](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/builds_using_buildconfig/understanding-image-builds){: external} in your Red Hat OpenShift on IBM Cloud cluster, you can [set up the internal registry to push the image to your external repository](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/builds_using_buildconfig/creating-build-inputs){: external} in IBM Cloud Container Registry. By default, the image pull secret in the `default` project of your cluster only has read access to pull images from IBM Cloud Container Registry. To push images, you must add a secret with write access. {: shortdesc} 1. [Access your Red Hat OpenShift cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-access_cluster&format=markdown). 1. Switch to the `default` project. ```sh oc project default ``` {: pre} 1. [Follow the steps to set up an IBM Cloud IAM API key](https://cloud.ibm.com/docs/openshift?topic=openshift-registry&format=markdown#other_registry_accounts) with the `Reader` and `Writer` service access roles to pull from and push images to your `icr.io` registries. Keep in mind that any user with access to the project can use this secret to push images to your private registry. You might want to set up [logging and monitoring](https://cloud.ibm.com/docs/openshift?topic=openshift-health&format=markdown) tools so that you can observe who does what actions in your cluster. {: note} 4. Repeat the previous step for each `icr.io` region that you want to push images to. 5. Add the secret to the build service account and refer to the secrets in the build configuration file. For more information, see the [Red Hat OpenShift documentation](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/builds_using_buildconfig/creating-build-inputs){: external}. 1. Add the secret to the build service account by linking the secret that you just created to the `builder` role that all builds in the cluster use. ```sh oc secrets link builder ``` {: pre} 1. List the build configurations and note the ones that you want to give push and pull access to IBM Cloud Container Registry. ```sh oc get bc ``` {: pre} 1. Set the image push secret for the build configuration to use the secret that you just created with `Writer` service access to IBM Cloud Container Registry. ```sh oc set build-secret --push bc/ ``` {: pre} 4. Set the image pull secret for the build configuration to pull from the registry that you want to pull the initial build image from. For example, you can use the secret that you just created with `Reader` service access to IBM Cloud Container Registry if the source image is in an IBM Cloud Container Registry repository. ```sh oc set build-secret --pull bc/ ``` {: pre} 6. After you update the build service account and build configuration file to push to IBM Cloud Container Registry, restart your build. ```sh oc start-build ``` {: pre} 7. Get the name of your build pod, such as `-2-build`. ```sh oc get pods ``` {: pre} 8. Check the logs of the build and note where the image was pushed. ```sh oc logs ``` {: pre} Example of a successful image push log. ```sh ... Successfully pushed .icr.io//@sha256: Push successful ``` {: screen} 9. Check your images in your private registry to confirm that the image is created. ```sh ibmcloud cr image list ``` {: pre} Example output ```sh Repository Tag Digest Namespace Created Size Security status .icr.io// latest 2 minutes ago 182 MB 33 Issues ``` {: screen} Your Red Hat OpenShift build can now pull images from and push images to IBM Cloud Container Registry. ## Using IBM Cloud Container Registry {: #openshift_iccr} By default, your Red Hat OpenShift on IBM Cloud cluster is set up to pull images from the remote, private IBM Cloud Container Registry `icr.io` domains in the `default` project. If you want to use images that are stored in IBM Cloud Container Registry for other projects, you can pull the image to the internal registry in an image stream, or create image pull secrets for each global and regional registry in each project. {: shortdesc} **To import images into the internal registry**: See [Importing images from IBM Cloud Container Registry into the internal registry image stream](#imagestream_registry). **To pull images directly from the external IBM Cloud Container Registry**: See the following topics. * [Understanding how to authorize your cluster to pull images from a registry](#cluster_registry_auth). * [Copying the `all-icr-io` secret](#copy_imagePullSecret) from the `default` project to the project that you want to pull images from. * [Creating your own image pull secret](#other_registry_accounts). * [Adding the image pull secret](#use_imagePullSecret) to your deployment configuration or to the project service account. ## Understanding how to authorize your cluster to pull images from a private registry {: #cluster_registry_auth} To pull images from a registry, your Red Hat OpenShift on IBM Cloud cluster uses a special type of Kubernetes secret, an `imagePullSecret`. This image pull secret stores the credentials to access a container registry. {: shortdesc} The container registry can be: * A private namespace in your own IBM Cloud Container Registry. * A private namespace in IBM Cloud Container Registry that belongs to a different IBM Cloud account. * Any other private registry such as Docker. However, by default, your cluster is set up to pull images from only your account's namespaces in IBM Cloud Container Registry, and deploy containers from these images to the `default` Red Hat OpenShift project in your cluster. If you need to pull images in other projects of the cluster or from other container registries, then you must set up your own image pull secrets. ### Default image pull secret setup {: #cluster_registry_auth_default} Generally, your Red Hat OpenShift on IBM Cloud cluster is set up to pull images from all IBM Cloud Container Registry `icr.io` domains from the `default` Red Hat OpenShift project only. Review the following FAQ to learn more about how to pull images in other Red Hat OpenShift projects or accounts, restrict pull access, or why your cluster might not have the default image pull secrets. {: shortdesc} How is my cluster set up to pull images from the `default` Red Hat OpenShift project? : When you create a cluster, the cluster has an IBM Cloud IAM service ID that is given an IAM **Reader** service access role policy to IBM Cloud Container Registry. The service ID credentials are impersonated in a non-expiring API key that is stored in image pull secrets in your cluster. The image pull secrets are added to the `default` Kubernetes namespace and the list of secrets in the `default` service account for this Red Hat OpenShift project. By using image pull secrets, your deployments can pull images (read-only access) from the [global and regional IBM Cloud Container Registry](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overview&format=markdown#registry_regions) to deploy containers in the `default` Red Hat OpenShift project. * The global registry securely stores public images that are provided by IBM. You can refer to these public images across your deployments instead of having different references for images that are stored in each regional registry. * The regional registry securely stores your own private Docker images. What if I don't have image pull secrets in the `default` Red Hat OpenShift project? : You can check the image pull secrets by [logging in to your cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-access_cluster&format=markdown) and running `oc get secrets -n default | grep "icr-io"`. If no `icr` secrets are listed, the person who created the cluster might not have had the required permissions to IBM Cloud Container Registry in IAM. See [Updating existing clusters to use the API key image pull secret](#imagePullSecret_migrate_api_key). Can I restrict pull access to a certain regional registry? : Yes, you can [edit the existing IAM policy of the service ID](https://cloud.ibm.com/docs/iam?topic=iam-serviceids&interface=ui&format=markdown#update_serviceid) that restricts the **Reader** service access role to that regional registry or a registry resource such as a namespace. Before you can customize registry IAM policies, you must [enable IBM Cloud IAM policies for IBM Cloud Container Registry](https://cloud.ibm.com/docs/Registry?topic=Registry-user&format=markdown#user). Want to make your registry credentials even more secured Ask your cluster admin to [enable a key management service provider](https://cloud.ibm.com/docs/openshift?topic=openshift-encryption&format=markdown) in your cluster to encrypt Kubernetes secrets in your cluster, such as the image pull secret that stores your registry credentials. {: tip} Can I pull images in a Red Hat OpenShift project other than `default`? : Not by default. By using the default cluster setup, you can deploy containers from any image that is stored in your IBM Cloud Container Registry namespace into the `default` Red Hat OpenShift project of your cluster. To use these images in any other Red Hat OpenShift projects or other IBM Cloud accounts, [you have the option to copy or create your own image pull secrets](#other). Can I pull images from a different IBM Cloud account? : Yes, create an API key in the IBM Cloud account that you want to use. Then, in each project of each cluster that you want to pull images from the IBM Cloud account, create a secret that holds the API key. For more information, [follow along with this example that uses an authorized service ID API key](#other_registry_accounts). To use a non-IBM Cloud registry such as Docker, see [Accessing images that are stored in other private registries](#private_images). Does the API key need to be for a service ID? What happens if I reach the limit of service IDs for my account? : The default cluster setup creates a service ID to store IBM Cloud IAM API key credentials in the image pull secret. However, you can also create an API key for an individual user and store those credentials in an image pull secret. If you reach the [IAM limit for service IDs](https://cloud.ibm.com/docs/account?topic=account-ts_create_service&format=markdown), your cluster is created without the service ID and image pull secret and can't pull images from the `icr.io` registry domains by default. You must [create your own image pull secret](#other_registry_accounts), but by using an API key for an individual user such as a functional ID, not an IBM Cloud IAM service ID. I see image pull secrets for the regional registry domains and all registry domains. Which one do I use? : Previously, Red Hat OpenShift on IBM Cloud created separate image pull secrets for each regional, public `icr.io` registry domain. Now, all the public and private `icr.io` registry domains for all regions are stored in a single `all-icr-io` image pull secret that is automatically created in the `default` Kubernetes project of your cluster. For workloads in other Kubernetes namespaces in the cluster to pull container images from a private registry, you can now copy only the `all-icr-io` image pull secret to that Kubernetes project. Then, specify the `all-icr-io` secret in your service account or deployment. You don't need to copy the image pull secret that matches the regional registry of your image anymore. Also, keep in mind that you don't need image pull secrets for public registries, which don't require authentication. After I copy or create an image pull secret in another Red Hat OpenShift project, am I done? : Not quite. Your containers must be authorized to pull images by using the secret that you created. You can add the image pull secret to the service account for the namespace, or refer to the secret in each deployment. For instructions, see [Using the image pull secret to deploy containers](#use_imagePullSecret). ### Private network connection to `icr.io` registries {: #cluster_registry_auth_private} When you set up your IBM Cloud account to use service endpoints, you can use a private network connection to push images to and to pull images from IBM Cloud Container Registry. {: shortdesc} What do I need to do to set up my cluster to use the private connection to `icr.io` registries? 1. Enable a [Virtual Router Function (VRF)](https://cloud.ibm.com/docs/account?topic=account-vrf-service-endpoint&interface=ui&format=markdown) for your IBM Cloud infrastructure account so that you can use the IBM Cloud Container Registry private cloud service endpoint. To enable VRF, see [Enabling VRF](https://cloud.ibm.com/docs/account?topic=account-vrf-service-endpoint&interface=ui&format=markdown). To check whether a VRF is already enabled, use the `ibmcloud account show` command. 1. [Enable your IBM Cloud account to use service endpoints](https://cloud.ibm.com/docs/account?topic=account-vrf-service-endpoint&format=markdown#service-endpoint). IBM Cloud Container Registry automatically uses the private cloud service endpoint. You don't need to enable the private cloud service endpoint for your Red Hat OpenShift on IBM Cloud clusters. ## Updating existing clusters to use the API key image pull secret {: #imagePullSecret_migrate_api_key} New Red Hat OpenShift on IBM Cloud clusters store an API key in [image pull secrets to authorize access to IBM Cloud Container Registry](#cluster_registry_auth). With these image pull secrets, you can deploy containers from images that are stored in the `icr.io` registry domains. You can add the image pull secrets to your cluster if your cluster was not created with the secrets. {: shortdesc} Before you begin 1. [Access your Red Hat OpenShift cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-access_cluster&format=markdown). 1. Make sure that you have the following permissions: IBM Cloud IAM **Operator or Administrator** platform access role for Red Hat OpenShift on IBM Cloud. The account owner can give you the role by running the following command. ```sh ibmcloud iam user-policy-create EMAIL --service-name containers-kubernetes --roles "Administrator,Operator" ``` {: pre} 1. IBM Cloud IAM **Administrator** platform access role for IBM Cloud Container Registry, across all regions and resource groups. The policy can't be scoped to a particular region or resource group. The account owner can give you the role by running the following command. Verify that the secret was created successfully ```sh ibmcloud iam user-policy-create YOUR_USER_EMAIL --service-name container-registry --roles Administrator ``` {: pre} 4. If your account [restricts service ID creation](https://cloud.ibm.com/docs/iam?topic=iam-restrict-service-id-create&format=markdown), add the **Service ID creator** role to **Identity and Access Management** in the console (`iam-identity` in the API or CLI). 5. If your account [restricts API key creation](https://cloud.ibm.com/docs/iam?topic=iam-allow-api-create&format=markdown), add the **User API key creator** role to **Identity and Access Management** in the console (`iam-identity` in the API or CLI). ### Updating your image pull secret {: #update-pull-secret} To update your cluster image pull secret in the `default` Kubernetes namespace. {: shortdesc} 1. Get your cluster ID. ```sh ibmcloud oc cluster ls ``` {: pre} 1. Run the following command to create a service ID for the cluster and assign the service ID an IAM **Reader** service access role for IBM Cloud Container Registry. The command also creates an API key to impersonate the service ID credentials and stores the API key in a Kubernetes image pull secret in the cluster. The image pull secret is in the `default` Red Hat OpenShift project. ```sh ibmcloud oc cluster pull-secret apply --cluster CLUSTER_NAME_OR_ID ``` {: pre} When you run this command, the creation of IAM credentials and image pull secrets is initiated and can take some time to complete. You can't deploy containers that pull an image from the IBM Cloud Container Registry `icr.io` domains until the image pull secrets are created. {: important} 1. Verify that the image pull secrets are created in your cluster. ```sh oc get secrets | grep icr-io ``` {: pre} Example output ```sh all-icr-io           kubernetes.io/dockerconfigjson        1         16d ``` {: screen} 4. Update your [container deployments](https://cloud.ibm.com/docs/openshift?topic=openshift-app&format=markdown#image) to pull images from the `icr.io` domain name. 5. Optional: If you have a firewall, make sure that you [allow outbound network traffic to the registry subnets](https://cloud.ibm.com/docs/openshift?topic=openshift-firewall&format=markdown#firewall_outbound) for the domains that you use. 6. Complete your setup by using one of the following options. * To pull images in Red Hat OpenShift projects other than `default` or from other IBM Cloud accounts, [copy or create another image pull secret](#other). * To restrict the image pull secret access to particular registry resources such as namespaces or regions: 1. Make sure that [IBM Cloud IAM policies for IBM Cloud Container Registry are enabled](https://cloud.ibm.com/docs/Registry?topic=Registry-user&format=markdown). 1. [Edit the IBM Cloud IAM policies](https://cloud.ibm.com/docs/iam?topic=iam-serviceids&interface=ui&format=markdown#update_serviceid) for the service ID, or [create another image pull secret](#other_registry_accounts). ## Using an image pull secret to access images in external private registries {: #other} Set up your own image pull secret in your cluster to deploy containers to Red Hat OpenShift projects other than `default`, use images that are stored in other IBM Cloud accounts, or use images that are stored in external private registries. Further, you might create your own image pull secret to apply IAM access policies that restrict permissions to specific registry image namespaces, or actions (such as `push` or `pull`). {: shortdesc} After you create the image pull secret, your containers must use the secret to be authorized to pull an image from the registry. You can add the image pull secret to the service account for the project, or refer to the secret in each deployment. For instructions, see [Using the image pull secret to deploy containers](#use_imagePullSecret). Image pull secrets are valid only for the Red Hat OpenShift projects that they were created for. Repeat these steps for every namespace where you want to deploy containers. {: tip} Before you begin: 1. [Set up a namespace in IBM Cloud Container Registry and push images to this namespace](https://cloud.ibm.com/docs/Registry?topic=Registry-getting-started&format=markdown#gs_registry_namespace_add). 1. [Create a cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-clusters&format=markdown). 1. [Access your Red Hat OpenShift cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-access_cluster&format=markdown). To use your own image pull secret, choose among the following options: - [Copy the image pull secret](#copy_imagePullSecret) from the default Red Hat OpenShift project to other projects in your cluster. - [Create new IAM API key credentials and store them in an image pull secret](#other_registry_accounts) to access images in other IBM Cloud accounts or to apply IAM policies that restrict access to certain registry domains or namespaces. - [Create an image pull secret to access images in external private registries](#private_images). If you already created an image pull secret in your project that you want to use in your deployment, see [Deploying containers by using the created `imagePullSecret`](#use_imagePullSecret). ### Copying an existing image pull secret {: #copy_imagePullSecret} You can copy an image pull secret, such as the one that is automatically created for the `default` Red Hat OpenShift project, to other projects in your cluster. If you want to use different IBM Cloud IAM API key credentials for this project such as to restrict access to specific projects, or to pull images from other IBM Cloud accounts, [create an image pull secret](#other_registry_accounts) instead. {: shortdesc} 1. List available Red Hat OpenShift projects in your cluster, or create a project to use. ```sh oc get projects ``` {: pre} Example output ```sh default Active ibm-cert-store Active ibm-system Active kube-public Active kube-system Active ``` {: screen} To create a project ```sh oc new-project ``` {: pre} 1. List the existing image pull secrets in the `default` Red Hat OpenShift project for IBM Cloud Container Registry. ```sh oc get secrets -n default | grep icr-io ``` {: pre} Example output ```sh all-icr-io          kubernetes.io/dockerconfigjson        1         16d ``` {: screen} 1. Copy the `all-icr-io` image pull secret from the `default` project to the project of your choice. The new image pull secrets are named `-icr--io`. ```sh oc get secret all-icr-io -n default -o yaml | sed 's/default//g' | oc create -n -f - ``` {: pre} 4. Verify that the secrets are created successfully. ```sh oc get secrets -n | grep icr-io ``` {: pre} 5. To deploy containers, [add the image pull secret](#use_imagePullSecret) to each deployment or to the service account of the project so that any deployment in the project can pull images from the registry. ### Creating an image pull secret with different IAM API key credentials {: #other_registry_accounts} You can assign IBM Cloud IAM access policies to users or a service ID to restrict permissions to specific registry image namespaces or actions (such as `push` or `pull`). Then, create an API key and store these registry credentials in an image pull secret for your cluster. {: shortdesc} For example, to access images in other IBM Cloud accounts, create an API key that stores the IBM Cloud Container Registry credentials of a user or service ID in that account. Then, in your cluster's account, save the API key credentials in an image pull secret for each cluster and cluster project. The following steps create an API key that stores the credentials of an IBM Cloud IAM service ID. Instead of using a service ID, you might want to create an API key for a user ID that has an IBM Cloud IAM service access policy to IBM Cloud Container Registry. However, make sure that the user is a functional ID or have a plan in case the user leaves so that the cluster can still access the registry. {: note} 1. List available Red Hat OpenShift projects in your cluster, or create a project to use where you want to deploy containers from your registry images. ```sh oc get projects ``` {: pre} Example output ```sh default Active ibm-cert-store Active ibm-system Active kube-public Active kube-system Active ``` {: screen} To create a project ```sh oc new-project ``` {: pre} 1. Create an IBM Cloud IAM service ID for your cluster that is used for the IAM policies and API key credentials in the image pull secret. Be sure to give the service ID a description that helps you retrieve the service ID later, such as including both the cluster and project name. ```sh ibmcloud iam service-id-create --id --description "Service ID for IBM Cloud Container Registry in Red Hat OpenShift on IBM Cloud cluster project " ``` {: pre} 1. Create a custom IBM Cloud IAM policy for your cluster service ID that grants access to IBM Cloud Container Registry. ```sh ibmcloud iam service-policy-create --roles --service-name container-registry [--region ] [--resource-type namespace --resource ] ``` {: pre} `cluster_service_ID` : Required. Replace with the `--id` service ID that you previously created for your Kubernetes cluster. `--service-name container-registry` : Required. Enter `container-registry` so that the IAM policy is for IBM Cloud Container Registry. `--roles ` : Required. Enter the [service access role](https://cloud.ibm.com/docs/Registry?topic=Registry-iam&format=markdown#service_access_roles) for IBM Cloud Container Registry that you want to scope the service ID access to. Possible values are `Reader`, `Writer`, and `Manager`. `--region ` : Optional. If you want to scope the access policy to certain IAM regions, enter the regions in a comma-separated list. Possible values are `global` and the [local registry regions](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overview&format=markdown#registry_regions_local). `--resource-type namespace --resource ` : Optional. If you want to limit access to only images in certain [IBM Cloud Container Registry namespaces](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_setup_cli_namespace&format=markdown#registry_setup_cli_namespace_plan), enter `namespace` for the resource type and specify the ``. To list registry namespaces, run `ibmcloud cr namespaces`. 1. Create an API key for the service ID. Name the API key similar to your service ID, and include the service ID that you previously created, `--id`. Be sure to give the API key a description that helps you retrieve the key later. ```sh ibmcloud iam service-api-key-create --key --id --description "API key for service ID in Red Hat OpenShift on IBM Cloud cluster project " ``` {: pre} 1. Retrieve your **API Key** value from the output of the previous command. ```sh Please preserve the API key! It can't be retrieved after it's created. Name --key Description key_for_registry_for_serviceid_for_kubernetes_cluster_multizone_namespace_test Bound To crn:v1:bluemix:public:iam-identity::a/1bb222bb2b33333ddd3d3333ee4ee444::serviceid:ServiceId-ff55555f-5fff-6666-g6g6-777777h7h7hh Created At 2019-02-01T19:06+0000 API Key i-8i88ii8jjjj9jjj99kkkkkkkkk_k9-llllll11mmm1 Locked false UUID ApiKey-222nn2n2-o3o3-3o3o-4p44-oo444o44o4o4 ``` {: screen} 1. Create an image pull secret to store the API key credentials in the cluster project. Repeat this step for each project of each cluster for each `icr.io` domain that you want to pull images from. ```sh oc --namespace create secret docker-registry --docker-server= --docker-username=iamapikey --docker-password= --docker-email= ``` {: pre} `--namespace ` : Required. Specify the Red Hat OpenShift project of your cluster that you used for the service ID name. `` : Required. Enter a name for your image pull secret. `--docker-server ` : Required. Set the URL to the image registry where your registry namespace is set up. For available domains, see [Local regions](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overview&format=markdown#registry_regions). `--docker-username iamapikey` : Required. Enter the username to log in to your private registry. If you use IBM Cloud Container Registry, enter `iamapikey`. `--docker-password ` : Required. Enter the value of your `API Key` that you previously retrieved. `--docker-email ` : Required. If you have one, enter your Docker email address. If you don't, enter a fictional email address, such as `a@b.c`. This email is required to create a Kubernetes secret, but is not used after creation. 1. Verify that the secret was created successfully. Replace `` with the `project` where you created the image pull secret. ```sh oc get secrets --namespace ``` {: pre} 1. [Add the image pull secret to a Kubernetes service account so that any pod in the project can use the image pull secret when you deploy a container](#use_imagePullSecret). ### Accessing images that are stored in other private registries {: #private_images} If you already have a private registry, you must store the registry credentials in a Kubernetes image pull secret and reference this secret from your configuration file. {: shortdesc} Before you begin: 1. [Create a cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-clusters&format=markdown). 1. [Access your Red Hat OpenShift cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-access_cluster&format=markdown). To create an image pull secret: 1. Create the Kubernetes secret to store your private registry credentials. ```sh oc --namespace create secret docker-registry --docker-server= --docker-username= --docker-password= --docker-email= ``` {: pre} `--namespace ` : Required. The Red Hat OpenShift project of your cluster where you want to use the secret and deploy containers to. To list available projects in your cluster, run `oc get projects`. `` : Required. The name that you want to use for your image pull secret. `--docker-server ` : Required. The URL to the registry where your private images are stored. `--docker-username ` : Required. The username to log in to your private registry. `--docker-password ` : Required. The password to log in to your private registry, such as a token value. `--docker-email ` : Required. If you have one, enter your Docker email address. If you don't have one, enter a fictional email address, such as `a@b.c`. This email is required to create a Kubernetes secret, but is not used after creation. 1. Verify that the secret was created successfully. Replace `` with the name of the project where you created the image pull secret. ```sh oc get secrets --namespace ``` {: pre} 1. [Create a pod that references the image pull secret](#use_imagePullSecret). ## Using the image pull secret to deploy containers {: #use_imagePullSecret} You can define an image pull secret in your pod deployment or store the image pull secret in your Kubernetes service account so that it is available for all deployments that don't specify a Kubernetes service account in the project. {: shortdesc} To plan how image pull secrets are used in your cluster, choose between the following options. * Referring to the image pull secret in your pod deployment: Use this option if you don't want to grant access to your registry for all pods in your project by default. Developers can [include the image pull secret in each pod deployment](https://cloud.ibm.com/docs/openshift?topic=openshift-images&format=markdown#pod_imagePullSecret) that must access your registry. * Storing the image pull secret in the Kubernetes service account: Use this option to grant access to images in your registry for all deployments in the selected Red Hat OpenShift projects. To store an image pull secret in the Kubernetes service account, use the [following steps](#store_imagePullSecret). ### Storing the image pull secret in the Kubernetes service account for the selected project {: #store_imagePullSecret} Every Red Hat OpenShift project has a Kubernetes service account that is named `default`. Within the project, you can add the image pull secret to this service account to grant access for pods to pull images from your registry. Deployments that don't specify a service account automatically use the `default` service account for this Red Hat OpenShift project. {: shortdesc} 1. Check if an image pull secret already exists for your default service account. ```sh oc describe serviceaccount default -n ``` {: pre} When `` is displayed in the **Image pull secrets** entry, no image pull secret exists. 1. Add the image pull secret to your default service account. - Example command to add the image pull secret when no image pull secret is defined. ```sh oc patch -n serviceaccount/default -p '{"imagePullSecrets":[{"name": ""}]}' ``` {: pre} - Example command to add the image pull secret when an image pull secret is already defined. ```sh oc patch -n serviceaccount/default --type='json' -p='[{"op":"add","path":"/imagePullSecrets/-","value":{"name":""}}]' ``` {: pre} 1. Verify that your image pull secret was added to your default service account. ```sh oc describe serviceaccount default -n ``` {: pre} Example output ```sh Name: default Namespace: Labels: Annotations: Image pull secrets: Mountable secrets: default-token-sh2dx Tokens: default-token-sh2dx Events: ``` {: pre} If the **Image pull secrets** says ` (not found)`, verify that the image pull secret exists in the same project as your service account by running `oc get secrets -n project`. 1. Create a pod configuration file that is named `mypod.yaml` to deploy a container from an **image** in your registry. ```yaml apiVersion: v1 kind: Pod metadata: name: mypod spec: containers: - name: mypod-container image: .icr.io//: ``` {: codeblock} 1. Create the pod in the cluster by applying the `mypod.yaml` configuration file. ```sh oc apply -f mypod.yaml ``` {: pre} ## Setting up a cluster to pull entitled software {: #secret_entitled_software} You can set up your Red Hat OpenShift on IBM Cloud cluster to pull entitled software, which is a collection of protected container images that are packaged in Helm charts that you are licensed to use by IBM. Entitled software is stored in a special IBM Cloud Container Registry `cp.icr.io` domain. To access this domain, you must create an image pull secret with an entitlement key for your cluster and add this image pull secret to the Kubernetes service account of each project where you want to deploy this entitled software. {: shortdesc} Before you begin: [Access your Red Hat OpenShift cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-access_cluster&format=markdown). 1. Get the entitlement key for your entitled software library. 1. Log in to [MyIBM.com](https://myibm.ibm.com){: external} and scroll to the **Container software library** section. Click **View library**. 1. From the **Access your container software > Entitlement keys** page, click **Copy key**. This key authorizes access to all the entitled software in your container software library. 1. In the project that you want to deploy your entitled containers, create an image pull secret so that you can access the `cp.icr.io` entitled registry. Use the **entitlement key** that you previously retrieved as the `--docker-password` value. For more information, see [Accessing images that are stored in other private registries](#private_images). ```sh oc create secret docker-registry entitled-cp-icr-io --docker-server=cp.icr.io --docker-username=cp --docker-password= --docker-email= -n ``` {: pre} 1. Add the image pull secret to the service account of the namespace so that any container in the project can use the entitlement key to pull entitled images. For more information, see [Using the image pull secret to deploy containers](#use_imagePullSecret). ```sh oc patch -n serviceaccount/default --type='json' -p='[{"op":"add","path":"/imagePullSecrets/-","value":{"name":"entitled-cp-icr-io"}}]' ``` {: pre} 1. Create a pod in the project that builds a container from an image in the entitled registry. ```sh oc run --image=cp.icr.io/ -n --generator=run-pod/v1 ``` {: pre} 1. Check that your container was able to successfully build from the entitled image by verifying that the pod is in a **Running** status. ```sh oc get pod -n ``` {: pre} Wondering what to do next? You can [set up the **entitled** Helm chart repository](https://cloud.ibm.com/docs/openshift?topic=openshift-helm&format=markdown), where Helm charts that incorporate entitled software are stored. If you already have Helm installed in your cluster, run `helm repo add entitled https://raw.githubusercontent.com/IBM/charts/master/repo/entitled`. {: tip} ## Adding a private registry to the global pull secret {: #cluster_global_pull_secret} [RHEL worker nodes]{: tag-red} In clusters that use only RHEL worker nodes, you can set up a global image pull secret that each worker node in the cluster can use to pull images from a private registry. {: shortdesc} By default, your Red Hat OpenShift on IBM Cloud cluster has a global image pull secret for the following registries, so that default Red Hat OpenShift components can be deployed. * `cloud.openshift.com` * `quay.io` * `registry.connect.redhat.com` * `registry.redhat.io` Do not replace the global pull secret with a pull secret that does not have credentials to the default Red Hat registries. If you do, the default Red Hat OpenShift components that are installed in your cluster, such as the OperatorHub, might fail because they can't pull images from these registries. {: important} Before you begin: * [Download the `jq` JSON processor command line package](https://stedolan.github.io/jq/download/){: external}. You use `jq` to combine the JSON value of the default global pull secret with the private registry pull secret that you want to add. * [Access your Red Hat OpenShift cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-access_cluster&format=markdown). To add private registries, edit the global `pull-secret` in the `openshift-config` project. 1. Create a secret value that holds the credentials to access your private registry and store the decoded secret value in a JSON file. When you create the secret value, the credentials are automatically encoded to base64. By using the `--dry-run` option, the secret value is created only and no secret object is created in your cluster. The decoded secret value is then stored in a JSON file to later use in your global pull secret. ```sh oc create secret docker-registry --docker-server= --docker-username= --docker-password= --docker-email= --dry-run=true --output="jsonpath={.data.\.dockerconfigjson}" | base64 --decode > myregistryconfigjson ``` {: pre} `--namespace ` : Required. The Red Hat OpenShift project of your cluster where you want to use the secret and deploy containers to. To list available projects in your cluster, run `oc get ns`. `` : Required. The name that you want to use for your image pull secret. `--docker-server ` : Required. The URL to the registry where your private images are stored. `--docker-username ` : Required. The username to log in to your private registry. `--docker-password ` : Required. The password to log in to your private registry, such as a token value. `--docker-email ` : Required. If you have one, enter your Docker email address. If you don't have one, enter a fictional email address, such as `a@b.c`. This email is required to create a Kubernetes secret, but is not used after creation. `--dry-run=true` : Include this option to create the secret value only, and not create and store the secret object in your cluster. `--output="jsonpath={.data.\.dockerconfigjson}"` : Get only the `.dockerconfigjson` value from the data section of the Kubernetes secret. `| base64 --decode > myregistryconfigjson` : Download the decoded secret data to a local `myregistryconfigjson` file. 1. Retrieve the decoded secret value of the default global pull secret and store the value in a `dockerconfigjson` file. ```sh oc get secret pull-secret -n openshift-config --output="jsonpath={.data.\.dockerconfigjson}" | base64 --decode > dockerconfigjson ``` {: pre} 1. Combine the downloaded private registry pull secret `myregistryconfigjson` file with the default global pull secret `dockerconfigjson` file. ```sh jq -s '.[0] * .[1]' dockerconfigjson myregistryconfigjson > dockerconfigjson-merged ``` {: pre} 1. Update the global pull secret with the combined `dockerconfigjson-merged` file. ```sh oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=dockerconfigjson-merged ``` {: pre} 1. Verify that the global pull secret is updated. Check that your private registry and each of the default Red Hat registries are in the output of the following command. ```sh oc get secret pull-secret -n openshift-config --output="jsonpath={.data.\.dockerconfigjson}" | base64 --decode ``` {: pre} Example output ```sh { "auths": { "cloud.openshift.com": { "auth": "", "email": "email@example.com" }, "quay.io": { "auth": "", "email": "email@example.com" }, "registry.connect.redhat.com": { "auth": "", "email": "email@example.com" }, "registry.redhat.io": { "auth": "", "email": "email@example.com" }, "": { "username": "iamapikey", "password": "", "email": "email@example.com", "auth": "" } } } ``` {: screen} 1. To pick up the global configuration changes, reload all the worker nodes in your cluster. 1. Note the **ID** of the worker nodes in your cluster. ```sh ibmcloud oc worker ls -c ``` {: pre} 1. **For Classic clusters** Reload each worker node. You can reload multiple worker nodes by including multiple `-w` options, but make sure to leave enough worker nodes running at the same time for your apps to avoid an outage. ```sh ibmcloud oc worker reload -c -w -w ``` {: pre} 1. **For VPC clusters** Replace each worker node. Before you begin, make sure that your cluster has enough other worker nodes so that your pods can be rescheduled and continue to run. ```sh ibmcloud oc worker replace --cluster --worker ``` {: pre} 1. After the worker nodes are back in a healthy state, verify that the global pull secret is updated on a worker node. 1. Start a debugging pod to log in to a worker node. Use the **Private IP** that you retrieved earlier for the ``. ```sh oc debug node/ ``` {: pre} 1. Change the root directory to the host so that you can view files on the worker node. ```sh chroot /host ``` {: pre} 1. Verify that the Docker configuration file has the registry credentials that match the global pull secret that you set. ```sh vi /.docker/config.json ``` {: pre} ## Updating the global pull secret {: #oc-pull-secret-cli} [RHCOS worker nodes]{: tag-magenta} [RHEL worker nodes]{: tag-red} In clusters that use RHCOS workers or a combination or RHCOS and RHEL workers, you can complete the following steps to update the global pull secret in your Red Hat OpenShift on IBM Cloud cluster. By default, your Red Hat OpenShift on IBM Cloud cluster has a global image pull secret for the following registries, so that default Red Hat OpenShift components can be deployed. * `cloud.openshift.com` * `quay.io` * `registry.connect.redhat.com` * `registry.redhat.io` Do not replace the global pull secret with a pull secret that does not have credentials to the default Red Hat registries. If you do, the default Red Hat OpenShift components that are installed in your cluster, such as the OperatorHub, might fail because they can't pull images from these registries. {: important} 1. Create a secret that has the credentials for the registry you want to use. ```sh oc create secret docker-registry docker-auth-secret \ --docker-server=REGISTRY \ --docker-username=USERNAME \ --docker-password=PASSWORD \ --namespace kube-system ``` {: pre} 1. Create a DaemonSet to apply the secret across all worker nodes. ```yaml ocp_release_pull_secret_image=$(oc get ds -n openshift-dns node-resolver -o jsonpath='{.spec.template.spec.containers[0].image}') cat << EOF | oc create -f - apiVersion: apps/v1 kind: DaemonSet metadata: name: update-docker-config namespace: kube-system labels: app: update-docker-config spec: selector: matchLabels: name: update-docker-config template: metadata: labels: name: update-docker-config spec: initContainers: - command: ["/bin/sh", "-c"] args: - > echo "Checking if RHEL or RHCOS host"; [[ -s /docker-config/.docker/config.json ]] && CONFIG_PATH=/docker-config/.docker || CONFIG_PATH=/docker-config/root/.docker; echo "Backing up or restoring config.json"; [[ -s \$CONFIG_PATH/config.json ]] && cp \$CONFIG_PATH/config.json \$CONFIG_PATH/config.json.bak || cp \$CONFIG_PATH/config.json.bak \$CONFIG_PATH/config.json; echo "Merging secret with config.json"; /host/usr/bin/jq -s '.[0] * .[1]' \$CONFIG_PATH/config.json /auth/.dockerconfigjson > \$CONFIG_PATH/config.tmp; mv \$CONFIG_PATH/config.tmp \$CONFIG_PATH/config.json; echo "Sending signal to reload crio config"; pidof crio; kill -1 \$(pidof crio) image: ${ocp_release_pull_secret_image} imagePullPolicy: IfNotPresent name: updater resources: {} securityContext: privileged: true volumeMounts: - name: docker-auth-secret mountPath: /auth - name: docker mountPath: /docker-config - name: bin mountPath: /host/usr/bin - name: lib64 mountPath: /lib64 containers: - resources: requests: cpu: 0.01 image: ${ocp_release_pull_secret_image} name: sleepforever command: ["/bin/sh", "-c"] args: - > while true; do sleep 100000; done hostPID: true volumes: - name: docker-auth-secret secret: secretName: docker-auth-secret - name: docker hostPath: path: / - name: bin hostPath: path: /usr/bin - name: lib64 hostPath: path: /lib64 hostPathType: Directory EOF ``` {: codeblock} 1. Verify the pods are running. ```sh oc get daemonset -n kube-system update-docker-config ``` {: pre} ## Updating an IBM Cloud Kubernetes Service containerd custom registry configuration {: #update_containerd_registry_config} With Kubernetes version 1.22 or later, you can use containerd configuration files on worker nodes to configure pulling from a container registry. You can use a daemonset to update the configurations across all nodes in a cluster, which prevents configurations from being wiped when worker nodes reload or when new workers are added. ### Example daemonset to update a containerd custom registry configuration {: #ds-example-registry} Use the example YAML file to define a daemonset that runs on all worker nodes to set or update a containerd registry host configuration and mount to the corresponding containerd registry path. The example sets the following registry host configuration for dockerhub. This registry host configuration is already provided and automatically configured during the worker provisioning phase. The `init` container initializes `hosts.toml` on every worker node after deployment and after worker nodes reload or restart. ```sh server = "https://docker.io" [host."https://registry-1.docker.io"] capabilities = ["pull", "resolve"] ``` {: screen} **Example YAML file**: ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: labels: name: containerd-dockerhub-registry-config name: containerd-dockerhub-registry-config namespace: kube-system spec: selector: matchLabels: name: containerd-dockerhub-registry-config template: metadata: labels: name: containerd-dockerhub-registry-config spec: initContainers: - image: alpine:3.13.6 name: containerd-dockerhub-registry-config command: - /bin/sh - -c - | #!/bin/sh set -uo pipefail cat << EOF > /etc/containerd/certs.d/docker.io/hosts.toml server = "https://docker.io" [host."https://registry-1.docker.io"] capabilities = ["pull", "resolve"] EOF volumeMounts: - mountPath: /etc/containerd/certs.d/docker.io/ name: dockerhub-registry-config containers: - name: pause image: "us.icr.io/armada-master/pause:3.5" imagePullPolicy: IfNotPresent volumes: - name: dockerhub-registry-config hostPath: path: /etc/containerd/certs.d/docker.io/ ``` {: codeblock} For more information on updating a containerd registry host configuration, see the [containerd documentation](https://github.com/containerd/containerd/blob/v1.5.6/docs/hosts.md){: external}.