--- name: openshift-openshift-storage-update-vpc title: Updating or replacing VPC worker nodes that use OpenShift Data Foundation description: For VPC clusters with a storage solution such as OpenShift Data Foundation you must cordon, drain, and update each worker node sequentially. For bare metal worker nodes, you can now use the `worker reload` command instead of `worker replace`. If you deployed OpenShift Data Foundation to a subset of worker nodes in your cluster, then after you update the worker node, you must then edit the `ocscluster` resource to include the new worker node. last-updated: 2026-08-06 --- > ## Documentation Index > The table of contents for this documentation set is at https://cloud.ibm.com/docs/openshift?format=markdown > The index for all IBM Cloud docs is at: https://cloud.ibm.com/docs/llms.txt > Use these files to discover more information as needed. # Updating or replacing VPC worker nodes that use OpenShift Data Foundation {: #openshift-storage-update-vpc} {: toc-content-type="tutorial"} {: toc-services="openshift, vpc"} {: toc-completion-time="60m"} [Virtual Private Cloud]{: tag-vpc} For VPC clusters with a storage solution such as OpenShift Data Foundation you must cordon, drain, and update each worker node sequentially. For bare metal worker nodes, you can now use the `worker reload` command instead of `worker replace`. If you deployed OpenShift Data Foundation to a subset of worker nodes in your cluster, then after you update the worker node, you must then edit the `ocscluster` resource to include the new worker node. {: shortdesc} The following tutorial covers both major and minor updates and worker node updates. [Major update]{: tag-red} : Complete the steps with this label to apply a major update; for example, if you are updating your worker nodes to a new major version, such as from `4.11` to `4.12` as well as OpenShift Data Foundation from `4.11` to `4.12`. [Minor update]{: tag-blue} : Complete the steps with this label to apply a patch update, for example if you are updating from `4.12.15_1542_openshift` to `4.12.16_1544_openshift` while keeping OpenShift Data Foundation at version `4.12`. You must repeat these steps for each node you want to update. [Worker replace]{: tag-green} : Complete the steps with this label steps if you are replacing a worker node at the same patch version. You must repeat these steps for each node you want to replace. Skipping versions during an upgrade, such as from 4.8 to 4.12 is not supported. {: important} [Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.](https://cloud.ibm.com/docs/containers?topic=containers-access_cluster&format=markdown) Before updating your worker nodes, make sure to back up your app data. Also, plan to complete the following steps for one worker node at a time. Repeat the steps for each worker node that you want to update. {: important} ## Check the status of your storage cluster {: #storage-cluster-status} {: step} [Major update]{: tag-red} [Minor update]{: tag-blue} [Worker replace]{: tag-green} 1. List the pods by running the following command. Verify that all the pods in `openshift-storage` namespace are in a good state. Address any pods that are not in “Running” or “Completed” state. ```sh oc get pods -n openshift-storage ``` {: pre} 1. Run the following command and verify the `Phase` of the `ocs-storagecluster` is `Ready`. ```sh oc get storagecluster -n openshift-storage ``` {: pre} Example output ```sh NAME AGE PHASE EXTERNAL CREATED AT VERSION ocs-storagecluster 3m49s Ready 2025-04-06T09:37:49Z 4.16.9 ``` {: screen} 1. Check the state of the Ceph Storage by running the following command. Verify that the health is `HEALTH_OK`, all OSDs are `up` and `IN` and that all the `pgs` are `active+clean`. If any of these checks fail, Open a [support case](https://cloud.ibm.com/docs/support?topic=support-using-avatar&format=markdown). In the case details, be sure to include any relevant log files, error messages, or command outputs.. Resolve any issues before continuing. ```sh oc rsh -n openshift-storage $(oc get pods -n openshift-storage -o name -l app=rook-ceph-operator) ceph status -c /var/lib/rook/openshift-storage/openshift-storage.config ``` {: pre} Example output ```txt health: HEALTH_OK # Verify health is HEALTH_OK services: mon: 3 daemons, quorum a,b,c (age 3h) mgr: a(active, since 6h) mds: ocs-storagecluster-cephfilesystem:1 {0=ocs-storagecluster-cephfilesystem-b=up:active} 1 up:standby-replay osd: 27 osds: 27 up (since 2h), 27 in (since 111m) # Verify OSDs are “up” and “in” rgw: 2 daemons active (ocs.storagecluster.cephobjectstore.a, ocs.storagecluster.cephobjectstore.b) data: pools: 10 pools, 1136 pgs objects: 5.50M objects, 3.3 TiB usage: 9.9 TiB used, 43 TiB / 53 TiB avail pgs: 1136 active+clean # Verify psgs are active+clean io: client: 93 KiB/s rd, 2.0 MiB/s wr, 5 op/s rd, 29 op/s wr ``` {: screen} Repeat these health checks before repeating the update procedure for additional nodes. Bringing down more then one OSD pod at a time might jeopardize user data. {: important} ## Update the cluster master {: #update-cluster-master-vpc} {: step} [Major update]{: tag-red} 1. If you are updating your worker nodes to a new major version, such as from `4.11` to `4.12`, update the cluster master first. ```sh ibmcloud oc cluster master update --cluster CLUSTER [--version MAJOR.MINOR.PATCH] [--force-update] [-f] [-q] ``` {: pre} Example command: ```sh ibmcloud oc cluster master update --cluster mycluster --version 4.21.27 --force-update ``` {: pre} 1. Wait until the master update finishes. ## Decide which storage nodes you want to update or replace {: #determine-storage-nodes-vpc} {: step} [Major update]{: tag-red} [Minor update]{: tag-blue} [Worker replace]{: tag-green} 1. List your worker nodes by using `oc get nodes` and determine which storage nodes you want to update. ```sh oc get nodes ``` {: pre} Example output ```sh NAME STATUS ROLES AGE VERSION 10.241.0.4 Ready master,worker 106s v1.21.6+4b61f94 10.241.128.4 Ready master,worker 22d v1.21.6+bb8d50a 10.241.64.4 Ready master,worker 22d v1.21.6+bb8d50a ``` {: screen} ## Make sure the storage cluster is healthy {: #cluster-health} {: step} [Major update]{: tag-red} [Minor update]{: tag-blue} [Worker replace]{: tag-green} Run the following commands to check the storage cluster health. ```sh oc get storagecluster -n openshift-storage oc get cephcluster -n openshift-storage ``` {: pre} Make sure the storage cluster is healthy before continuing. ## Scale down OpenShift Data Foundation {: #scale-down-odf-vpc} {: step} [Major update]{: tag-red} [Minor update]{: tag-blue} [Worker replace]{: tag-green} 1. For each worker node that you found in the previous step, find the `rook-ceph-mon` and `rook-ceph-osd` deployments. ```sh oc get pods -n openshift-storage -o wide | grep -i ``` {: pre} If the Noobaa pods get stuck while draining, you can manually delete the NooBaa pods, so that they get scheduled on a different node. {: note} 1. Delete any remaining the Noobaa pods in the following order. ```txt noobaa-db noobaa-core noobaa-endpoint noobaa-operator ``` {: codeblock} 1. Scale down the deployments that you found in the previous step. ```sh oc scale deployment rook-ceph-mon-c --replicas=0 -n openshift-storage ``` {: pre} ```sh oc scale deployment rook-ceph-osd-2 --replicas=0 -n openshift-storage ``` {: pre} ```sh oc scale deployment --selector=app=rook-ceph-crashcollector,node_name=NODE-NAME --replicas=0 -n openshift-storage ``` {: pre} ## Cordon and drain the worker node {: #cordon-drain-worker-node-vpc} {: step} [Major update]{: tag-red} [Minor update]{: tag-blue} [Worker replace]{: tag-green} 1. Cordon the node. Cordoning the node prevents any pods from being scheduled on this node. ```sh oc adm cordon NODE_NAME ``` {: pre} Example output ```sh node/10.241.0.4 cordoned ``` {: screen} 1. Drain the node to remove all the pods. When you drain the worker node, the pods move to the other worker nodes ensuring there is no downtime. Draining also ensures that there is no disruption of the pod disruption budget. ```sh oc adm drain NODE_NAME --force --delete-emptydir-data --ignore-daemonsets ``` {: pre} Example output ```sh evicting pod "managed-storage-validation-webhooks-7fd79bc9f7-pdpv6" evicting pod "calico-kube-controllers-647dbbd685-fmrp9" evicting pod "certified-operators-2v852" evicting pod "csi-snapshot-controller-77fbf474df-47ddt" evicting pod "calico-typha-8574d89b8c-7f2cc" evicting pod "dns-operator-6d48cbff67-vrrsw" evicting pod "router-default-6fc798b98b-9m6kh" evicting pod "prometheus-adapter-5b77ffdd5f-hzqrp" evicting pod "alertmanager-main-1" evicting pod "prometheus-k8s-0" evicting pod "network-check-source-66c7fbb86-2r78z" ``` {: screen} 1. Wait until draining finishes, then complete the following steps to update the worker node. ## Clean up persistent volumes for bare metal worker nodes {: #cleanup-pv-baremetal-vpc} {: step} [Major update]{: tag-red} [Minor update]{: tag-blue} [Worker replace]{: tag-green} **Bare metal worker nodes only**: If you are updating or replacing a bare metal worker node, complete the following steps to clean up persistent volumes and prepare the node for the new deployment. If you are working with virtual server instance (VSI) worker nodes, skip this section and continue to [Update the worker node](#upgrade-worker-node-vpc). {: important} Before you begin, make sure you have completed the previous steps to cordon and drain the worker node. 1. Identify any persistent volumes (PVs) in `Released` state that are associated with the `localblock` storage class on the node you are updating. ```sh oc get pv -L kubernetes.io/hostname | grep localblock | grep Released ``` {: pre} Example output ```sh local-pv-d6bf175b 1490Gi RWO Delete Released openshift-storage/ocs-deviceset-0-data-0-6c5pw localblock 2d22h compute-1 ``` {: screen} 1. If there are any PVs in `Released` state, delete them. Replace `` with the name of the PV from the previous step. ```sh oc delete pv ``` {: pre} Example command ```sh oc delete pv local-pv-d6bf175b ``` {: pre} Example output ```sh persistentvolume "local-pv-d6bf175b" deleted ``` {: screen} 1. Wipe the ODF disks on the bare metal node to prepare for new persistent volume creation. Start a debug pod on the node you are updating. Replace `` with the name of your bare metal worker node. ```sh kubectl debug node/ -it --image=registry.access.redhat.com/ubi8/ubi ``` {: pre} Example command ```sh kubectl debug node/kube-d8g2ek0l0bd8e4oss13g-bhargavibmu-default-000003b3 -it --image=registry.access.redhat.com/ubi8/ubi ``` {: pre} 1. Inside the debug pod, change to the host root directory. ```sh chroot /host ``` {: pre} 1. Wipe each NVMe disk that was used by ODF. Adjust the disk range (`nvme{0..7}n1`) based on the number of disks in your configuration. ```sh for disk in /dev/nvme{0..7}n1; do echo "Wiping $disk..." wipefs -af $disk dd if=/dev/zero of=$disk bs=1M count=100 sgdisk --zap-all $disk 2>/dev/null || true done ``` {: pre} 1. Verify that the disks are clean and no longer have any filesystem signatures. ```sh for disk in /dev/nvme{0..7}n1; do echo "=== $disk ===" blkid $disk 2>&1 || echo "Clean" done ``` {: pre} Each disk should show "Clean" in the output, indicating that all filesystem signatures have been removed. 1. Exit the debug pod. ```sh exit exit ``` {: pre} 1. List the `localvolumediscoveryresults` resources to find the entry for the node you are updating. ```sh kubectl get localvolumediscoveryresults -n openshift-local-storage ``` {: pre} Example output ```sh NAME AGE discovery-result-kube-d8g2ek0l0bd8e4oss13g-bhargavibmu-default-000003b3 5d discovery-result-kube-d8g2ek0l0bd8e4oss13g-bhargavibmu-default-000004a1 5d ``` {: screen} 1. Delete the `localvolumediscoveryresults` resource for the node you are updating. Replace `` with the name from the previous step. ```sh kubectl delete localvolumediscoveryresults -n openshift-local-storage ``` {: pre} Example command ```sh kubectl delete localvolumediscoveryresults discovery-result-kube-d8g2ek0l0bd8e4oss13g-bhargavibmu-default-000003b3 -n openshift-local-storage ``` {: pre} After completing these steps, new persistent volumes will be automatically created and scheduled on the bare metal node after it is reloaded. Continue to the next section to update the worker node. ## Update the worker node {: #upgrade-worker-node-vpc} {: step} [Major update]{: tag-red} [Minor update]{: tag-blue} [Worker replace]{: tag-green} 1. List your worker nodes by using the `ibmcloud oc worker ls` command and find the worker node that you cordoned and drained in the previous step. ```sh ibmcloud oc worker ls -c CLUSTER ``` {: pre} Example output ```sh ID Primary IP Flavor State Status Zone Version kube-c85ra07w091uv4nid9ug-vpcoc-default-000001c1 10.241.128.4 bx2.4x16 normal Ready us-east-3 4.8.29_1544_openshift* kube-c85ra07w091uv4nid9ug-vpcoc-default-00000288 10.241.0.4 bx2.4x16 normal Ready us-east-1 4.8.29_1544_openshift* kube-c85ra07w091uv4nid9ug-vpcoc-default-00000352 10.241.64.4 bx2.4x16 normal Ready us-east-2 4.8.29_1544_openshift* ``` {: pre} 1. Update the worker node. For bare metal worker nodes, use the `worker reload` command. For virtual server instance (VSI) worker nodes, use the `worker replace` command. **Bare metal worker nodes**: Use the `worker reload` command to reload the worker node. This command is supported for VPC bare metal workers. ```sh ibmcloud oc worker reload --worker kube-*** ``` {: pre} **VSI worker nodes**: [Minor update]{: tag-blue} Example command to replace the worker node and apply the latest patch update. ```sh ibmcloud oc worker replace -c CLUSTER --worker kube-*** --update ``` {: pre} **VSI worker nodes**: [Worker replace]{: tag-green} Example command to replace the worker node without applying the latest patch update. ```sh ibmcloud oc worker replace -c CLUSTER --worker kube-*** ``` {: pre} Example output for VSI worker node replacement: ```sh The replacement worker node is created in the same zone with the same flavor, but gets new public or private IP addresses. During the replacement, all pods might be rescheduled onto other worker nodes and data is deleted if not stored outside the pod. To avoid downtime, ensure that you have enough worker nodes to handle your workload while the selected worker nodes are being replaced. Replace worker node kube-c85ra07w091uv4nid9ug-cluster-default-00000288? [y/N]> y Deleting worker node kube-c85ra07w091uv4nid9ug-cluster-default-00000288 and creating a new worker node in cluster ``` {: screen} 1. Wait for the worker node to be reloaded or replaced and then list your worker nodes. Note that this process might take 20 minutes or more. ```sh oc get nodes ``` {: pre} Example output ```sh NAME STATUS ROLES AGE VERSION 10.241.0.4 Ready master,worker 22d v1.21.6+bb8d50a 10.241.128.4 Ready master,worker 22d v1.21.6+bb8d50a 10.241.64.4 Ready master,worker 22d v1.21.6+bb8d50a ``` {: screen} ## Clean up the resources from the old node {: #cleanup-os-storage-vpc} {: step} [Major update]{: tag-red} [Minor update]{: tag-blue} [Worker replace]{: tag-green} 1. Verify OSD pod has come up on the replaced node in a `running` state. If the pod is running, continue to [step 7](#add-storage-node-vpc). If the pod has failed, perform the following steps.If more than one OSD pod is not `Running`, stop and contact support. Open a [support case](https://cloud.ibm.com/docs/support?topic=support-using-avatar&format=markdown). In the case details, be sure to include any relevant log files, error messages, or command outputs. 1. Navigate to the `openshift-storage` project. ```sh oc project openshift-storage ``` {: pre} 1. Remove the failed OSD from the cluster. You can specify multiple failed OSDs if required. ```sh oc process -n openshift-storage ocs-osd-removal -p FAILED_OSD_IDS= -p FORCE_OSD_REMOVAL=true | oc create -f - ``` {: pre} The `FAILED_osd_id` value is the integer in the pod name immediately after the `rook-ceph-osd` prefix. The `FORCE_OSD_REMOVAL` value must be changed to `true` in clusters that have only three OSDs, or clusters with insufficient space to restore all three replicas of the data after the OSD is removed. {: note} 1. Verify that the OSD was removed successfully by checking the status of the `ocs-osd-removal-job` pod. ```sh oc get pod -l job-name=ocs-osd-removal-job -n openshift-storage ``` {: pre} 1. Verify that the OSD removal is completed. ```sh oc logs -l job-name=ocs-osd-removal-job -n openshift-storage --tail=-1 | egrep -i 'completed removal' ``` {: pre} Example output ```sh 2023-03-10 06:50:04.501511 I | cephosd: completed removal of OSD 0 ``` {: screen} ## Add the new storage node {: #add-storage-node-vpc} {: step} Before adding new storage nodes, make sure you've completed the previous steps for all storage nodes in the cluster. {: important} [Major update]{: tag-red} [Minor update]{: tag-blue} [Worker replace]{: tag-green} 1. If you limited your ODF deployment to a subset of worker nodes by specifying node names during installation, you must update the `ocscluster` CRD to include the new name. If you didn't limit your configuration to only certain worker nodes, you don't need to update the `ocscluster` CRD. {: important} ```sh oc edit ocscluster ``` {: pre} ```yaml apiVersion: ocs.ibm.io/v1 kind: OcsCluster metadata: name: ocscluster-auto spec: . . . osdSize: 250Gi osdStorageClassName: ibmc-vpc-block-metro-10iops-tier workerNodes: - NODE-NAME # Example 10.248.128.42 - NODE-NAME - NODE-NAME ``` {: screen} 1. Wait for the OpenShift Data Foundation pods to deploy to the new worker. Verify the new persistent volumes are created and that all pods are in a `Running` state. ```sh oc get pv oc get ocscluster oc get pods -n openshift-storage ``` {: pre} 1. Verify that all other required OpenShift Data Foundation pods are in Running state. ```sh oc get pod -n openshift-storage | grep mon ``` {: pre} Example output: ```sh rook-ceph-mon-a-cd575c89b-b6k66 2/2 Running 0 38m rook-ceph-mon-b-6776bc469b-tzzt8 2/2 Running 0 38m rook-ceph-mon-d-5ff5d488b5-7v8xh 2/2 Running 0 4m8s ``` {: screen} 1. Verify that new OSD pods are running on the replacement node. ```sh oc get pods -o wide -n openshift-storage| egrep -i | egrep osd ``` {: pre} ## Update the OpenShift Data Foundation add-on {: #update-ocs-add-on-vpc} {: step} [Major update]{: tag-red} 1. Check the existing version. ```sh ibmcloud oc cluster addon ls --cluster CLUSTER ``` {: pre} 1. Update the add-on. ```sh ibmcloud oc cluster addon update openshift-data-foundation --cluster CLUSTER --version VERSION ``` {: pre} 1. Verify the add-on is updated. ```sh ibmcloud oc cluster addon ls --cluster CLUSTER ``` {: pre} ## Update your cluster resource {: #update-ocs-resource-yaml-vpc} {: step} [Major update]{: tag-red} 1. Get the name of your `ocscluster` resource. ```sh oc get ocscluster ``` {: pre} Example output ```sh NAME AGE ocscluster-vpc 19d ``` {: screen} 1. Run the following command to edit your `ocscluster` resource. ```sh oc edit ocscluster OCS-CLUSTER-NAME ``` {: pre} 1. Set the `ocsUpgrade` parameter to `true`. ```yaml ... spec: billingType: hourly monSize: 20Gi monStorageClassName: ibmc-vpc-block-10iops-tier numOfOsd: 1 ocsUpgrade: true osdSize: 250Gi osdStorageClassName: ibmc-vpc-block-10iops-tier status: storageClusterStatus: Decreasing the capacity not allowed ``` {: codeblock} 1. Save and close the file. 1. Wait for the update to complete. 1. Verify that the `storagecluster` and `cephcluster` resources are both deployed correctly. ```sh oc get storagecluster -n openshift-storage NAME AGE PHASE EXTERNAL CREATED AT VERSION ocs-storagecluster 43h Ready 2023-06-21T09:22:00Z 4.11.0 ``` {: screen} ```sh oc get cephcluster -n openshift-storage NAME DATADIRHOSTPATH MONCOUNT AGE PHASE MESSAGE HEALTH EXTERNAL ocs-storagecluster-cephcluster /var/lib/rook 3 43h Ready Cluster created successfully HEALTH_OK ``` {: screen} ```sh oc get csv -n openshift-storage NAME DISPLAY VERSION REPLACES PHASE mcg-operator.v4.11.8 NooBaa Operator 4.11.8 mcg-operator.v4.11.7 Succeeded ocs-operator.v4.11.8 OpenShift Container Storage 4.11.8 ocs-operator.v4.11.7 Succeeded odf-csi-addons-operator.v4.11.8 CSI Addons 4.11.8 odf-csi-addons-operator.v4.11.7 Succeeded odf-operator.v4.11.8 OpenShift Data Foundation 4.11.8 odf-operator.v4.11.7 Succeeded ``` {: screen}