Verifying your OpenShift Data Foundation Regional Disaster Recovery configuration

Virtual Private Cloud 4.21 and later

After you set up OpenShift Data Foundation (ODF) Regional Disaster Recovery (RDR), verify that each component of the stack is healthy and properly configured. Work through each section on the appropriate cluster, hub or managed, as indicated.

Verifying the ACM add-on on the hub cluster

Hub cluster

Complete the following steps on the hub cluster.

Verify that the ACM add-on is installed and in a normal state. Look for the acm add-on in the output and confirm the status shows Addon Ready.
```
ibmcloud oc cluster addon ls --cluster HUB_CLUSTER_NAME | grep -i acm
```
Example output:
```
acm   2.15.0   normal   Addon Ready. For more info: http://ibm.biz/addon-state (H1500)
```
If the ACM add-on is not listed, install the ACM add-on.

Verify that the ACM add-on pods are running in the kube-system namespace. Look for the acm-agent and ibm-acm-operator-controller-manager pods and confirm that both show Running status.

oc get pods -n kube-system | grep acm

Example output:

acm-agent-6d9d49cbf6-tsmxd                           1/1   Running   118 (107m ago)   19d
ibm-acm-operator-controller-manager-bc7b7f969-pr9lj   1/1   Running   3 (7d17h ago)    19d

Verify that the ACM operator pods and MultiClusterHub pods are running in the open-cluster-management namespace.

oc get pods -n open-cluster-management

Example output:

multiclusterhub-operator-dcd787649-sqrcx     1/1   Running   0   14d
console-chart-console-v2-7c8c6649cc-hmg6g    1/1   Running   0   15d
submariner-addon-76986fdf45-v9kgm            1/1   Running   0   15d
volsync-addon-controller-6575568dc6-6pjsb    1/1   Running   0   15d

If any pods are missing or not in Running status, see Installing the ACM add-on.

Verifying that managed clusters are imported and available

Hub cluster

Complete the following step on the hub cluster.

Verify that all managed clusters are successfully imported and show True in both the JOINED and AVAILABLE columns.

oc get managedclusters

Example output:

NAME             HUB ACCEPTED   MANAGED CLUSTER URLS                                 JOINED   AVAILABLE   AGE
local-cluster    true           https://c100.au-syd.containers.cloud.ibm.com:30253   True     True        14d
managedcluster1  true           https://c100.au-syd.containers.cloud.ibm.com:31003   True     True        14d
managedcluster2  true           https://c100.au-syd.containers.cloud.ibm.com:31888   True     True        14d

If a managed cluster is missing or unavailable, reimport the cluster.

Verifying Submariner connectivity

Hub cluster Managed cluster

Submariner provides network connectivity between your managed clusters. Complete the steps in each subsection on the cluster indicated.

Verify the Submariner operator on the hub cluster

Hub cluster

Verify that the Submariner operator pod is running on the hub cluster.

oc get pods -n submariner-operator

Example output:

NAME                                  READY   STATUS    RESTARTS      AGE
submariner-operator-6bf8d56f74-7v44v  1/1     Running   3 (17h ago)   14d

Verify that the Submariner add-on is registered for each managed cluster. Confirm that the AVAILABLE column shows True for each managed cluster.
```
oc get managedclusteraddon -A | grep submariner
```
Example output:
```
managedcluster1   submariner   True   False
managedcluster2   submariner   True   False
```

Verify that the Submariner broker exists.

oc get broker -A

Example output:

NAMESPACE                 NAME                AGE
submariner-set1-broker    submariner-broker   12d

Verify Submariner on the managed clusters

Managed cluster

Run the following steps on each managed cluster.

Verify that the required Submariner pods are running. Confirm that submariner-gateway, submariner-routeagent, and submariner-globalnet pods are present and show Running status.

oc get pods -n submariner-operator

Example output:

NAME                                              READY   STATUS    RESTARTS      AGE
submariner-addon-bc6bbf579-wjqhq                  1/1     Running   0             12d
submariner-gateway-hxcbh                          1/1     Running   0             12d
submariner-globalnet-jbmxw                        1/1     Running   3 (12d ago)   12d
submariner-lighthouse-agent-858896855b-xghxn      1/1     Running   0             12d
submariner-lighthouse-coredns-66d5579b66-52f6j    1/1     Running   0             12d
submariner-lighthouse-coredns-66d5579b66-jhqkm    1/1     Running   0             12d
submariner-metrics-proxy-65n4c                    2/2     Running   3 (12d ago)   12d
submariner-operator-7c99f89cfb-rppth              1/1     Running   2 (17h ago)   12d
submariner-routeagent-27p7v                       1/1     Running   0             12d
submariner-routeagent-jjw66                       1/1     Running   0             12d
submariner-routeagent-nt78s                       1/1     Running   0             12d

Verify that the Submariner connection is healthy. In the output, check that RouteAgentConnectionDegraded has a status of "False", which confirms that all route agent connections to remote endpoints are established and healthy.
```
oc -n <managed_cluster_name> get managedclusteraddons submariner -o yaml
```
Example output:
```
status:
  - lastTransitionTime: "2026-03-18T03:36:24Z"
    message: All RouteAgent connections to remote endpoints are established and healthy.
    reason: ConnectionsEstablished
    status: "False"
    type: RouteAgentConnectionDegraded
```
If connectivity is degraded, reinstall Submariner by following Step 4 of the ODF Regional Disaster Recovery setup.

Verifying ODF on the managed clusters

Managed cluster

Run the following steps on each managed cluster.

Verify that the ODF StorageCluster is ready and Ceph health is HEALTH_OK. In the output, confirm that the PHASE column shows Ready for the StorageCluster and the HEALTH column shows HEALTH_OK for the CephCluster.

oc get storagecluster -n openshift-storage
oc get cephcluster -n openshift-storage

Example output:

NAME                 AGE   PHASE   EXTERNAL   CREATED AT             VERSION
ocs-storagecluster   12d   Ready              2026-03-17T10:51:46Z   4.20.7

NAME                              DATADIRHOSTPATH   MONCOUNT   AGE   PHASE   MESSAGE                       HEALTH      EXTERNAL   FSID
ocs-storagecluster-cephcluster    /var/lib/rook     3          12d   Ready   Cluster created successfully  HEALTH_OK              5fb59e70-bd54-438f-9190-c09ddcf04c0c

If this is a fresh ODF installation and no applications have been deployed, consider reinstalling ODF by following the ODF installation guide.

Verify that the required ServiceExports are present. Confirm that ocs-provider-server and the rook-ceph-mon-* and rook-ceph-osd-* entries are listed.
```
oc get serviceexports -n openshift-storage
```
Example output:
```
NAME                  AGE
ocs-provider-server   12d
rook-ceph-mon-d       12d
rook-ceph-mon-e       12d
rook-ceph-mon-f       12d
rook-ceph-osd-0       12d
rook-ceph-osd-1       12d
rook-ceph-osd-2       12d
```
If the ocs-provider-server ServiceExport is missing, recreate it by following Step 5 of the ODF Regional Disaster Recovery setup.
Verify that the multicluster network configuration is properly set on the StorageCluster. In the output, confirm that multiClusterService.enabled is true and that clusterID matches the name of the managed cluster. Also confirm that the ocs.openshift.io/api-server-exported-address annotation is present and correctly set.
```
oc get storagecluster -n openshift-storage -o yaml
```
Look for the following values in the output:
```
network:
  multiClusterService:
    clusterID: <cluster-name>
    enabled: true
```
And the following annotation:
```
ocs.openshift.io/api-server-exported-address: <cluster-name>.ocs-provider-server.openshift-storage.svc.clusterset.local:50051
```

Verifying the ODF Multicluster Orchestrator on the hub cluster

Hub cluster

Complete the following steps on the hub cluster.

Verify that the OpenShift GitOps operator pods are running. The ODF Multicluster Orchestrator requires GitOps to be installed first.

oc get pods -n openshift-gitops

Example output:

NAME                                                           READY   STATUS    RESTARTS   AGE
cluster-6484df7d8f-fbk9b                                       1/1     Running   0          4d12h
gitops-plugin-8646dcfd5d-kj5mf                                 1/1     Running   0          4d12h
openshift-gitops-application-controller-0                      1/1     Running   0          4d12h
openshift-gitops-applicationset-controller-664645457b-8n9gj    1/1     Running   0          4d12h
openshift-gitops-dex-server-74ddb4bb94-trxk6                   1/1     Running   0          4d12h
openshift-gitops-redis-68469975f8-mlkrq                        1/1     Running   0          14d
openshift-gitops-repo-server-76567d7c46-j9fjq                  1/1     Running   0          4d12h
openshift-gitops-server-6d74f8d998-f97zx                       1/1     Running   0          4d12h

If GitOps is not installed, see Installing Red Hat OpenShift GitOps Operator in web console.

Verify that the ODF Multicluster Orchestrator operator pods are running in the openshift-operators namespace. Confirm that odf-multicluster-console and odfmo-controller-manager pods both show Running status.
```
oc get pods -n openshift-operators | grep -i odf
```
Example output:
```
odf-multicluster-console-cdd786658-2bdbz   1/1   Running   0             12d
odfmo-controller-manager-8585c4bf9-mwhvf   1/1   Running   2 (16h ago)   12d
```

Verify that the Ramen hub operator pod is running in the submariner-operator namespace.

oc get pods -n submariner-operator

Example output:

NAME                                  READY   STATUS    RESTARTS      AGE
submariner-operator-6bf8d56f74-7v44v  1/1     Running   3 (17h ago)   14d

Verify that the Ramen DR cluster operator pod is running on each managed cluster in the openshift-dr-system namespace.
```
oc get pods -n openshift-dr-system | grep -i ramen
```
Example output:
```
ramen-dr-cluster-operator-785d878dbc-lbrgz   2/2   Running   0   2d15h
```

Verifying the DR policy and MirrorPeer on the hub cluster

Hub cluster

Complete the following steps on the hub cluster.

Verify that the DRPolicy is validated. In the output, confirm that the Validated condition has a status of "True".
```
oc get drpolicy -A
oc describe drpolicy <drpolicy_name>
```
Example output from oc describe:
```
status:
  conditions:
    - type: Validated
      status: "True"
```
If the DRPolicy is not validated, verify that NooBaa object buckets were created successfully on both managed clusters.
Verify that the MirrorPeer phase is ExchangedSecret. Describe the MirrorPeer resource and check the Phase field.
```
oc get mirrorpeers -A
oc describe mirrorpeer <mirrorpeer_name>
```
Example output from oc describe:
```
Status:
  Phase: ExchangedSecret
Events: <none>
```
If the phase shows ExchangingSecret, verify Submariner connectivity by following Verifying Submariner connectivity.

Verifying optional operators on the managed clusters

Managed cluster

If you installed optional operators as part of your ODF RDR setup, verify that they are running on each managed cluster.

You are responsible for managing optional operators, including updating, monitoring, recovery, and reinstallation.

Verify that the OpenShift GitOps operator pods are running. Confirm that the openshift-gitops-application-controller, openshift-gitops-server, and other GitOps pods show Running status.
```
oc get pods -n openshift-gitops
```

Verify that the OpenShift API for Data Protection (OADP) operator pods are running. Confirm that openshift-adp-controller-manager and velero pods both show Running status.

oc get pods -n openshift-adp

Example output:

NAME                                                  READY   STATUS    RESTARTS   AGE
openshift-adp-controller-manager-7c8fc7f964-j2jnq    1/1     Running   1          12d
velero-9cdf64b8b-9c2kk                               1/1     Running   1          12d

If an operator is missing, reinstall it by following the relevant installation guide.

Collecting must-gather logs

Hub cluster Managed cluster

If you cannot resolve an issue by working through the previous verification steps, collect must-gather logs from your hub and managed clusters and share them with IBM Support.

Collect hub cluster logs

Hub cluster

Run the following script on the hub cluster to collect diagnostic information.

#!/bin/bash
# Usage: ./hub-check.sh <cluster-name>
CLUSTER_NAME=$1
if [[ -z "$CLUSTER_NAME" ]]; then
  echo "Usage: $0 <cluster-name>"
  exit 1
fi

echo "===== Verify ACM ====="
ibmcloud oc cluster addon ls --cluster "$CLUSTER_NAME" | grep -i acm
oc get pods -n kube-system | grep acm
oc get pods -n open-cluster-management
oc get managedclusters

echo "===== Verify Submariner ====="
oc get pods -n submariner-operator
oc get managedclusteraddon -A | grep submariner
oc get managedclusteraddon -A -o yaml
oc get broker -A

echo "===== Verify Multicluster Operator ====="
oc get pods -n openshift-operators | grep -i odf
oc get drpolicy -A
oc get mirrorpeers -A

echo "===== Verify Additional Operators ====="
oc get pods -n openshift-gitops
oc get pods -n openshift-adp

Collect managed cluster logs

Managed cluster

Run the following script on each managed cluster to collect diagnostic information.

#!/bin/bash
# Usage: ./managedCluster-check.sh

echo "===== Verify Submariner ====="
oc get pods -n submariner-operator
oc get submariner -A

echo "===== Verify DR Operator ====="
oc get pods -n openshift-dr-system | grep -i ramen

echo "===== Verify ODF ====="
oc get storagecluster -n openshift-storage
oc get cephcluster -n openshift-storage
oc get serviceexports -n openshift-storage
oc get storagecluster -n openshift-storage -o yaml

echo "===== Verify Additional Operators ====="
oc get pods -n openshift-gitops
oc get pods -n openshift-adp

For additional must-gather guidance, see the following resources:

Next steps

If all components are healthy, test your disaster recovery configuration by running a failover and relocate scenario.
If you need further assistance, contact IBM Support and include the must-gather logs you collected.
Review the ODF Regional Disaster Recovery setup guide to confirm all installation steps were completed correctly.

Verifying your OpenShift Data Foundation Regional Disaster Recovery configuration

Verifying your OpenShift Data Foundation Regional Disaster Recovery configuration

Verifying the ACM add-on on the hub cluster

Verifying that managed clusters are imported and available

Verifying Submariner connectivity

Verify the Submariner operator on the hub cluster

Verify Submariner on the managed clusters

Verifying ODF on the managed clusters

Verifying the ODF Multicluster Orchestrator on the hub cluster

Verifying the DR policy and MirrorPeer on the hub cluster

Verifying optional operators on the managed clusters

Collecting must-gather logs

Collect hub cluster logs

Collect managed cluster logs

Next steps

Related links