Before you begin

Before you create the clusters, gather the VPC and Cloud Object Storage details you need to populate in the cluster creation commands.

1. Retrieve your VPC IDs. Note the ID of the VPC you want to use for each cluster.

   ibmcloud is vpcs
   

2. Retrieve the subnet details for a specific VPC. Note the subnet IDs you want to use for each cluster.

   ibmcloud is subnets --vpc VPC_ID
   

3. List your Cloud Object Storage instances.

   ibmcloud resource service-instances --service-name cloud-object-storage
   

4. Retrieve the CRN of the instance you want to use. Note the value in the ID field.

   ibmcloud resource service-instance SERVICE_INSTANCE
   

Step 1. Create the hub cluster

Hub cluster

This is the cluster you install ACM on to manage the primary and secondary ODF clusters. Make sure your hub cluster has at least 16 vCPU x 64 GB compute capacity available.

For each cluster, make sure to allow outbound traffic by including the --disable-outbound-traffic-protection parameter in the CLI or selecting the option to disable outbound traffic protection in the UI.

1. Create a VPC cluster in us-east to install ACM on. This is the hub cluster that you can use to manage your ODF clusters. Make sure your hub cluster has at least 3 worker nodes that run RHCOS, available compute capacity of at least 16 vCPU and 64 GB, outbound traffic disabled, and meets all of the prerequisites for ACM. The following example command creates a cluster for ACM in us-east.

   ibmcloud ks cluster create vpc-gen2 --flavor bx2.16x64 --name acm-hub-cluster-dr-odf --subnet-id SUBNET_ID --vpc-id VPC_ID --zone us-east-2 --version 4.21.31_openshift --workers 3 --cos-instance COS_CRN --disable-outbound-traffic-protection --cni OVNKubernetes
   

2. Note the cluster ID from the output. You need it in a later step.

Step 2. Create a trusted profile for the hub cluster

Hub cluster

1. Create the trusted profile.

   ibmcloud iam trusted-profile-create acm-operator-profile
   

2. Create the compute resource trust rule, scoped to the kube-system namespace on Red Hat OpenShift compute resources.

   ibmcloud iam trusted-profile-rule-create acm-operator-profile \
     --name kube-system-rule \
     --type Profile-CR \
     --conditions claim:namespace,operator:EQUALS,value:kube-system \
     --cr-type ROKS_SA
   

3. Assign the IAM access policy to the profile. Replace CLUSTER_ID with your hub cluster ID.

   ibmcloud iam trusted-profile-policy-create acm-operator-profile \
     --roles Reader,Viewer,Operator,Editor \
     --service-name containers-kubernetes \
     --service-instance CLUSTER_ID
   

4. Assign the trusted profile to the hub cluster. After you assign a trusted profile to a cluster, it cannot be removed.

   ibmcloud oc experimental trusted-profile set --cluster CLUSTER_NAME_OR_ID --trusted-profile TRUSTED_PROFILE_ID
   

5. Verify that the trusted profile secret was created in the cluster. This command can take up to 10 minutes to complete. Wait for the secret to appear before proceeding to install the ACM add-on. If you proceed before the secret is created, the ACM add-on installation will fail.

   oc get secrets -n kube-system | grep ibm-cloud-credentials
   

6. If you are using ODF version 4.21 or later, install the OpenShift GitOps operator on the hub cluster.

   1. On the Core platform perspective of the hub cluster's OpenShift web console, navigate to Ecosystem > Software Catalog and search for Red Hat OpenShift GitOps.

   2. Click the Red Hat OpenShift GitOps tile.

   3. On the Install Operator page, select an Update channel and a GitOps version to install.

   4. Choose an Installed Namespace. The default installation namespace is openshift-gitops-operator.

      For GitOps version 1.10 and later, the default namespace changed from openshift-operators to openshift-gitops-operator.

   5. Select the Enable Operator recommended cluster monitoring on this Namespace checkbox to enable cluster monitoring.

   6. Click Install. Red Hat OpenShift GitOps is installed in all namespaces of the cluster.

   7. Verify that the Red Hat OpenShift GitOps Operator is listed in Operators > Installed Operators and that the Status shows Succeeded.

   After installation, OpenShift GitOps automatically sets up a ready-to-use Argo CD instance in the openshift-gitops namespace, and an Argo CD icon is displayed in the console toolbar.

Step 3. Create the managed clusters

Managed cluster

1. Create a VPC cluster in us-east with at least 3 worker nodes that run RHCOS, available compute capacity of at least 16 vCPU and 64 GB, and outbound traffic protection disabled. This will be the primary managed ODF cluster. The following example command creates a cluster in us-east.

   ibmcloud ks cluster create vpc-gen2 --flavor bx2.16x64 --name managed-cluster-1-dr-odf --subnet-id SUBNET_ID --vpc-id VPC_ID --zone us-east-2 --version 4.21.31_openshift --workers 3 --cos-instance COS_CRN --disable-outbound-traffic-protection --cni OVNKubernetes
   

2. Create a VPC cluster in jp-tok with at least 3 worker nodes that run RHCOS, available compute capacity of at least 16 vCPU and 64 GB, and outbound traffic protection disabled. This will be the secondary managed ODF cluster. For high availability, make sure that the secondary cluster's network does not overlap with the primary cluster's network. The following example command creates a cluster in jp-tok.

   ibmcloud ks cluster create vpc-gen2 --flavor bx2.16x64 --name managed-cluster-2-dr-odf --subnet-id SUBNET_ID --vpc-id VPC_ID --zone jp-tok --version 4.21.31_openshift --workers 3 --cos-instance COS_CRN --disable-outbound-traffic-protection --cni OVNKubernetes
   

Step 4. Install the ACM add-on on the hub cluster

Hub cluster

Use the CLI to install the ACM add-on on the hub cluster.

1. Find the default version of the ACM add-on.

   ibmcloud oc cluster addon versions
   

2. Review the ACM add-on options. In the command, specify the default version found in the previous step. Note any options you want to include when you install the add-on.

   ibmcloud oc cluster addon options --addon acm --version DEFAULT_VERSION
   

3. Run the command to enable the add-on. Be sure to specify the billingPlan and isLicenseAccepted parameters.

   ibmcloud oc cluster addon enable acm --cluster HUB_CLUSTER_ID --param 'billingPlan=PLAN' --param 'isLicenseAccepted=BOOLEAN'
   

   Command parameters. See the example command below for an example of each parameter type.

   --cluster

   Required. The ID of the hub cluster to install the ACM add-on to.

   --param 'billingPlan='

   Required. The billing plan you want to select for ACM. Specify KUBERNETES for the ACM for Kubernetes plan.

   --param 'isLicenseAccepted='

   Required. Set this to true to accept the license agreement for the selected billing plan. The add-on will not install successfully unless the license is accepted. By accepting this license, you agree to the applicable terms and conditions and acknowledge your understanding of the services included in the selected plan.

   Example command to install the ACM add-on with the ACM for Kubernetes billing plan.

   ibmcloud oc cluster addon enable acm --cluster a5bcde982dfer2nwxq73 --param 'billingPlan=KUBERNETES' --param 'isLicenseAccepted=true'
   

4. Verify that the add-on installed. It might take several minutes for the add-on to show in the following outputs.

   1. On the hub cluster, check that the acmhub resource is created.

      oc get acmhub
      

      Example output.

          NAME       AGE
          acm-auto   1h
      

   2. On the hub cluster, check the acmhub status.

      oc describe acmhub
      

      Example output.

      Status
          Message: ACM installed successfully
      

Step 5. Import the managed clusters into ACM

Hub cluster Managed cluster

Import both managed clusters into ACM so that the hub cluster can manage them.

1. Open the OpenShift web console for the hub cluster.

2. From the Fleet Management perspective, click Import cluster.

3. Enter the Name of the first managed cluster, select a Cluster set if applicable, and enter Additional labels if applicable.

4. For Import mode, select Run import commands manually and click Next.

5. Optionally select an automation template and click Next.

6. Review the details and click Generate command. Copy the command that is displayed.

7. Log in to the first managed cluster and run the copied command with kubectl configured for that cluster.

8. Repeat steps 2–7 for the second managed cluster.

9. In the Fleet Management perspective, verify that both managed clusters are listed and show a Ready status before proceeding.

Step 6. Configure the Submariner add-on

Hub cluster Managed cluster

Configure the Submariner add-on to establish cross-cluster networking between your two managed clusters. Choose one of the following options based on your infrastructure and cluster types:

• Option 1: Transit Gateway (Preferred) — Supported for both Red Hat OpenShift on IBM Cloud and Red Hat OpenShift Virtualization Service (ROVS) clusters with Virtual Server Instances (VSI) and Bare Metal worker nodes.
• Option 2: Network Load Balancer (NLB) — Supported for Red Hat OpenShift on IBM Cloud clusters with VSI worker nodes only.

Step 7. Install and configure OpenShift Data Foundation

Managed cluster

Install and configure ODF on your 2 managed clusters. Make sure to complete these steps on both the primary and secondary managed cluster.

Before running any oc commands in this section, make sure your context is set to the managed cluster you are configuring. Run ibmcloud oc cluster config --cluster MANAGED_CLUSTER_NAME_OR_ID --admin to switch contexts, then verify with oc config current-context.

1. For each managed cluster, install the OpenShift Data Foundation add-on from the IBM Cloud console.

   1. Navigate to your cluster's Overview page and scroll down to the Add-ons section.
   2. Under OpenShift Data Foundation, click Install.
   3. Check the box for Deploy NooBaa Multi-Cloud Object Gateway.
   4. Click Install again to confirm.
   5. Wait for the ODF add-on status to change from Enabling to Normal (green checkmark) before proceeding.

2. Verify that ODF installed successfully. In the output, check that the status says Ready.

   The UI Normal status reflects that the add-on was deployed, but the ODF operator may still need a few minutes to finish initializing and registering its resources.

   oc get storagecluster -n openshift-storage ocs-storagecluster -o jsonpath='{.status.phase}{"\n"}'
   

The following steps must be completed on each managed cluster. Switch your context to the first managed cluster using ibmcloud oc cluster config --cluster MANAGED_CLUSTER_NAME_OR_ID --admin, complete all steps through the end of this section, then switch to the second managed cluster and repeat.

1. Run the command to update the ACM Managed Cluster Name in the storageCluster resource's multiClusterService section. This allows ODF to use GlobalNet. For more information, see Creating an OpenShift Data Foundation cluster on managed clusters.

   Replace MANAGED_CLUSTER_NAME with the name of the cluster your context is currently pointed at.

   kubectl patch storagecluster -n openshift-storage ocs-storagecluster --type merge -p'{"spec":{"network":{"multiClusterService":{"clusterID":"MANAGED_CLUSTER_NAME","enabled":true}}}}'
   

   Example output.

   storagecluster.ocs.openshift.io/ocs-storagecluster patched
   

2. Verify the service exports. This might take a few minutes to show in the output.

   oc get serviceexport -n openshift-storage
   

   Example output:

   NAME              AGE
   rook-ceph-mon-d   4d14h
   rook-ceph-mon-e   4d14h
   rook-ceph-mon-f   4d14h
   rook-ceph-osd-0   4d14h
   rook-ceph-osd-1   4d14h
   rook-ceph-osd-2   4d14h
   

3. Create a service export for ocs-provider-server.

   oc apply -f - <<EOF
   apiVersion: multicluster.x-k8s.io/v1alpha1
   kind: ServiceExport
   metadata:
     name: ocs-provider-server
     namespace: openshift-storage
   EOF
   

   Example output.

   serviceexport.multicluster.x-k8s.io/ocs-provider-server created
   

4. Run the command to update the storageCluster resource to use the ocs-provider-server service export you created.

   oc annotate storagecluster ocs-storagecluster -n openshift-storage ocs.openshift.io/api-server-exported-address=MANAGED_CLUSTER_NAME.ocs-provider-server.openshift-storage.svc.clusterset.local:50051.
   

   Example output.

   storagecluster.ocs.openshift.io/ocs-storagecluster annotated
   

5. Verify that the storageCluster resource is ready.

   oc get storagecluster -n openshift-storage
   

   Example output.

   NAME                    PHASE  
   ocs-storagecluster      Ready   
   

Step 8. Configure the Regional Disaster Recovery policy

Hub cluster

Install the ODF Multicluster Orchestrator on your hub cluster and create the disaster recovery (DR) policy that enables mirroring between your two managed clusters.

1. Install the ODF Multicluster Orchestrator on the hub cluster.

   1. Install the OpenShift GitOps operator on the hub cluster if you haven't already. For installation steps, see the end of Step 2. Create a trusted profile for the hub cluster.
   2. On the Core platform perspective of the hub cluster's OpenShift web console, navigate to Ecosystem > Software Catalog and search for ODF Multicluster Orchestrator.
   3. Click the ODF Multicluster Orchestrator tile. Make sure to select the same version number as the ODF version you installed onto the managed clusters in the previous section. Keep all other default settings and click Install.
   4. Ensure that the operator resources are installed in the openshift-operators project and available to all namespaces. Click Install again to confirm.

   The ODF Multicluster Orchestrator also installs the OpenShift DR Hub Operator on the hub cluster as a dependency.

2. Verify the installation by checking that the operator pods are running. Make sure your CLI context is set to the hub cluster before running this command.

   oc get pods -n openshift-operators
   

   Example output.

   NAME                                        READY   STATUS       RESTARTS    AGE
   odf-multicluster-console-6845b795b9-blxrn   1/1     Running      0           4d20h
   odfmo-controller-manager-f9d9dfb59-jbrsd    1/1     Running      0           4d20h
   ramen-hub-operator-6fb887f885-fss4w         2/2     Running      0           4d20h
   

3. On the hub cluster, create a DR policy with a 5 minute sync interval and specify each managed cluster in the parameters. This creates NooBaa object buckets on both managed clusters and enables ODF Ceph block pool mirroring for volume replication.

   1. On the Fleet Management perspective of the hub cluster's OpenShift web console, navigate to Data Services > Disaster recovery > Policies > Create DRPolicy.

   2. Create a DR policy that includes the following parameters.

      • Connected clusters: PRIMARY_MANAGED_CLUSTER_NAME, SECONDARY_MANAGED_CLUSTER_NAME
      • Replication policy: Asynchronous
      • Replication interval: 5m
      • If applicable, select Enable disaster recovery support for restored and cloned PersistentVolumeClaims (For Data Foundation only) under Advanced settings.

      Red Hat explicitly states that this option should only be used with discovered applications and environments where cloned/restored RBD volumes are actively supported.

4. On the hub cluster, run the following commands to verify that the DR policy was created and applied to the managed clusters. Make sure your CLI context is set to the hub cluster before running these commands.

   ibmcloud oc cluster config --cluster HUB_CLUSTER_NAME --admin
   

   oc get drpolicy DRPOLICY_NAME -o jsonpath='{.status.conditions[].reason}{"\n"}'
   

   Example output.

   Succeeded
   

   oc get drclusters
   

   Example output.

   NAME               AGE
   managed-cluster1   4m42s
   managed-cluster2   4m42s
   

5. On each managed cluster, verify that the DR policy was applied and is in a healthy state. Switch your CLI context to each managed cluster before running these commands.

   ibmcloud oc cluster config --cluster MANAGED_CLUSTER_NAME --admin
   

   oc get csv,pod -n openshift-dr-system
   

   Example output.

   NAME                                                                          DISPLAY                         VERSION        REPLACES   PHASE
   clusterserviceversion.operators.coreos.com/odr-cluster-operator.v4.15.0       Openshift DR Cluster Operator   4.15.0                    Succeeded
   clusterserviceversion.operators.coreos.com/volsync-product.v0.8.0             VolSync                         0.8.0                     Succeeded
   
   NAME                                             READY   STATUS    RESTARTS   AGE
   pod/ramen-dr-cluster-operator-6467cf5d4c-cc8kz   2/2     Running   0          3d12h
   

   oc get cephblockpool ocs-storagecluster-cephblockpool -n openshift-storage -o jsonpath='{.status.mirroringStatus.summary}{"\n"}'
   

   Example output.

   {"daemon_health":"OK","health":"OK","image_health":"OK","states":{}}
   

6. Optional: Review the operators you can install to enhance ODF Regional Disaster Recovery features.

7. Optional: Test your disaster recovery configuration.

Optional operators for ODF Regional Disaster Recovery

Review the optional operators you can install on your ACM hub or managed clusters to enhance ODF Regional Disaster Recovery features. Note that IBM is not responsible for managing these operators.

You are responsible for managing these operators, including but not limited to updating, monitoring, recovery, and re-installation.

Testing your disaster recovery configuration

Create a sample application to test your disaster recovery solution. For more information, see Create sample application for testing disaster recovery application.

1. Deploy a subscription-based application from the ACM Console. The application's topology tab shows green when all application resources are deployed successfully.

2. On the application page, go to Actions > Manage Data Policy.

3. Assign the DR policy created earlier to this application.

4. Verify that the application pods are running on the primary cluster.

5. On the application page, go to Actions > Failover application. Select your secondary ODF cluster as the target cluster. Click Initiate.

6. Verify that the application pods are moved to the secondary cluster.

7. On the application page, go to Actions > Relocate application. Select your primary ODF cluster as the target cluster. Click Initiate.

8. Verify that the application pods are moved back to the primary cluster.

Upgrading your ODF Regional Disaster Recovery environment

For information about when and how to upgrade the components of your ODF-RDR environment, see Upgrading your ODF Regional Disaster Recovery environment.

Troubleshooting

If you encounter issues with your ODF Regional Disaster Recovery configuration, see Verifying your OpenShift Data Foundation Regional Disaster Recovery configuration to check the health of each component in your setup.

Supported applications and workloads

Review the types of applications and workloads that you can apply Regional Disaster Recovery for after you complete the setup.

Subscription-based

An application is deployed from an external source, such as GitHub, a Helm repo, or Object Storage.

For more information, see Creating a sample Subscription-based application in the Red Hat documentation.

ApplicationSet-based

An application is deployed from a GitHub repo using the GitOps operator, which manages continuous delivery. This includes two subtypes:

• GitOps Pull Model (ArgoCD pull): A managed cluster pulls the application from GitHub using the GitOps operator.

• GitOps Push Model (ArgoCD push): The GitOps operator pushes the application to the managed cluster during deployments and updates.

For more information, see Creating Application-set based applications in the Red Hat documentation.

For more information on the GitOps subtypes, see Deploying Argo CD with Push and Pull model in the Red Hat documentation.

Discovered applications

An application was pre-deployed in a managed cluster without using ACM. In this case, you can use ACM discovery for the pre-installed app and still configure the DR policy.

For more information, see Disaster recovery protection for discovered applications in the Red Hat documentation.

Applications that include VM deployments

A VM-based application is deployed onto the managed cluster from the ACM console. These VM applications can be subscription based, ApplicationSet-based, or discovered, as described previously. Options to start, stop, pause, and delete VM operations are available from the ACM console for these types of applications.

For more information, see Red Hat Advanced Cluster Management for Virtualization in the Red Hat documentation.