--- name: sap-powervs-ha-rhel-mz title: Implementing a Red Hat Enterprise Linux High Availability Add-On cluster in a multizone region environment description: The information describes how to transform the individual virtual server instances into a cluster. last-updated: 2026-07-22 --- > ## Documentation Index > The table of contents for this documentation set is at https://cloud.ibm.com/docs/sap?format=markdown > The index for all IBM Cloud docs is at: https://cloud.ibm.com/docs/llms.txt > Use these files to discover more information as needed. # Implementing a Red Hat Enterprise Linux High Availability Add-On cluster in a multizone region environment {: #ha-rhel-mz} Use the following information and procedures to implement a Red Hat Enterprise Linux (RHEL) High Availability Add-On cluster in a multizone region environment. The cluster uses instances in [IBM® Power® Virtual Server](https://www.ibm.com/products/power-virtual-server){: external} as cluster nodes. The virtual server instances run in different zones in a multizone region. The setup uses either the *powervs-move-ip* or the *powervs-subnet* cluster resource agent to manage the service IP address of an application in a multizone region implementation. The recommended resource agent is *powervs-move-ip*. {: note} The resource agent works only in zones that belong to the same multizone region. Cluster deployments across multiple regions are not supported. See [Multizone regions (MZR)](https://cloud.ibm.com/docs/overview?topic=overview-locations&format=markdown#table-mzr){: external} and [IBM Cloud regions](https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-ibm-cloud-reg&format=markdown){: external} for more information about multizone regions and available locations. The information describes how to transform the individual virtual server instances into a cluster. {: shortdesc} These procedures include installing the high availability packages and agents on each cluster node and configuring the fencing devices. This information is intended for architects and specialists who are planning a high availability deployment of SAP applications on Power Virtual Server. It is not intended to replace existing SAP or Red Hat documentation. {: note} ## Before you begin {: #ha-rhel-mz-begin} Review the general requirements, product documentation, support articles, and SAP notes listed in [Implementing high availability for SAP applications on IBM Power Virtual Server References](https://cloud.ibm.com/docs/sap?topic=sap-ha-rhel-refs&format=markdown). ## Creating virtual server instances for the cluster {: #ha-rhel-mz-create-virtual-server-instances} Use the instructions in [Deploying an Power Virtual Server instance for the SAP HANA system](https://cloud.ibm.com/docs/sap?topic=sap-powervs-set-up-power-instances&format=markdown#powervs-set-up-power-hana-instance) to create virtual servers instances for SAP HANA cluster nodes. For SAP NetWeaver cluster nodes, use the instructions on [Deploying an Power Virtual Server instance for SAP NetWeaver](https://cloud.ibm.com/docs/sap?topic=sap-powervs-set-up-power-instances&format=markdown#powervs-set-up-power-netweaver-instance). Create two workspaces in two zones of a multizone region. Create a [Transit Gateway](https://cloud.ibm.com/docs/transit-gateway?format=markdown) and add both workspaces to the connections. Create two virtual server instances, one in each workspace. {: overview} ## Preparing the nodes for RHEL HA Add-On installation {: #ha-rhel-mz-prepare-nodes-for-rhel-ha-installation} The following section describes basic preparation steps on the cluster nodes. Make sure that you follow the steps on both nodes. Log in as the root user to each of the cluster nodes. ### Adding cluster node entries to the hosts file {: #ha-rhel-mz-update-hosts-file} On both nodes, add the IP addresses and hostnames of both nodes to the `/etc/hosts` file. For more information, see [Setting up `/etc/hosts` files on RHEL cluster nodes](https://access.redhat.com/solutions/81123){: external}. ### Preparing environment variables {: #ha-rhel-mz-prepare-environment-variables} To simplify the setup process, prepare some environment variables for the root user. These environment variables are used with later operating system commands in this information. On both nodes, set the following environment variables. ```sh # General settings export CLUSTERNAME="SAP_CLUSTER" # Cluster name export APIKEY= # API Key of the IBM Cloud IAM ServiceID export CLOUD_REGION= # Multizone region name export PROXY_IP= # IP address of proxy server (optional) # Workspace 1 export IBMCLOUD_CRN_1= # Workspace CRN (required by fence_ibm_powervs, powervs-subnet) export GUID_1= # Workspace GUID (required by fence_ibm_powervs, powervs-subnet) # Workspace 2 export IBMCLOUD_CRN_2= # Workspace CRN (required by fence_ibm_powervs, powervs-subnet) export GUID_2= # Workspace GUID (required by fence_ibm_powervs, powervs-subnet) # Virtual server instance 1 export NODE1= # Virtual server instance hostname export POWERVSI_1= # Virtual server instance id (required by fence_ibm_powervs, powervs-subnet) # Virtual server instance 2 export NODE2= # Virtual server instance hostname export POWERVSI_2= # Virtual server instance id (required by fence_ibm_powervs, powervs-subnet) ``` {: codeblock} To find the settings for the `APIKEY`, `IBMCLOUD_CRN_?`, `GUID_?`, and `POWERVSI_?` variables, follow the steps in [Collecting parameters for configuring a high availability cluster](https://cloud.ibm.com/docs/sap?topic=sap-powervs-ha-rhel-vsi&format=markdown#ha-rhel-collect-parameters-for-cluster-config). ## Installing and configuring a RHEL HA Add-On cluster {: #ha-rhel-mz-install-and-configure-rhel-ha-cluster} Use the following steps to set up a two-node cluster for an IBM Power Virtual Server. The instructions are based on the Red Hat product documentation and articles that are listed in [Implementing high availability for SAP applications on IBM Power Virtual Server References](https://cloud.ibm.com/docs/sap?topic=sap-ha-rhel-refs&format=markdown). You need to complete some steps on both nodes and some steps on either NODE1 or on NODE2. {: tip} ### Installing RHEL HA Add-On software {: #ha-rhel-mz-install-rhel-ha-software} Install the required software packages. The minimum operating system version required to use the *powervs-subnet* resource agent is RHEL 9.2. The *@server* group must be installed on the operating system. This installation is a standard requirement for SAP applications. {: attention} #### Checking the RHEL HA repository {: #ha-rhel-mz-check-rhel-ha-repository} See [Checking the RHEL HA repository](https://cloud.ibm.com/docs/sap?topic=sap-ha-rhel&format=markdown#ha-rhel-check-rhel-ha-repository) for the steps to enable the RHEL HA repository. #### Installing the RHEL HA Add-On software packages {: #ha-rhel-mz-install-ha-software-packages} Install the required software packages on both nodes by running the following command. ```sh dnf install -y pcs pacemaker fence-agents-ibm-powervs ``` {: pre} Make sure that you install the minimal version of the *fence-agents-ibm-powervs* package dependent on your Red Hat Enterprise Linux release: RHEL 9 : fence-agents-ibm-powervs-4.10.0-43.el9 ### Configuring a RHEL HA Add-On cluster {: #ha-rhel-mz-configure-rhel-ha-cluster} Use the following steps to configure a RHEL HA Add-On cluster. #### Configuring firewall services {: #ha-rhel-mz-configure-firewall-services} Add the high availability service to the RHEL firewall if *firewalld.service* is installed and enabled. On both nodes, run the following commands. ```sh firewall-cmd --permanent --add-service=high-availability ``` {: pre} ```sh firewall-cmd --reload ``` {: pre} #### Starting the PCS daemon {: #ha-rhel-mz-start-pcsd-daemon} Start the PCS daemon that is used for controlling and configuring RHEL HA Add-On clusters through PCS. On both nodes, run the following commands. ```sh systemctl enable --now pcsd.service ``` {: pre} Make sure that the PCS service is running. ```sh systemctl status pcsd.service ``` {: pre} #### Setting a password for the hacluster user ID {: #ha-rhel-mz-set-hacluster-password} Set the password for the hacluster user ID. On both nodes, run the following command. ```sh passwd hacluster ``` {: pre} #### Authenticating the cluster nodes {: #ha-rhel-mz-authenticate-cluster-nodes} Use the following command to authenticate the user hacluster to the PCS daemon on the nodes in the cluster. When you run the command, you are prompted to enter the password that you set in the previous step. On NODE1, run the following command. ```sh pcs host auth ${NODE1} ${NODE2} -u hacluster ``` {: pre} #### Configuring and starting the cluster nodes {: #ha-rhel-mz-configure-and-start-cluster-nodes} Configure the cluster configuration file and synchronize the configuration to the specified nodes. The `--start` option also starts the cluster service on the nodes. On NODE1, run the following command. ```sh pcs cluster setup ${CLUSTERNAME} --start ${NODE1} ${NODE2} ``` {: pre} ```sh pcs status ``` {: pre} #### Creating the fencing device {: #ha-rhel-mz-create-fencing-device} STONITH is an acronym for "Shoot The Other Node In The Head" and protects your data from corruption in a split-brain situation. You must enable STONITH (fencing) for a RHEL HA Add-On production cluster. {: important} This document explains how to configure the *fence_ibm_powervs* fence agent in Red Hat Enterprise Linux High Availability Add-On clusters. As an alternative to *fence_ibm_powervs*, you can implement STONITH Block Device (SBD) fencing by following the instructions in [Implementing SBD poison-pill fencing in a Red Hat Enterprise Linux High Availability Add-On cluster](https://cloud.ibm.com/docs/sap?topic=sap-ha-sbd&format=markdown){: external}. {: note} You must configure a fencing device for each of the two workspaces in the multizone region. The fence agent connects to the [Power Cloud API](https://cloud.ibm.com/apidocs/power-cloud){: external} by using the common `APIKEY` and `CLOUD_REGION` parameters. The parameters `IBMCLOUD_CRN_`, `GUID_`, and the instance ID `POWERVSI_` are specific to the workspace. You can test the agent invocation by using the parameters that you gathered in the [Collecting parameters for configuring a high availability cluster](https://cloud.ibm.com/docs/sap?topic=sap-powervs-ha-rhel-vsi&format=markdown#ha-rhel-collect-parameters-for-cluster-config) section. #### Identifying the virtual server instances for fencing {: #ha-rhel-mz-identify-virtual-instances-for-fencing} Use the *list* option of *fence_ibm_powervs* to identify and or verify the instance IDs of the two cluster nodes. On any node, run the following commands. ```sh fence_ibm_powervs \ --token=${APIKEY} \ --crn=${IBMCLOUD_CRN_1} \ --instance=${GUID_1} \ --region=${CLOUD_REGION} \ --api-type=public \ -o list ``` {: pre} ```sh fence_ibm_powervs \ --token=${APIKEY} \ --crn=${IBMCLOUD_CRN_2} \ --instance=${GUID_2} \ --region=${CLOUD_REGION} \ --api-type=public \ -o list ``` {: pre} If the virtual server instances have access to only a private network, you must use the `--api-type=private` option, which also requires an extra `--proxy` option. Example: ```sh fence_ibm_powervs \ --token=${APIKEY} \ --crn=${IBMCLOUD_CRN_1} \ --instance=${GUID_1} \ --region=${CLOUD_REGION} \ --api-type=private \ --proxy=http://${PROXY_IP}:3128 \ -o list ``` {: pre} The following examples use the `--api-type=private` option. #### Checking the status of both virtual server instances {: #ha-rhel-mz-check-virtual-instances-status} On both nodes, run the following commands. ```sh time fence_ibm_powervs \ --token=${APIKEY} \ --crn=${IBMCLOUD_CRN_1} \ --instance=${GUID_1} \ --region=${CLOUD_REGION} \ --plug=${POWERVSI_1} \ --api-type=private \ --proxy=http://${PROXY_IP}:3128 \ -o status ``` {: pre} ```sh time fence_ibm_powervs \ --token=${APIKEY} \ --crn=${IBMCLOUD_CRN_2} \ --instance=${GUID_2} \ --region=${CLOUD_REGION} \ --plug=${POWERVSI_2} \ --api-type=private \ --proxy=http://${PROXY_IP}:3128 \ -o status ``` {: pre} The `status` action of the fence agent against a virtual server instance `--plug=` displays its power status. On both nodes, the two commands must report `Status: ON`. The output of the `time` command might be useful later when you choose timeouts for the STONITH device. You can add the `-v` flag for verbose output, which shows more information about connecting to the Power Cloud API and querying virtual server power status. #### Creating the stonith devices {: #ha-rhel-mz-create-stonith-device} The following command shows the device-specific options for the *fence_ibm_powervs* fencing agent. ```sh pcs stonith describe fence_ibm_powervs ``` {: pre} Create the stonith device for both virtual server instances. On NODE1, run the following commands. ```sh pcs stonith create fence_node1 fence_ibm_powervs \ token=${APIKEY} \ crn=${IBMCLOUD_CRN_1} \ instance=${GUID_1} \ region=${CLOUD_REGION} \ api_type=private \ proxy=http://${PROXY_IP}:3128 \ pcmk_host_map="${NODE1}:${POWERVSI_1}" \ pcmk_reboot_timeout=600 \ pcmk_monitor_timeout=600 \ pcmk_status_timeout=60 ``` {: pre} ```sh pcs stonith create fence_node2 fence_ibm_powervs \ token=${APIKEY} \ crn=${IBMCLOUD_CRN_2} \ instance=${GUID_2} \ region=${CLOUD_REGION} \ api_type=private \ proxy=http://${PROXY_IP}:3128 \ pcmk_host_map="${NODE2}:${POWERVSI_2}" \ pcmk_reboot_timeout=600 \ pcmk_monitor_timeout=600 \ pcmk_status_timeout=60 ``` {: pre} Although the `fence_ibm_powervs` agent uses the `api`-type option when it is started interactively, the STONITH resource must be created by using `api_type`. {: important} Verify the configuration with the following commands. ```sh pcs config ``` {: pre} ```sh pcs status ``` {: pre} ```sh pcs stonith config ``` {: pre} ```sh pcs stonith status ``` {: pre} #### Setting the stonith-action cluster property {: #ha-rhel-mz-set-stonith-action-cluster-property} For the *powervs-subnet* resource agent to work, you must set the *stonith-action* cluster property to *off*. When the cluster initiates a fencing action, it triggers an *off* operation instead of a *reboot* for the fenced instance. {: attention} After this change, you always need to log in to the IBM Cloud Console, and manually start an instance that was fenced by the cluster. ```sh pcs property set stonith-action=off ``` {: pre} Verify the change. ```sh pcs config ``` {: pre} #### Testing fencing operations {: #ha-rhel-mz-test-fencing-operations} To test the STONITH configuration, manually fence the nodes. On NODE1, run the following commands. ```sh pcs stonith fence ${NODE2} ``` {: pre} ```sh pcs status ``` {: pre} As a result, NODE2 stops. Activate NODE2, then start the cluster on the node and try to fence NODE1. On NODE2, run the following commands. ```sh pcs cluster start ``` {: pre} ```sh pcs status ``` {: pre} ```sh pcs stonith status ``` {: pre} ```sh pcs stonith fence ${NODE1} ``` {: pre} NODE1 stops. Activate NODE2, then start the cluster on the node. On NODE1, run the following command. ```sh pcs cluster start ``` {: pre} ```sh pcs status ``` {: pre} ```sh pcs stonith status ``` {: pre} ### Disabling the automatic startup of cluster services when the server boots {: #ha-rhel-mz-disable-cl-auto-start} After a virtual server instance restarts, it takes some time for its *STATUS* to become *ACTIVE* and its *Health Status* to become *OK*. The *powervs-subnet* resource agent requires these states to function properly. Therefore, you must disable automatic cluster startup and start the cluster manually after the instance reaches the required states. On any node, disable the automatic startup of cluster services at boot time. ```sh pcs cluster disable --all ``` {: pre} When you restart an instance, check the instance status in the IBM Cloud Console and wait until the *Status* field shows *Active* with a green checkmark. Then, use the following command to manually start the cluster. ```sh pcs cluster start ``` {: pre} ## Preparing a multizone RHEL HA Add-On cluster for a virtual IP address resource {: #ha-rhel-mz-create-vip} Use the following steps to prepare a multizone RHEL HA Add-on cluster for a virtual IP address resource. Two specific resource agents are available to manage a service IP address in a multizone region environment: - `powervs-move-ip` resource agent During a takeover event, the resource agent `powervs-move-ip` updates predefined static routes in the IBM Power Virtual Server, and configures an overlay IP address as IP alias address on the virtual server instance. - `powervs-subnet` resource agent During a takeover event, the resource agent `powervs-subnet` moves the entire subnet, including the IP address, from one workspace to another. ### Installing the IBM Cloud resource agents {: #ha-rhel-mz-install-vip-resource-agents} The resource agents are provided by the `resource-agents-cloud` package. The package requires RHEL 9.4 or later and became available with a recent update to the RHEL 9.4 minor release level. Run the following command on each cluster node to install the resource agents. ```sh dnf install -y resource-agents-cloud ``` {: pre} The following steps describe the preparation for the `powervs-move-ip` resource agent. If you use the `powervs-subnet` resource agent, continue with the steps in [Preparing the powervs-subnet resource agent](#ha-rhel-mz-prepare-powervs-subnet-resource-agent) ### Verifying the powervs-move-ip resource agent {: #ha-rhel-mz-verify-powervs-move-ip-resource-agent} The *powervs-move-ip* resource agent is included in the `resource-agents-cloud` package in RHEL 9.4 or later and does not require a separate download or manual installation. After you install the resource agents package, verify that the powervs‑move‑ip resource agent is available and registered correctly. Use the following command on either node to display a brief description of the resource agent. ```sh pcs resource describe powervs-move-ip ``` {: pre} ### Creating the static route in the workspace for the powervs-move-ip resource agent {: #ha-rhel-mz-install-powervs-move-ip-create-route} 1. Determine the next hop IP addresses of the virtual server instances of the cluster. Follow these steps: 1. Open the Power Virtual Server user interface in [IBM Cloud](https://cloud.ibm.com/power/overview){: external}. 1. Click **workspaces** in the left navigation menu. 1. Select the workspace where the cluster node is provisioned. The "Workspace details" panel is displayed. 1. Click **View virtual servers**. The list of virtual server instances is displayed. 1. Identify your virtual server instance and its IP address. Note the IP address. You need to enter that IP address as **Next Hop** in the route. 1. For each virtual IP address that you configure as a cluster resource, create a static route in both Power Virtual Server workspaces. Follow these steps: 1. Open the Power Virtual Server user interface in [IBM Cloud](https://cloud.ibm.com/power/overview){: external}. 1. Click **Workspaces** in the left navigation menu. 1. Select the workspace in which you want to create the static route. The "Workspace details" panel is displayed. 1. Click **View virtual servers**. 1. In the navigation pane, click **Networking** > **Routes**. The **Static routes** page lists the existing static routes (if any). 1. Click **Create static route** to create a new route. 1. In the "Create static route" panel 1. Enter a name for the static route in the **Name** field. 1. Optionally, enter user tags in the **User tags (optional)** field. 1. In the **Destination** field, enter a valid IP address. The destination IP address must not belong to any of the CIDR blocks of the subnets in the scenario. 1. In the **Next hop** field, enter a valid IP address. The next hop IP address must: - Be within the CIDR range of a subnet in the workspace. - Match the primary IP address that is assigned to a network adapter of the cluster node virtual server instance. 1. **Advertise** and **Status**: Set both switches to **Enabled** (default). The **Advertise** switch controls whether the static route is propagated outside of the workspace to the Power Edge Router (PER). If **Advertise** is disabled, the route remains internal and is not visible to external network connections. The **Status** switch determines whether the static route is active within the network fabric. If **Status** is set to **Disabled**, the route is not used, even if **Advertise** is enabled. 1. Click **Create route**. Repeat the step for both cluster nodes. Note the cloud resource name *CRN* for each of the routes. You need to enter the CRNs during the cluster resource configuration steps for the specific high availability scenario. Continue with the steps in [Creating a service ID for the resource agent](#ha-rhel-mz-iam-custom-role) ### Preparing the powervs-subnet resource agent {: #ha-rhel-mz-prepare-powervs-subnet-resource-agent} The *powervs-subnet* resource agent is included in the `resource-agents-cloud` package in RHEL 9.4 or later and does not require a separate download or manual installation. However, specific NetworkManager settings are required for the resource agent to operate correctly. 1. Verify that the `NetworkManager-config-server` package is installed. On both nodes, run the following command. ```sh dnf list NetworkManager-config-server ``` {: pre} Sample output: ```sh # dnf list NetworkManager-config-server Installed Packages NetworkManager-config-server.noarch 1:1.42.2-16.el9_2 @rhel-9-for-ppc64le-baseos-e4s-rpms ``` {: screen} 1. Make sure that the NetworkManager `no-auto-default` configuration variable is set to `*`. ```sh NetworkManager --print-config | grep "no-auto-default=" ``` {: pre} Sample output: ```sh # NetworkManager --print-config | grep "no-auto-default=" no-auto-default=* ``` {: screen} If the `no-auto-default` shows a value other than `*`, edit the `/etc/NetworkManager/conf.d/00-server.conf` file and change the variable as needed. After you complete these steps, verify that the powervs-subnet resource agent is available. ```sh pcs resource describe powervs-subnet ``` {: pre} ### Creating a custom role, service ID, and API key for the resource agent {: #ha-rhel-mz-iam-custom-role} Follow the steps in [Configuring IAM for Cluster Resource and Fencing Agents on Power Virtual Server](https://cloud.ibm.com/docs/sap?topic=sap-powervs-ha-rhel-vsi&format=markdown#ha-vsi-configure-iam) to authenticate and authorize the agents in Identity and Access Management (IAM). ## Conclusion {: #ha-rhel-mz-conclusion} The basic cluster implementation and the necessary preparations are now complete. The IP address cluster resource is created during the configuration of the specific high availability scenario by using either the `powervs-move-ip` or the `powervs-subnet` resource agent.