Implementing a Red Hat Enterprise Linux High Availability Add-On cluster in a multizone region environment

Adding cluster node entries to the hosts file

On both nodes, add the IP addresses and hostnames of both nodes to the /etc/hosts file.

For more information, see Setting up /etc/hosts files on RHEL cluster nodes.

Preparing environment variables

To simplify the setup process, prepare some environment variables for the root user. These environment variables are used with later operating system commands in this information.

On both nodes, set the following environment variables.

# General settings
export CLUSTERNAME="SAP_CLUSTER"         # Cluster name

export APIKEY=<APIKEY>                   # API Key of the IBM Cloud IAM ServiceID
export CLOUD_REGION=<CLOUD_REGION>       # Multizone region name
export PROXY_IP=<IP_ADDRESS>             # IP address of proxy server (optional)

# Workspace 1
export IBMCLOUD_CRN_1=<IBMCLOUD_CRN_1>   # Workspace CRN  (required by fence_ibm_powervs, powervs-subnet)
export GUID_1=<GUID_1>                   # Workspace GUID (required by fence_ibm_powervs, powervs-subnet)

# Workspace 2
export IBMCLOUD_CRN_2=<IBMCLOUD_CRN_2>   # Workspace CRN  (required by fence_ibm_powervs, powervs-subnet)
export GUID_2=<GUID_2>                   # Workspace GUID (required by fence_ibm_powervs, powervs-subnet)

# Virtual server instance 1
export NODE1=<HOSTNAME_1>                # Virtual server instance hostname
export POWERVSI_1=<POWERVSI_1>           # Virtual server instance id (required by fence_ibm_powervs, powervs-subnet)

# Virtual server instance 2
export NODE2=<HOSTNAME_2>                # Virtual server instance hostname
export POWERVSI_2=<POWERVSI_2>           # Virtual server instance id (required by fence_ibm_powervs, powervs-subnet)

To find the settings for the APIKEY, IBMCLOUD_CRN_?, GUID_?, and POWERVSI_? variables, follow the steps in Collecting parameters for configuring a high availability cluster.

Installing RHEL HA Add-On software

Install the required software packages. The minimum operating system version required to use the powervs-subnet resource agent is RHEL 9.2.

The @server group must be installed on the operating system. This installation is a standard requirement for SAP applications.

dnf install -y pcs pacemaker fence-agents-ibm-powervs

Configuring a RHEL HA Add-On cluster

Use the following steps to configure a RHEL HA Add-On cluster.

firewall-cmd --permanent --add-service=high-availability

firewall-cmd --reload

systemctl enable --now pcsd.service

systemctl status pcsd.service

passwd hacluster

pcs host auth ${NODE1} ${NODE2} -u hacluster

pcs cluster setup ${CLUSTERNAME} --start ${NODE1} ${NODE2}

pcs status

fence_ibm_powervs \
    --token=${APIKEY} \
    --crn=${IBMCLOUD_CRN_1} \
    --instance=${GUID_1} \
    --region=${CLOUD_REGION} \
    --api-type=public \
    -o list

fence_ibm_powervs \
    --token=${APIKEY} \
    --crn=${IBMCLOUD_CRN_2} \
    --instance=${GUID_2} \
    --region=${CLOUD_REGION} \
    --api-type=public \
    -o list

fence_ibm_powervs \
    --token=${APIKEY} \
    --crn=${IBMCLOUD_CRN_1} \
    --instance=${GUID_1} \
    --region=${CLOUD_REGION} \
    --api-type=private \
    --proxy=http://${PROXY_IP}:3128 \
    -o list

time fence_ibm_powervs \
    --token=${APIKEY} \
    --crn=${IBMCLOUD_CRN_1} \
    --instance=${GUID_1} \
    --region=${CLOUD_REGION} \
    --plug=${POWERVSI_1} \
    --api-type=private \
    --proxy=http://${PROXY_IP}:3128 \
    -o status

time fence_ibm_powervs \
    --token=${APIKEY} \
    --crn=${IBMCLOUD_CRN_2} \
    --instance=${GUID_2} \
    --region=${CLOUD_REGION} \
    --plug=${POWERVSI_2} \
    --api-type=private \
    --proxy=http://${PROXY_IP}:3128 \
    -o status

pcs stonith describe fence_ibm_powervs

pcs stonith create fence_node1 fence_ibm_powervs \
    token=${APIKEY} \
    crn=${IBMCLOUD_CRN_1} \
    instance=${GUID_1} \
    region=${CLOUD_REGION} \
    api_type=private \
    proxy=http://${PROXY_IP}:3128 \
    pcmk_host_map="${NODE1}:${POWERVSI_1}" \
    pcmk_reboot_timeout=600 \
    pcmk_monitor_timeout=600 \
    pcmk_status_timeout=60

pcs stonith create fence_node2 fence_ibm_powervs \
    token=${APIKEY} \
    crn=${IBMCLOUD_CRN_2} \
    instance=${GUID_2} \
    region=${CLOUD_REGION} \
    api_type=private \
    proxy=http://${PROXY_IP}:3128 \
    pcmk_host_map="${NODE2}:${POWERVSI_2}" \
    pcmk_reboot_timeout=600 \
    pcmk_monitor_timeout=600 \
    pcmk_status_timeout=60

pcs config

pcs status

pcs stonith config

pcs stonith status

pcs property set stonith-action=off

pcs config

pcs stonith fence ${NODE2}

pcs status

pcs cluster start

pcs status

pcs stonith status

pcs stonith fence ${NODE1}

pcs cluster start

pcs status

pcs stonith status

Disabling the automatic startup of cluster services when the server boots

After a virtual server instance restarts, it takes some time for its STATUS to become ACTIVE and its Health Status to become OK. The powervs-subnet resource agent requires these states to function properly. Therefore, you must disable automatic cluster startup and start the cluster manually after the instance reaches the required states.

On any node, disable the automatic startup of cluster services at boot time.

pcs cluster disable --all

When you restart an instance, check the instance status in the IBM Cloud Console and wait until the Status field shows Active with a green checkmark. Then, use the following command to manually start the cluster.

pcs cluster start

Installing the IBM Cloud resource agents

The resource agents are provided by the resource-agents-cloud package. The package requires RHEL 9.4 or later and became available with a recent update to the RHEL 9.4 minor release level.

Run the following command on each cluster node to install the resource agents.

dnf install -y resource-agents-cloud

Preparing the powervs-subnet resource agent

The powervs-subnet resource agent is included in the resource-agents-cloud package in RHEL 9.4 or later and does not require a separate download or manual installation. However, specific NetworkManager settings are required for the resource agent to operate correctly.

1. Verify that the NetworkManager-config-server package is installed.

   On both nodes, run the following command.

   dnf list NetworkManager-config-server
   

   Sample output:

   # dnf list NetworkManager-config-server
   Installed Packages
   NetworkManager-config-server.noarch                                    1:1.42.2-16.el9_2                                     @rhel-9-for-ppc64le-baseos-e4s-rpms
   

2. Make sure that the NetworkManager no-auto-default configuration variable is set to *.

   NetworkManager --print-config | grep "no-auto-default="
   

   Sample output:

   # NetworkManager --print-config | grep "no-auto-default="
   no-auto-default=*
   

   If the no-auto-default shows a value other than *, edit the /etc/NetworkManager/conf.d/00-server.conf file and change the variable as needed.

After you complete these steps, verify that the powervs-subnet resource agent is available.

pcs resource describe powervs-subnet

Continue with the steps in Creating a service ID for the resource agent

Verifying the powervs-move-ip resource agent

The powervs-move-ip resource agent is included in the resource-agents-cloud package in RHEL 9.4 or later and does not require a separate download or manual installation. After you install the resource agents package, verify that the powervs‑move‑ip resource agent is available and registered correctly.

Use the following command on either node to display a brief description of the resource agent.

pcs resource describe powervs-move-ip

Creating the static route in the workspace for the powervs-move-ip resource agent

1. Determine the next hop IP addresses of the virtual server instances of the cluster. Follow these steps:

   1. Open the Power Virtual Server user interface in IBM Cloud.
   2. Click workspaces in the left navigation menu.
   3. Select the workspace where the cluster node is provisioned. The "Workspace details" panel is displayed.
   4. Click View virtual servers. The list of virtual server instances is displayed.
   5. Identify your virtual server instance and its IP address. Note the IP address. You need to enter that IP address as Next Hop in the route.

2. For each virtual IP address that you configure as a cluster resource, create a static route in both Power Virtual Server workspaces. Follow these steps:

   1. Open the Power Virtual Server user interface in IBM Cloud.
   2. Click Workspaces in the left navigation menu.
   3. Select the workspace in which you want to create the static route. The "Workspace details" panel is displayed.
   4. Click View virtual servers.
   5. In the navigation pane, click Networking > Routes. The Static routes page lists the existing static routes (if any).
   6. Click Create static route to create a new route.
   7. In the "Create static route" panel

      1. Enter a name for the static route in the Name field.

      2. Optionally, enter user tags in the User tags (optional) field.

      3. In the Destination field, enter a valid IP address. The destination IP address must not belong to any of the CIDR blocks of the subnets in the scenario.

      4. In the Next hop field, enter a valid IP address. The next hop IP address must:

         • Be within the CIDR range of a subnet in the workspace.
         • Match the primary IP address that is assigned to a network adapter of the cluster node virtual server instance.

      5. Advertise and Status: Set both switches to Enabled (default).

         The Advertise switch controls whether the static route is propagated outside of the workspace to the Power Edge Router (PER). If Advertise is disabled, the route remains internal and is not visible to external network connections.

         The Status switch determines whether the static route is active within the network fabric. If Status is set to Disabled, the route is not used, even if Advertise is enabled.

      6. Click Create route.

   Repeat the step for both cluster nodes. Note the cloud resource name CRN for each of the routes. You need to enter the CRNs during the cluster resource configuration steps for the specific high availability scenario.

Creating a custom role, service ID, and API key for the resource agent

Follow the steps in Configuring IAM for Cluster Resource and Fencing Agents on Power Virtual Server to authenticate and authorize the agents in Identity and Access Management (IAM).