Implementing a RHEL HA Add-On cluster on IBM Power Virtual Server in a Multizone Region Environment

Checking the RHEL HA repository

Check that the appropriate repository is enabled on both nodes by using the following command.

dnf repolist

Use the following commands to enable the HA repository if it is missing.

subscription-manager repos \
    --enable="rhel-9-for-ppc64le-highavailability-e4s-rpms"

dnf clean all

dnf repolist

Installing the RHEL HA Add-On software packages

Install the required software packages on both nodes by running the following command.

dnf install -y pcs pacemaker fence-agents-ibm-powervs

Make sure that you install the minimal version of the fence-agents-ibm-powervs package dependent on your Red Hat Enterprise Linux release:

RHEL 9

fence-agents-ibm-powervs-4.10.0-43.el9

Configuring firewall services

Add the high availability service to the RHEL firewall if firewalld.service is installed and enabled.

On both nodes, run the following commands.

firewall-cmd --permanent --add-service=high-availability

firewall-cmd --reload

Starting the PCS daemon

Start the PCS daemon that is used for controlling and configuring RHEL HA Add-On clusters through PCS.

On both nodes, run the following commands.

systemctl enable --now pcsd.service

Make sure that the PCS service is running.

systemctl status pcsd.service

Setting a password for the hacluster user ID

Set the password for the hacluster user ID.

On both nodes, run the following command.

passwd hacluster

Authenticating the cluster nodes

Use the following command to authenticate the user hacluster to the PCS daemon on the nodes in the cluster. The command prompts you for the password that you set in the previous step.

On NODE1, run the following command.

pcs host auth ${NODE1} ${NODE2} -u hacluster

Configuring and starting the cluster nodes

Configure the cluster configuration file and synchronize the configuration to the specified nodes.

The --start option also starts the cluster service on the nodes.

On NODE1, run the following command.

pcs cluster setup ${CLUSTERNAME} --start ${NODE1} ${NODE2}

pcs status

Creating the fencing device

STONITH is an acronym for "Shoot The Other Node In The Head" and protects your data from corruption in a split-brain situation.

You must enable STONITH (fencing) for a RHEL HA Add-On production cluster.

Fence agent fence_ibm_powervs is the only supported agent for a STONITH device on Power Virtual Server clusters.

You must configure a fencing device for each of the two workspaces in the multizone region. The fence agent connects to the Power Cloud API by using the common APIKEY and CLOUD_REGION parameters. The parameters IBMCLOUD_CRN_<n>, GUID_<n>, and the instance ID POWERVSI_<n> are specific to the workspace. You can test the agent invocation by using the parameters that you gathered in the Gathering required parameters for the cluster configuration section.

Identifying the virtual server instances for fencing

Use the list option of fence_ibm_powervs to identify and or verify the instance IDs of the two cluster nodes.

On any node, run the following commands.

fence_ibm_powervs \
    --token=${APIKEY} \
    --crn=${IBMCLOUD_CRN_1} \
    --instance=${GUID_1} \
    --region=${CLOUD_REGION} \
    --api-type=public \
    -o list

fence_ibm_powervs \
    --token=${APIKEY} \
    --crn=${IBMCLOUD_CRN_2} \
    --instance=${GUID_2} \
    --region=${CLOUD_REGION} \
    --api-type=public \
    -o list

If the virtual server instances have access to only a private network, you must use the --api-type=private option, which also requires an extra --proxy option.

Example:

fence_ibm_powervs \
    --token=${APIKEY} \
    --crn=${IBMCLOUD_CRN_1} \
    --instance=${GUID_1} \
    --region=${CLOUD_REGION} \
    --api-type=private \
    --proxy=http://${PROXY_IP}:3128 \
    -o list

The following examples use the --api-type=private option.

Checking the status of both virtual server instances

On both nodes, run the following commands.

time fence_ibm_powervs \
    --token=${APIKEY} \
    --crn=${IBMCLOUD_CRN_1} \
    --instance=${GUID_1} \
    --region=${CLOUD_REGION} \
    --plug=${POWERVSI_1} \
    --api-type=private \
    --proxy=http://${PROXY_IP}:3128 \
    -o status

time fence_ibm_powervs \
    --token=${APIKEY} \
    --crn=${IBMCLOUD_CRN_2} \
    --instance=${GUID_2} \
    --region=${CLOUD_REGION} \
    --plug=${POWERVSI_2} \
    --api-type=private \
    --proxy=http://${PROXY_IP}:3128 \
    -o status

The status action of the fence agent against a virtual server instance --plug=<POWERVSI_n> displays its power status.

On both nodes, the two commands must report Status: ON.

The output of the time command might be useful later when you choose timeouts for the STONITH device.

You can add the -v flag for verbose output, which shows more information about connecting to the Power Cloud API and querying virtual server power status.

Creating the stonith devices

The following command shows the device-specific options for the fence_ibm_powervs fencing agent.

pcs stonith describe fence_ibm_powervs

Create the stonith device for both virtual server instances.

On NODE1, run the following commands.

pcs stonith create fence_node1 fence_ibm_powervs \
    token=${APIKEY} \
    crn=${IBMCLOUD_CRN_1} \
    instance=${GUID_1} \
    region=${CLOUD_REGION} \
    api_type=private \
    proxy=http://${PROXY_IP}:3128 \
    pcmk_host_map="${NODE1}:${POWERVSI_1}" \
    pcmk_reboot_timeout=600 \
    pcmk_monitor_timeout=600 \
    pcmk_status_timeout=60

pcs stonith create fence_node2 fence_ibm_powervs \
    token=${APIKEY} \
    crn=${IBMCLOUD_CRN_2} \
    instance=${GUID_2} \
    region=${CLOUD_REGION} \
    api_type=private \
    proxy=http://${PROXY_IP}:3128 \
    pcmk_host_map="${NODE2}:${POWERVSI_2}" \
    pcmk_reboot_timeout=600 \
    pcmk_monitor_timeout=600 \
    pcmk_status_timeout=60

Although the fence_ibm_powervs agent uses api-type as an option when started from the command line, the stonith resource needs to be created by using api_type.

Verify the configuration with the following commands.

pcs config

pcs status

pcs stonith config

pcs stonith status

Setting the stonith-action cluster property

For the powervs-subnet resource agent to work, you must set the stonith-action cluster property to off. When the cluster performs a fencing action, it triggers a power-off operation instead of a reboot for the fenced instance.

After this change, you always need to log in to the IBM Cloud Console, and manually start an instance that was fenced by the cluster.

pcs property set stonith-action=off

Verify the change.

pcs config

Testing fencing operations

To test the STONITH configuration, manually fence the nodes.

When fencing is manually triggered by pcs stonith fence, the stonith-action cluster attribute is not used and the node restarts.

On NODE1, run the following commands.

pcs stonith fence ${NODE2}

pcs status

As a result, NODE2 restarts.

After NODE2 is running, start the cluster on NODE2 and try to fence NODE1.

On NODE2, run the following commands.

pcs cluster start

pcs status

pcs stonith status

pcs stonith fence ${NODE1}

NODE1 restarts.

After the node is running, start the cluster on NODE1 again.

On NODE1, run the following command.

pcs cluster start

pcs status

pcs stonith status