Configuring an Active-Passive NFS Server in a Red Hat High Availability Cluster

The following information describes the configuration of an active-passive NFS server in a Red Hat Enterprise Linux (RHEL) HA Add-On cluster. The cluster uses virtual server instances in IBM® Power® Virtual Server as cluster nodes.

The described setup uses shareable storage volumes that are accessible on both cluster nodes. The file systems for the NFS exports are created on those shareable storage volumes. HA-LVM makes sure that the volume group is active on one node at a time.

In the example setup, one shared volume group nfssharevg contains three logical volumes nfssharelv, sap${SID}lv, and saptranslv. XFS file systems are created on those logical volumes and are mounted on /nfsshare, /nfshare/export/sap${SID}, /nfsshare/export/saptrans.

The instructions are based on the Red Hat product documentation and articles that are listed in Implementing High Availability for SAP Applications on IBM Power Virtual Server References.

Before you begin

Review the general requirements, product documentation, support articles, and SAP notes listed in Implementing High Availability for SAP Applications on IBM Power Virtual Server References.

Prerequisites

A virtual hostname and IP address is required for the NFS server. Make sure that the virtual IP address is defined on the network interface, and reachable in the network.

Name resolution and reverse lookup for physical and virtual IP names and addresses must be unique and consistent on all NFS server and client nodes. Details of the NFS clients (subnet, required NFS export options) must be available. You need to enter them during the cluster setup.

Preparing for a highly available NFS server

Use the following information to prepare the environment for a highly available NFS server.

Installing NFS software packages

On both nodes, run the following commands.

dnf install -y nfs-utils

Preparing LVM objects

All cluster nodes need access to the shared storage volumes, but only one node has exclusive read and write access to a volume.

Preparing active-passive HA LVM

On both nodes, edit file /etc/lvm/lvm.conf to include the system ID in the volume group. Search for the configuration setting system_id_source and change its value to "uname".

Sample setting of system_id_source in /etc/lvm/lvm.conf:

# grep "system_id_source ="  /etc/lvm/lvm.conf
system_id_source = "uname"

Verify that the LVM system ID for the node matches the uname -n output.

lvm systemid

uname -n

Sample output:

# lvm systemid
  system ID: cl-nfs-1
# uname -n
  cl-nfs-1

Identifying the World Wide Names of shared storage volumes

Identify the World Wide Names (WWN) for all volumes that are used in the shared volume group.

Log in to IBM Cloud® and go to the Storage volumes view of Power Virtual Server.
Select your workspace.
Filter for the volume prefix in the Storage volumes list, and identify all the World Wide Names of the volumes in scope (the World Wide Name is a 32-digit hexadecimal number).

Make sure that the attribute Shareable is On for those volumes.
In the Virtual server instances view, go to both virtual server instances of the cluster and verify that in scope volumes are attached to both virtual server instances.

Discovering new SAN volumes on cluster nodes

When you attach a new storage volume to a virtual server instance, you need to rescan the SCSI bus to detect the new volume. Then, update the multipath configuration of the virtual server instance.

On the nodes with new storage volume attachments, run the following command.

rescan-scsi-bus.sh && sleep 10 && multipathd reconfigure

The WWN value of a volume can also be found with the pvs --all command.

Preparing environment variables

To simplify the setup, prepare the following environment variables for user ID root on both nodes. These environment variables are used in subsequent commands in the remainder of this document.

On both nodes, create a file with the environment variables. Then, adapt them to your configuration.

Adapt NFS_VH, NFS_IP, NFS_CLIENTSPEC, and NFS_OPTIONS to your environment. For NFS_PVID, use the WWN that you identified previously. In addition to the file system that is used for the NFS share, the example shows two more file systems that are used for an SAP system landscape with system ID ${SID} and the SAP transport directory. The sample sizes ${NFS_SZ1}, ${NFS_SZ2}, and ${NFS_SZ3} are percentages of the ${NFS_VG} volume group size and need to be modified according to your requirements. The volume group names and mount point names are suggestions and need to be changed to match your own naming conventions.

Make sure that you set the NFS_PVID environment variable by using lowercase letters in the hexadecimal number.

# virtual hostnames
export NFS_VH=<virtual hostname>        # virtual hostname for NFS server
export NFS_IP=<IP address>              # virtual IP address for NFS server

# LVM storage for NFS file systems
export NFS_PVID=3<WWN>                  # WWN of shareable storage volume used for NFS
export NFS_VG="nfssharevg"              # volume group name for NFS exported file systems

# NFS share file system
export NFS_LV1="nfssharelv"             # logical volume name export #1
export NFS_SZ1="5%VG"                   # logical volume size
export NFS_FS1="/nfsshare"              # file system mount point
export NFS_ROOT="${NFS_FS1}/export"     # base export directory

# NFS share file system for SAP system ID <SID>
export SID=<SID>                        # SAP system ID
export NFS_LV2="sap${SID}lv"            # logical volume name export #2
export NFS_SZ2="40%VG"                  # logical volume size
export NFS_FS2="${NFS_ROOT}/sap${SID}"  # file system mount point

# NFS share file system for SAP transport directory
export NFS_LV3="saptranslv"             # logical volume name export #3
export NFS_SZ3="40%VG"                  # logical volume size
export NFS_FS3="${NFS_ROOT}/saptrans"   # file system mount point

# NFS client options
export NFS_CLIENTSPEC="10.111.1.0/24"   # client specs (subnet and netmask) for allowed NFS clients
export NFS_OPTIONS="rw,sync,no_root_squash,no_subtree_check,crossmnt"   # options for NFS export

You must source this file before you use the sample commands in the remainder of this document.

For example, if you created a file that is named nfs_envs.sh, run the following command on both nodes to set the environment variables.

source nfs_envs.sh

Every time that you start a new terminal session, you must run the source command. Alternatively, you can add the environment variables file to the /etc/profile.d directory during the cluster configuration. In this example, the file is sourced automatically each time you log in to the server.

Creating LVM objects

Use the following information to create LVM objects.

Creating physical volumes

On NODE1, run the following command.

pvcreate /dev/mapper/${NFS_PVID}

Sample output:

pvcreate /dev/mapper/${NFS_PVID}
  Physical volume "/dev/mapper/360050768108103357000000000002ddc" successfully created.

Creating a volume group

On NODE1, create the volume group for the NFS export.

vgcreate ${NFS_VG} /dev/mapper/${NFS_PVID}

Verify that the System ID is set.

vgs -o+systemid

Sample output:

# vgs -o+systemid
  VG          #PV #LV #SN Attr   VSize   VFree   System ID
  nfssharevg    1   0   0 wz--n- <50.00g <50.00g cl-sap-1

Creating logical volumes

On NODE1, create three logical volumes for the NFS export.

lvcreate -l ${NFS_SZ1} -n ${NFS_LV1} ${NFS_VG}

lvcreate -l ${NFS_SZ2} -n ${NFS_LV2} ${NFS_VG}

lvcreate -l ${NFS_SZ3} -n ${NFS_LV3} ${NFS_vg}

Creating the file systems

On NODE1, create the file systems for NFS exports.

The example uses file system type xfs. Other file system types are possible. Then, the resource definitions need to be changed.

mkfs.xfs /dev/${NFS_VG}/${NFS_LV1}

mkfs.xfs /dev/${NFS_VG}/${NFS_LV2}

mkfs.xfs /dev/${NFS_VG}/${NFS_LV3}

Creating the mount point for the NFS export

On both nodes, run the following command.

mkdir -p ${NFS_FS1}

Making sure that a volume group is not activated on multiple cluster nodes

Volume groups that are managed by Pacemaker must not activate automatically on startup.

For RHEL 8.5 and later, you can disable autoactivation for a volume group when you create the volume group by specifying the --setautoactivation n flag for the vgcreate command.

On both nodes, edit file /etc/lvm/lvm.conf and modify the auto_activation_volume_list entry to limit autoactivation to specific volume groups. Search for parameter auto_activation_volume_list and add the volume groups, other than the volume group that you defined for the NFS cluster, as entries in that list.

Sample setting of the auto_activation_volume_list entry in /etc/lvm/lvm.conf:

auto_activation_volume_list = [ "rhel_root" ]

Rebuild the initramfs boot image to make sure that the boot image does not activate a volume group that is controlled by the cluster.

On both nodes, run the following command.

dracut -H -f /boot/initramfs-$(uname -r).img $(uname -r)

Reboot both nodes.

Installing and setting up the RHEL HA Add-On cluster

Use the following instructions to perform the initial cluster configuration.

Install and set up the RHEL HA Add-On cluster according to Implementing a RHEL HA Add-On Cluster on IBM Power Virtual Server.
Configure and test fencing as described in Creating the fencing device.

Sample output of the cluster status at this stage.

# pcs status
Cluster name: SAP_NFS
Cluster Summary:
  * Stack: corosync
  * Current DC: cl-nfs-1 (version 2.0.5-9.el8_4.5-ba59be7122) - partition with quorum
  * Last updated: Fri Mar 10 10:35:42 2023
  * Last change:  Fri Mar 10 09:52:08 2023 by root via cibadmin on cl-nfs-1
  * 2 nodes configured
  * 1 resource instance configured

Node List:
  * Online: [ cl-nfs-1 cl-nfs-2 ]

Full List of Resources:
  * res_fence_ibm_powervs	(stonith:fence_ibm_powervs):	 Started cl-nfs-1

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

Configuring general cluster properties

To prevent the cluster from moving healthy resources to another node (for example when you restart the cluster on a previously failed node), you can set the default value for the resource-stickiness meta attribute to 1.

On NODE1, run the following command.

pcs resource defaults update resource-stickiness=1

Configuring NFS resource group and resources

Use the following steps to configure the NFS resources in the cluster.

Creating the LVM-activate resource

To make sure that all NFS resources run on the same node, configure them as part of the resource group nfsgroup.

This resource group is created with the first resource. Resources start in the order in which they are added to the group. The resources stop in reverse order.

On NODE1, run the following command.

pcs resource create nfs_vg ocf:heartbeat:LVM-activate \
    vgname=${NFS_VG} \
    vg_access_mode=system_id \
    --group nfsgroup

To avoid data corruption, don't configure more than one LVM-activate resource that uses the same LVM volume group in an active-passive HA configuration. Don't configure an LVM-activate resource as a clone resource in an active-passive HA configuration.

Check the status of the cluster and verify that resource nfs_vg is active.

On NODE1, run the following command.

pcs resource status

Sample output:

# pcs resource status
  * Resource Group: nfsgroup:
    * nfs_vg    (ocf::heartbeat:LVM-activate):   Started cl-nfs-1

The following command configures the xfs file system resources for the nfsgroup resource group. The file systems use LVM volume group ${NFS_vg} and the logical volumes (${NFS_LV1}, ${NFS_LV2}, ${NFS_LV3}) that were created before.

On NODE1, run the following command.

pcs resource create nfs_fs1 Filesystem \
    device=/dev/${NFS_VG}/${NFS_LV1} \
    directory=${NFS_FS1} \
    fstype=xfs \
    --group nfsgroup

You can specify mount options as part of the resource configuration for a file system resource by using the options=<options> parameter. Run pcs resource describe filesystem for a complete list of configuration options.

Check the status of the cluster and verify that the resource nfs_fs1 is active.

pcs resource status

Sample output:

# pcs resource status
  * Resource Group: nfsgroup:
    * nfs_vg    (ocf::heartbeat:LVM-activate):   Started cl-nfs-1
    * nfs_fs1   (ocf::heartbeat:Filesystem):     Started cl-nfs-1

On the node with the active resource group nfsgroup, create two subdirectories in ${NFS_FS1}. ${NFS_FS1}/stat is used as nfs_shared_infodir for NFS lock information and ${NFS_FS1}/export is used as NFS root.

mkdir ${NFS_FS1}/stat ${NFS_FS1}/export

Create the mount points for the other exported file systems.

On both nodes, run the following command.

mkdir ${NFS_FS2} ${NFS_FS3}

Create the resources for the other two NFS file systems.

On NODE1, run the following commands.

pcs resource create nfs_fs2 Filesystem \
    device=/dev/${NFS_VG}/${NFS_LV2} \
    directory=${NFS_FS2} \
    fstype=xfs \
    --group nfsgroup

pcs resource create nfs_fs3 Filesystem \
    device=/dev/${NFS_VG}/${NFS_LV3} \
    directory=${NFS_FS3} \
    fstype=xfs \
    --group nfsgroup

Check the status of the cluster and verify that all three file system resources (nfs_fs1, nfs_fs2, nfs_fs3) are active.

pcs resource status

Sample output:

# pcs resource status
  * Resource Group: nfsgroup:
    * nfs_vg    (ocf::heartbeat:LVM-activate):   Started cl-nfs-1
    * nfs_fs1   (ocf::heartbeat:Filesystem):     Started cl-nfs-1
    * nfs_fs2   (ocf::heartbeat:Filesystem):     Started cl-nfs-1
    * nfs_fs3   (ocf::heartbeat:Filesystem):     Started cl-nfs-1

Creating the nfsserver resource

On NODE1, create a resource for managing the NFS server.

pcs resource create nfs_daemon nfsserver \
    nfs_shared_infodir=${NFS_FS1}/stat \
    nfs_no_notify=true \
    --group nfsgroup

The nfs_shared_infodir parameter of the nfsserver resource specifies a directory where the NFS server stores stateful information.

Check the status of the cluster and verify that the NFS server is started.

pcs resource status

Sample output:

# pcs resource status
  * Resource Group: nfsgroup:
    * nfs_vg    (ocf::heartbeat:LVM-activate):   Started cl-nfs-1
    * nfs_fs1   (ocf::heartbeat:Filesystem):     Started cl-nfs-1
    * nfs_fs2   (ocf::heartbeat:Filesystem):     Started cl-nfs-1
    * nfs_fs3   (ocf::heartbeat:Filesystem):     Started cl-nfs-1
    * nfs_daemon        (ocf::heartbeat:nfsserver):      Started cl-nfs-1

Creating the exportfs resource

To export the ${NFS_ROOT} directory, add the exportfs resources to the nfsgroup group, which builds a virtual directory for NFSv4 clients. NFSv3 clients can access these exports too.

On NODE1, run the following command.

pcs resource create nfs_export exportfs \
    clientspec=${NFS_CLIENTSPEC} \
    options=${NFS_OPTIONS} \
    directory=${NFS_ROOT} \
    fsid=0 \
    --group nfsgroup

Configuring a floating IP address resource

Review the information in Reserving virtual IP addresses and reserve a virtual IP address for the NFS cluster.

Create a resource for the virtual IP address of the NFS Server. NFS clients access the NFS share by using the floating IP address.

On NODE1, run the following command.

pcs resource create nfs_ip IPaddr2 \
    ip=${NFS_IP} \
    --group nfsgroup

Configuring a notify resource

The nfsnotify resource sends FSv3 reboot notifications after the entire NFS deployment initializes.

On NODE1, run the following command.

pcs resource create nfs_notify nfsnotify \
    source_host=${NFS_IP} \
    --group nfsgroup

The NFS cluster setup is now complete.

On NODE1, run the following command to check the status.

pcs resource status

Sample output:

# pcs resource status
  * Resource Group: nfsgroup:
    * nfs_vg    (ocf::heartbeat:LVM-activate):   Started cl-nfs-1
    * nfs_fs1   (ocf::heartbeat:Filesystem):     Started cl-nfs-1
    * nfs_fs2   (ocf::heartbeat:Filesystem):     Started cl-nfs-1
    * nfs_fs3   (ocf::heartbeat:Filesystem):     Started cl-nfs-1
    * nfs_daemon        (ocf::heartbeat:nfsserver):      Started cl-nfs-1
    * nfs_export        (ocf::heartbeat:exportfs):       Started cl-nfs-1
    * nfs_ip    (ocf::heartbeat:IPaddr2):        Started cl-nfs-1
    * nfs_notify        (ocf::heartbeat:nfsnotify):      Started cl-nfs-1

Testing the NFS server cluster

You can validate the NFS resource configuration in a high availability cluster by using the following procedures. You can mount the exported file system with either NFSv3 or NFSv4. Run the following tests to verify that the NFS cluster functions.

Test1 - Testing the NFS export

Use the following information to test the NFS export.

Run all the commands on an NFS client node outside the HA NFS cluster.

Verify the NFS exports.

showmount -e ${NFS_IP}

The showmount command displays information about file systems that are exported by an NFS Server (NFS v3). Verify that the output lists all the exported directories.

Create a temporary directory on the NFS client node. Then, mount the NFS file system and create the directory structure that is required for the SAP installation.

In the first example, only /usr/sap/trans and /sapmnt/${SID} are NFS mounted on the SAP application server instance.

Prepare the mount points that are used for the SAP installation.

mkdir -p /sapmnt/${SID} \
         /usr/sap/trans

Change the attributes of the mount points.

chattr +i /sapmnt/${SID} \
          /usr/sap/trans

Mount the NFS shares.

mount -t nfs -o vers=4,minorversion=1,sec=sys ${NFS_VH}:/saptrans /usr/sap/trans

mount -t nfs -o vers=4,minorversion=1,sec=sys ${NFS_VH}:/sap${SID} /sapmnt/${SID}

Change the ownership and the permissions.

chown ${sid}adm:sapsys /usr/sap/trans

chmod g+w /usr/sap/trans

chown -R ${sid}adm:sapsys /sapmnt/${SID}

Unmount the file systems.

umount /usr/sap/trans

umount /sapmnt/${SID}

Add the new file systems to /etc/fstab.

cat >> /etc/fstab << EOT
${NFS_VH}:/saptrans /usr/sap/trans  nfs vers=4,minorversion=1,sec=sys  0  0
${NFS_VH}:/sap${SID} /sapmnt/${SID}  nfs vers=4,minorversion=1,sec=sys  0  0
EOT

Check the updated file.

cat /etc/fstab

In the second example, /usr/sap/trans, /sapmnt/${SID}, and all instance directories are NFS mounted on the SAP application server instances.

Export environment variables for the ASCS and ERS system numbers. Change the following numbers to the system numbers that you used during ASCS and ERS installation.

export ASCS_NR=01

export ERS_NR=02

Prepare the final mount points that are used for the SAP installation.

mkdir -p /sapmnt/${SID} \
         /usr/sap/trans \
         /usr/sap/${SID}/SYS \
         /usr/sap/${SID}/ASCS${ASCS_INSTNO} \
         /usr/sap/${SID}/ERS${ERS_INSTNO}

Change the attributes of the mount points.

chattr +i /sapmnt/${SID} \
          /usr/sap/trans \
          /usr/sap/${SID}/SYS \
          /usr/sap/${SID}/ASCS${ASCS_INSTNO} \
          /usr/sap/${SID}/ERS${ERS_INSTNO}

Mount the NFS shares to create the required subdirectories, change the ownership, and change the permissions.

mount -t nfs -o vers=4,minorversion=1,sec=sys ${NFS_VH}:/saptrans /mnt

chown ${sid}adm:sapsys /mnt

chmod g+w /mnt

umount /mnt

mount -t nfs -o vers=4,minorversion=1,sec=sys ${NFS_VH}:/sap${SID} /mnt

mkdir -p /mnt/sapmnt \
         /mnt/ASCS \
         /mnt/ERS \
         /mnt/SYS \
         /mnt/PAS \
         /mnt/AS1

chown -R ${sid}adm:sapsys /mnt

umount /mnt

Add the new file systems to /etc/fstab.

cat >> /etc/fstab < EOT
${NFS_VH}:/saptrans /usr/sap/trans  nfs vers=4,minorversion=1,sec=sys  0  0
${NFS_VH}:/sap${SID}/sapmnt /sapmnt/${SID}  nfs vers=4,minorversion=1,sec=sys  0  0
${NFS_VH}:/sap${SID}/ASCS /usr/sap/${SID}/ASCS${ASCS_INSTNO} nfs vers=4,minorversion=1,sec=sys  0  0
${NFS_VH}:/sap${SID}/ERS  /usr/sap/${SID}/ERS${ERS_INSTNO} nfs vers=4,minorversion=1,sec=sys  0  0
${NFS_VH}:/sap${SID}/SYS  /usr/sap/${SID}/SYS  nfs vers=4,minorversion=1,sec=sys  0  0
EOT

Check the updated file.

cat /etc/fstab

Test2 - Testing the failover of the NFS server

Use the following information to test the failover of the NFS server.

Test2 - Description

Simulate a crash of the cluster node that has the NFS resources.

Test2 - Prerequisites

A functional two-node RHEL HA Add-On cluster for an NFS HA server.
Cluster is started on both nodes.
The file systems are mounted on an NFS client node outside the cluster and the applications can access the content.

Test2 - Test procedure

Crash the cluster node by sending a shutoff system request.

First, check the cluster status and identify the node where the nfsgroup resource group is running.

On NODE1, run the following command.

pcs status

Then, log in to the identified cluster node and send a shutoff system request.

sync; echo o > /proc/sysrq-trigger

Test2 - Expected behavior

The node with the active nfsgroup resource group shuts down.
The cluster detects the failed node and starts a fencing action.
The fencing operation sets the state of the fenced node to offline.
The cluster acquires the NFS Server resources on the failover node.

Check that all the resources started on the failover node.

pcs resource status

Sample output:

# pcs resource status
  * Resource Group: nfsgroup:
    * nfs_vg    (ocf::heartbeat:LVM-activate):   Started cl-nfs-2
    * nfs_fs1   (ocf::heartbeat:Filesystem):     Started cl-nfs-2
    * nfs_fs2   (ocf::heartbeat:Filesystem):     Started cl-nfs-2
    * nfs_fs3   (ocf::heartbeat:Filesystem):     Started cl-nfs-2
    * nfs_daemon        (ocf::heartbeat:nfsserver):      Started cl-nfs-2
    * nfs_export        (ocf::heartbeat:exportfs):       Started cl-nfs-2
    * nfs_ip    (ocf::heartbeat:IPaddr2):        Started cl-nfs-2
    * nfs_notify        (ocf::heartbeat:nfsnotify):      Started cl-nfs-2

Verify that the file system is still mounted on the NFS client node, and that the applications can still access the content.

Test2 - Recovery procedure

Log in to the IBM Cloud Console and start the stopped instance. Wait until the cluster node is available again, then restart the cluster framework.

On the cluster node, run the following command.

pcs cluster start

Check the cluster status.

pcs status

Sample output:

# pcs status
Cluster name: SAP_NFS
Cluster Summary:
  * Stack: corosync
  * Current DC: cl-nfs-1 (version 2.0.5-9.el8_4.5-ba59be7122) - partition with quorum
  * Last updated: Mon Mar 20 08:11:28 2023
  * Last change:  Mon Mar 20 07:56:25 2023 by hacluster via crmd on cl-nfs-1
  * 2 nodes configured
  * 9 resource instances configured

Node List:
  * Online: [ cl-nfs-1 cl-nfs-2 ]

Full List of Resources:
  * res_fence_ibm_powervs	(stonith:fence_ibm_powervs):	 Started cl-nfs-1
  * Resource Group: nfsgroup:
    * nfs_vg	(ocf::heartbeat:LVM-activate):	 Started cl-nfs-2
    * nfs_fs1	(ocf::heartbeat:Filesystem):	 Started cl-nfs-2
    * nfs_fs2	(ocf::heartbeat:Filesystem):	 Started cl-nfs-2
    * nfs_fs3	(ocf::heartbeat:Filesystem):	 Started cl-nfs-2
    * nfs_daemon	(ocf::heartbeat:nfsserver):	 Started cl-nfs-2
    * nfs_export	(ocf::heartbeat:exportfs):	 Started cl-nfs-2
    * nfs_ip	(ocf::heartbeat:IPaddr2):	 Started cl-nfs-2
    * nfs_notify	(ocf::heartbeat:nfsnotify):	 Started cl-nfs-2

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled