Setting extra environment variables for a single zone implementation

Review Reserving virtual IP addresses and reserve a virtual IP address for the SAP HANA system replication cluster. Set the VIP environment variable to the reserved IP address.

Setting extra environment variables for a multizone region implementation

Set the CLOUD_REGION, APIKEY, IBMCLOUD_CRN_?, POWERVSI_? variables as described in the Collecting parameters for configuring a high availability cluster section. Set API_TYPE to private to enable communication with the IBM Cloud IAM and IBM Power Cloud API through private endpoints.

Prepare the variables for the subnet:

• SUBNET_NAME specifies the name of the subnet.
• CIDR defines the subnet in Classless Inter-Domain Routing (CIDR) format: <IPv4_address>/number.
• VIP is the virtual IP address and must fall within the defined CIDR.
• Set JUMBO to true to enable support for large MTU sizes on the subnet.

The subnet SUBNET_NAME must not exist, and its CIDR range must not overlap with the IP ranges of any existing subnets in either workspace.

The following export commands show how to set the required environment variables for a multizone region implementation.

export CLOUD_REGION="eu-de"
export IBMCLOUD_CRN_1="crn:v1:bluemix:public:power-iaas:eu-de-2:a/a1b2c3d4e5f60123456789a1b2c3d4e5:a1b2c3d4-0123-4567-89ab-a1b2c3d4e5f6::"
export IBMCLOUD_CRN_2="crn:v1:bluemix:public:power-iaas:eu-de-1:a/a1b2c3d4e5f60123456789a1b2c3d4e5:e5f6a1b2-cdef-0123-4567-a1b2c3d4e5f6::"
export POWERVSI_1="a1b2c3d4-0123-890a-f012-0123456789ab"
export POWERVSI_2="e5f6a1b2-4567-bcde-3456-cdef01234567"
export APIKEY="@/root/.apikey.json"
export API_TYPE="private"
export SUBNET_NAME="vip-mh1-net"
export CIDR="10.40.41.100/30"
export VIP="10.40.41.102"
export JUMBO="true"

Configuring the hook HanaSR for all SAP HANA instances

The sap-hana-ha package includes the HanaSR.py hook script, which is located in the /usr/share/sap-hana-ha directory. Configure this hook on all SAP HANA cluster nodes.

Enable and test the SAP HANA srConnectionChanged() hook before you continue with the cluster setup.

1. Update the global.ini file on each SAP HANA node to enable the HanaSR hook script.

   On both nodes, run the following command.

   sudo -i -u ${sid}adm -- <<EOT
       python \$DIR_INSTANCE/exe/python_support/setParameter.py \
         -set SYSTEM/global.ini/ha_dr_provider_hanasr/provider=HanaSR \
         -set SYSTEM/global.ini/ha_dr_provider_hanasr/path=/usr/share/sap-hana-ha/ \
         -set SYSTEM/global.ini/ha_dr_provider_hanasr/execution_order=1 \
         -set SYSTEM/global.ini/trace/ha_dr_hanasr=info
   EOT
   

2. Verify the updated global.ini file.

   On both nodes, run the following command.

   cat /hana/shared/${SID}/global/hdb/custom/config/global.ini
   

3. Configure sudo settings for the SAP HANA operating system administrator sidadm.

   These settings allow the hook script srConnectionChanged() to update node attributes.

   Run the following commands on both nodes to create the required sudo configuration file.

   cat >> /etc/sudoers.d/20-saphana << EOT
   Cmnd_Alias DC1_SOK = /usr/sbin/crm_attribute -n hana_${sid}_site_srHook_${DC1} -v SOK -t crm_config -s SAPHanaSR
   Cmnd_Alias DC1_SFAIL = /usr/sbin/crm_attribute -n hana_${sid}_site_srHook_${DC1} -v SFAIL -t crm_config -s SAPHanaSR
   Cmnd_Alias DC2_SOK = /usr/sbin/crm_attribute -n hana_${sid}_site_srHook_${DC2} -v SOK -t crm_config -s SAPHanaSR
   Cmnd_Alias DC2_SFAIL = /usr/sbin/crm_attribute -n hana_${sid}_site_srHook_${DC2} -v SFAIL -t crm_config -s SAPHanaSR
   Cmnd_Alias FENCE_ME = /usr/bin/SAPHanaSR-hookHelper --sid=${SID} --case=fenceMe
   ${sid}adm ALL=(ALL) NOPASSWD: DC1_SOK, DC1_SFAIL, DC2_SOK, DC2_SFAIL, FENCE_ME
   Defaults!DC1_SOK, DC1_SFAIL, DC2_SOK, DC2_SFAIL, FENCE_ME !requiretty
   EOT
   

   The Cmnd_Alias FENCE_ME is only necessary when you configure the ChkSrv hook script with the parameter action_on_lost = fence.

   Set the correct ownership and permissions and use visudo to validate its syntax.

   chown root:root /etc/sudoers.d/20-saphana
   

   chmod 0440 /etc/sudoers.d/20-saphana
   

   cat /etc/sudoers.d/20-saphana
   

   visudo -c
   

Correct any issues reported by the visudo -c command.

Configuring the ChkSrv hook for all SAP HANA instances (optional)

The sap-hana-ha package includes the ChkSrv.py hook script, which is located in the /usr/share/sap-hana-ha directory. Configure this hook on all SAP HANA cluster nodes.

You can configure the hook srServiceStateChanged() to accelerate recovery when an indexserver process fails. This configuration is optional.

The ChkSrv.py script stops the entire SAP HANA instance for faster recovery. If automated failover is enabled in the cluster and the secondary node is in a healthy state, the system initiates a takeover operation. If failover is not possible, the script forces a restart of the SAP HANA instance on the local node.

The hook script analyzes instance events, filters the event details, and triggers actions based on the filtered results. It distinguishes between an SAP HANA indexserver process that is stopped and restarted by the system, and one that is stopped as part of an instance shutdown.

The actions that are triggered by the hook depend on the configuration of the action_on_lost parameter.

ignore

This action logs the event details and the corresponding decision logic to a file.

stop

This action gracefully stops the SAP HANA instance by using the sapcontrol command.

kill

This action triggers the HDB kill-<signal> command with a default signal 9. The signal can be configured.

Both the stop and the kill actions stop the SAP HANA instance, but the kill action is slightly faster.

fence

This action fences the cluster node when a failed indexserver process is detected. Grant sudo privileges to allow the sidadm user to run the /usr/bin/SAPHanaSR-hookHelper script.

1. Update the global.ini file on each SAP HANA node to enable the hook script.

   On both nodes, run the following command.

   sudo -i -u ${sid}adm -- <<EOT
       python \$DIR_INSTANCE/exe/python_support/setParameter.py \
         -set SYSTEM/global.ini/ha_dr_provider_chksrv/provider=ChkSrv \
         -set SYSTEM/global.ini/ha_dr_provider_chksrv/path=/usr/share/sap-hana-ha/ \
         -set SYSTEM/global.ini/ha_dr_provider_chksrv/execution_order=2 \
         -set SYSTEM/global.ini/ha_dr_provider_chksrv/action_on_lost=stop \
         -set SYSTEM/global.ini/trace/ha_dr_chksrv=info
   EOT
   

   In this example, the action_on_lost parameter is set to stop; the default value is ignore. You can optionally configure the stop_timeout parameter (default: 20 seconds) and the kill_signal parameter (default: 9).

Activating the hooks

1. Reload the hook configuration.

   On both nodes, run the following command to activate the hooks.

   sudo -i -u ${sid}adm -- hdbnsutil -reloadHADRProviders
   

2. Verify that the hooks are active by checking the trace logs.

   On both nodes, run the following command.

   sudo -i -u ${sid}adm -- sh -c 'grep "ha_dr_provider" $DIR_INSTANCE/$VTHOSTNAME/trace/nameserver_* | cut -d" " -f2,3,6-'
   

   Sample Output:

   2025-02-27 15:47:21.855038   HADRProviderManager.cpp(00087) : loading HA/DR Provider 'ChkSrv' from /usr/share/sap-hana-ha/
   2025-02-27 15:47:22.648229   HADRProviderManager.cpp(00087) : loading HA/DR Provider 'HanaSR' from /usr/share/sap-hana-ha/
   

Creating a virtual IP cluster resource in a single zone environment

Use the reserved IP address to create a virtual IP address cluster resource. This IP address enables access to the SAP HANA system replication primary instance.

Create the virtual IP address cluster resource with the following command.

pcs resource create vip_${SID}_${INSTNO} IPaddr2 \
    ip=$VIP \
    --disabled

Verify the virtual IP address resource configuration and cluster status.

pcs resource config vip_${SID}_${INSTNO}

pcs status --full

Proceed to the Creating cluster resource constraints section.

Creating a virtual IP cluster resource in a multizone region environment

Ensure that you completed all steps in the Preparing a multi-zone RHEL HA Add-On cluster for a virtual IP address resource section.

Use the pcs resource describe powervs-subnet command to get information about the resource agent parameters.

On NODE1, create a powervs-subnet cluster resource by running the following command.

pcs resource create vip_${SID}_${INSTNO} powervs-subnet \
    api_key=${APIKEY} \
    api_type=${API_TYPE} \
    cidr=${CIDR} \
    ip=${VIP} \
    crn_host_map="${NODE1}:${IBMCLOUD_CRN_1};${NODE2}:${IBMCLOUD_CRN_2}" \
    vsi_host_map="${NODE1}:${POWERVSI_1};${NODE2}:${POWERVSI_2}" \
    jumbo=${JUMBO} \
    region=${CLOUD_REGION} \
    subnet_name=${SUBNET_NAME} \
    op start timeout=720 \
    op stop timeout=300 \
    op monitor interval=60 timeout=30 \
    --disabled

When API_TYPE is set to public, you must also specify the proxy parameter.

Before you run the pcs resource config command, verify that both virtual server instances in the cluster have the status Active, and their health status is OK.

Verify the configured virtual IP resource and cluster status.

pcs resource config vip_${SID}_${INSTNO}

Sample output:

# pcs resource config vip_MH1_00
Resource: vip_MH1_00 (class=ocf provider=heartbeat type=powervs-subnet)
  Attributes: vip_MH1_00-instance_attributes
    api_key=@/root/.apikey.json
    api_type=private
    cidr=10.40.41.100/30
    crn_host_map=cl-mh1-1:crn:v1:bluemix:public:power-iaas:eu-de-2:a/**********************************:**********************************::;cl-mh1-2:crn:v1:bluemix:public:power-iaas:eu-
        de-1:a/**********************************:**********************************::
    ip=10.40.41.102
    jumbo=true
    region=eu-de
    subnet_name=vip-mh1-net
    vsi_host_map=cl-mh1-1:**********************************;cl-mh1-2:**********************************
  Operations:
    monitor: vip_MH1_00-monitor-interval-60
      interval=60 timeout=30
    start: vip_MH1_00-start-interval-0s
      interval=0s timeout=720
    stop: vip_MH1_00-stop-interval-0s
      interval=0s timeout=300

Test 1 - Description

Simulate a crash of the primary SAP HANA database instance that is running on NODE1.

Test 1 - Prerequisites

• A functional two-node RHEL HA Add-On cluster configured for HANA system replication
• The cluster is running on both nodes
• The SAPHanaController_${SID}_${INSTNO} resource is configured with AUTOMATED_REGISTER=false
• Verify the SAP HANA system replication status:
  • Primary SAP HANA database is running on NODE1
  • Secondary SAP HANA database is running on NODE2
  • System replication is activated and in sync

Test 1 - Test procedure

Simulate a crash of the primary SAP HANA instance by sending a SIGKILL signal as the sidadm user.

On NODE1, run the following command.

sudo -i -u ${sid}adm -- HDB kill-9

Test 1 - Expected behavior

• SAP HANA primary instance on NODE1 crashes.
• The cluster detects the stopped primary HANA database and marks the resource as failed.
• The cluster promotes the secondary HANA database on NODE2 to become the new primary.
• The virtual IP address is released from NODE1 and reassigned to NODE2.
• Applications such as SAP NetWeaver, if connected to a tenant database, automatically reconnect to the new primary.

Test 1 - Recovery procedure

Because the SAPHanaController_${SID}_${INSTNO} resource is configured with AUTOMATED_REGISTER=false, the cluster does not restart the failed SAP HANA instance or register it against the new primary. As a result, the new primary on NODE2 shows the former secondary in CONNECTION TIMEOUT status.

To manually reregister the previous primary as a new secondary, run the following command on NODE1.

sudo -i -u ${sid}adm -- <<EOT
    hdbnsutil -sr_register \
      --name=${DC1} \
      --remoteHost=${NODE2} \
      --remoteInstance=00 \
      --replicationMode=sync \
      --operationMode=logreplay \
      --online
EOT

Verify the SAP HANA system replication status.

sudo -i -u ${sid}adm -- <<EOT
    hdbnsutil -sr_state
    HDBSettings.sh systemReplicationStatus.py
EOT

After manual registration and resource refresh, the new secondary instance restarts and appears in SOK (synced) status.

On NODE1, run the following command.

pcs resource refresh SAPHanaController_${SID}_${INSTNO}

pcs status --full

Test 2 - Description

Simulate a crash of the node that hosts the primary SAP HANA database instance.

Test 2 - Preparation

Make sure that the SAPHanaController_${SID}_${INSTNO} resource is configured with AUTOMATED_REGISTER=true.

On NODE1, run the following commands.

pcs resource update SAPHanaController_${SID}_${INSTNO} AUTOMATED_REGISTER=true

pcs resource config SAPHanaController_${SID}_${INSTNO}

Test 2 - Prerequisites

• A functional two-node RHEL HA Add-On cluster configured for HANA system replication
• The cluster is running on both nodes
• Verify the SAP HANA system replication status:
  • Primary SAP HANA database is running on NODE2
  • Secondary SAP HANA database is running on NODE1
  • System replication is activated and in sync

Test 2 - Test procedure

Simulate a node failure on NODE2 by triggering a system crash.

On NODE2, run the following command.

sync; echo c > /proc/sysrq-trigger

Test 2 - Expected behavior

• NODE2 shuts down.
• The cluster detects the failed node and marks its state as OFFLINE.
• The cluster promotes the secondary SAP HANA instance on NODE1 to become the new primary.
• The virtual IP address is reassigned to NODE1.
• Applications such as SAP NetWeaver, if connected to a tenant database, automatically reconnect to the new primary.

Test 2 - Recovery procedure

Log in to the IBM Cloud® Console and start the NODE2 instance. After NODE2 becomes available, restart the cluster framework.

On NODE2, run the following commands.

pcs cluster start

pcs status --full

Because AUTOMATED_REGISTER=true is configured for the SAPHanaController_${SID}_${INSTNO} resource, SAP HANA automatically restarts when NODE_B1 rejoins the cluster. The former primary instance is reregistered as a secondary.

Test 3 - Description

This test simulates a crash of the secondary SAP HANA instance to validate cluster failover behavior.

Test 3 - Prerequisites

• A functional two-node RHEL HA Add-On cluster configured for HANA system replication
• The cluster is running in both nodes
• The SAPHanaController_${SID}_${INSTNO} resource is configured with AUTOMATED_REGISTER=true
• Verify the SAP HANA system replication status:
  • Primary SAP HANA database is running on NODE1
  • Secondary SAP HANA database is running on NODE2
  • HANA system replication is activated and in sync

Test 3 - Test Procedure

Simulate a crash of the secondary SAP HANA instance by sending a SIGKILL signal as the sidadm user.

On NODE2, run the following command.

sudo -i -u ${sid}adm -- HDB kill-9

Test 3 - Expected behavior

• The secondary SAP HANA instance on NODE2 crashes.
• The cluster detects the stopped secondary HANA database and marks the resource as failed.
• The cluster restarts the secondary HANA instance.
• The cluster verifies that system replication is reestablished and in sync.

Test 3 - Recovery procedure

Wait until the secondary SAP HANA instance starts and system replication status returns to SOK. Then, clean up any failed resource actions as indicated by pcs status.

On NODE2, run the following commands.

pcs resource refresh SAPHanaController_${SID}_${INSTNO}

pcs status --full

Test 4 - Description

Use cluster management commands to relocate the primary SAP HANA instance to another node for maintenance.

Test 4 - Prerequisites

• A functional two-node RHEL HA Add-On cluster configured for SAP HANA system replication
• The cluster is running on both nodes
• The SAPHanaController_${SID}_${INSTNO} resource is configured with AUTOMATED_REGISTER=true
• Verify the SAP HANA system replication status:
  • Primary SAP HANA instance is running on NODE1
  • Secondary SAP HANA instance is running on NODE2
  • HANA system replication is activated and in sync

Test 4 - Test procedure

Manually move the primary SAP HANA instance to another node by using the pcs resource move command.

On NODE1, run the following command.

pcs resource move SAPHanaController_${SID}_${INSTNO}-clone

Test 4 - Expected behavior

• The cluster creates temporary location constraints to relocate the resource to the target node.
• The cluster initiates a takeover, promoting the secondary SAP HANA instance to primary.
• Applications such as SAP NetWeaver, if connected to a tenant database, automatically reconnect to the new primary instance.

Test 4 - Recovery procedure

As part of the resource move process, the cluster automatically registers and starts the original primary node as the new secondary instance. After the move completes successfully, the cluster removes any temporary location constraints.

Run the following commands on NODE1 to confirm that temporary constraints are removed and to verify the cluster status.

pcs constraint

pcs status --full