Configuring SAP HANA Scale-Up System Replication in a RHEL HA Add-On cluster

Activating the srConnectionChanged() hook on all SAP HANA instances

1. Stop the cluster.

   On NODE1, run the following command.

   pcs cluster stop --all
   

2. Install the hook script that is provided by the resource-agents-sap-hana package in the /hana/shared/myHooks directory for each HANA instance, and set the required ownership.

   On both nodes, run the following commands.

   mkdir -p /hana/shared/myHooks
   

   cp /usr/share/SAPHanaSR/srHook/SAPHanaSR.py /hana/shared/myHooks
   

   chown -R ${sid}adm:sapsys /hana/shared/myHooks
   

3. Update the global.ini file on each HANA node to enable the hook script.

   On both nodes, run the following command.

   sudo -i -u ${sid}adm -- <<EOT
       python \$DIR_INSTANCE/exe/python_support/setParameter.py \
         -set SYSTEM/global.ini/ha_dr_provider_SAPHanaSR/provider=SAPHanaSR \
         -set SYSTEM/global.ini/ha_dr_provider_SAPHanaSR/path=/hana/shared/myHooks \
         -set SYSTEM/global.ini/ha_dr_provider_SAPHanaSR/execution_order=1 \
         -set SYSTEM/global.ini/trace/ha_dr_saphanasr=info
   EOT
   

4. Verify the changed file.

   On both nodes, run the following command.

   cat /hana/shared/${SID}/global/hdb/custom/config/global.ini
   

5. Create sudo settings for SAP HANA OS user.

   You need the following sudo settings to allow the ${sid}adm user script can update the node attributes when the srConnectionChanged() hook runs.

   On both nodes, run the following commands.

   Create a file with the required sudo aliases and user specifications.

   cat >> /etc/sudoers.d/20-saphana << EOT
   Cmnd_Alias DC1_SOK = /usr/sbin/crm_attribute -n hana_${sid}_site_srHook_${DC1} -v SOK -t crm_config -s SAPHanaSR
   Cmnd_Alias DC1_SFAIL = /usr/sbin/crm_attribute -n hana_${sid}_site_srHook_${DC1} -v SFAIL -t crm_config -s SAPHanaSR
   Cmnd_Alias DC2_SOK = /usr/sbin/crm_attribute -n hana_${sid}_site_srHook_${DC2} -v SOK -t crm_config -s SAPHanaSR
   Cmnd_Alias DC2_SFAIL = /usr/sbin/crm_attribute -n hana_${sid}_site_srHook_${DC2} -v SFAIL -t crm_config -s SAPHanaSR
   ${sid}adm ALL=(ALL) NOPASSWD: DC1_SOK, DC1_SFAIL, DC2_SOK, DC2_SFAIL
   Defaults!DC1_SOK, DC1_SFAIL, DC2_SOK, DC2_SFAIL !requiretty
   EOT
   

   Adjust the permissions and check for syntax errors.

   chown root:root /etc/sudoers.d/20-saphana
   

   chmod 0440 /etc/sudoers.d/20-saphana
   

   cat /etc/sudoers.d/20-saphana
   

   visudo -c
   

Any problems that are reported by the visudo -c command must be corrected.

1. Verify that the hook functions.

   • Restart both HANA instances and verify that the hook script works as expected.
   • Perform an action to trigger the hook, such as stopping a HANA instance.
   • Check whether the hook logged anything in the trace files.

   On both nodes, run the following commands.

   Restart and then stop the HANA instance.

   sudo -i -u ${sid}adm -- HDB restart
   

   sudo -i -u ${sid}adm -- HDB stop
   

   Check messages in trace files.

   sudo -i -u ${sid}adm -- sh -c 'grep "ha_dr_SAPHanaSR.*crm_attribute" $DIR_INSTANCE/$VTHOSTNAME/trace/nameserver_* | cut -d" " -f2,3,5,17'
   

   Start the HANA instance.

   sudo -i -u ${sid}adm -- HDB start
   

   After you verify that the hooks function, you can restart the HA cluster.

2. Start cluster.

   On NODE1, run the following commands.

   Start the cluster.

   pcs cluster start --all
   

   Check the status of the cluster.

   pcs status --full
   

Test1 - Description

Simulate a crash of the primary HANA database instance that is running on NODE1.

Test1 - Prerequisites

• A functional two-node RHEL HA Add-On cluster for HANA system replication.
• Both cluster nodes are active.
• Cluster that is started on NODE1 and NODE2.
• Cluster Resource SAPHana_${SID}_${INSTNO} that is configured with AUTOMATED_REGISTER=false.
• Check SAP HANA System Replication status:
  • Primary SAP HANA database is running on NODE1
  • Secondary SAP HANA database is running on NODE2
  • HANA System Replication is activated and in sync

Test1 - Test procedure

Crash SAP HANA primary by sending a SIGKILL signal as user ${sid}adm.

On NODE1, run the following command.

sudo -i -u ${sid}adm -- HDB kill-9

Test1 - Expected behavior

• SAP HANA primary instance on NODE1 crashes.
• The cluster detects the stopped primary HANA database and marks the resource as failed.
• The cluster promotes the secondary HANA database on NODE2 to take over as new primary.
• The cluster releases the virtual IP address on NODE1, and acquires it on the new primary on NODE2.
• If an application, such as SAP NetWeaver, is connected to a tenant database of SAP HANA, the application automatically reconnects to the new primary.

Test1 - Recovery procedure

As cluster resource SAPHana_${SID}_${INSTNO} is configured with AUTOMATED_REGISTER=false, the cluster doesn't restart the failed HANA database, and doesn't register it against the new primary. Which means that the status on the new primary (NODE2) also shows the secondary in status 'CONNECTION TIMEOUT'.

To reregister the previous primary as new secondary use the following commands.

On NODE1, run the following command.

sudo -i -u ${sid}adm -- <<EOT
    hdbnsutil -sr_register \
      --name=${DC1} \
      --remoteHost=${NODE2} \
      --remoteInstance=00 \
      --replicationMode=sync \
      --operationMode=logreplay
EOT

Verify the system replication status:

sudo -i -u ${sid}adm -- <<EOT
    hdbnsutil -sr_state
    HDBSettings.sh systemReplicationStatus.py
EOT

After the manual register and resource refreshes, the new secondary instance restarts and shows up in status synced (SOK).

On NODE1, run the following command.

pcs resource refresh SAPHana_${SID}_${INSTNO}

pcs status --full

Test2 - Description

Simulate a crash of the node that is running the primary HANA database.

Test2 - Preparation

Make sure that Cluster Resource SAPHana_${SID}_${INSTNO} is configured with AUTOMATED_REGISTER=true.

On NODE1, run the following command.

pcs resource update SAPHana_${SID}_${INSTNO} AUTOMATED_REGISTER=true

pcs resource config SAPHana_${SID}_${INSTNO}

Test2 - Prerequisites

• A functional two-node RHEL HA Add-On cluster for HANA system replication.
• Both nodes active.
• Cluster is started on NODE1 and NODE2.
• Check SAP HANA System Replication status.
  • Primary SAP HANA database is running on NODE2
  • Secondary SAP HANA database is running on NODE1
  • HANA System Replication is activated and in sync

Test2 - Test procedure

Crash primary on NODE2 by sending a shutoff system request.

On NODE2, run the following command.

sync; echo o > /proc/sysrq-trigger

Test2 - Expected behavior

• NODE2 shuts down.
• The cluster detects the failed node and sets its state to OFFLINE.
• The cluster promotes the secondary HANA database on NODE1 to take over as new primary.
• The cluster acquires the virtual IP address on NODE1 on the new primary.
• If an application, such as SAP NetWeaver, is connected to a tenant database of SAP HANA, the application automatically reconnects to the new primary.

Test2 - Recovery procedure

Log in to the IBM Cloud® Console and start the NODE2 instance. Wait until NODE2 is available again, then restart the cluster framework.

On NODE2, run the following command.

pcs cluster start

pcs status --full

As cluster resource SAPHana_${SID}_${INSTNO} is configured with AUTOMATED_REGISTER=true, SAP HANA restarts when NODE2 rejoins the cluster and the former primary reregisters as a secondary.

Test3 - Description

Simulate a crash of the secondary HANA database.

Test3 - Prerequisites

• A functional two-node RHEL HA Add-On cluster for HANA system replication.
• Both nodes are active.
• Cluster is started on NODE1 and NODE2.
• Cluster Resource SAPHana_${SID}_${INSTNO} is configured with AUTOMATED_REGISTER=true.
• Check SAP HANA System Replication status:
  • Primary SAP HANA database is running on NODE1
  • Secondary SAP HANA database is running on NODE2
  • HANA System Replication is activated and in sync

Test3 - Test Procedure

Crash SAP HANA secondary by sending a SIGKILL signal as user ${sid}adm.

On NODE2, run the following command.

sudo -i -u ${sid}adm -- HDB kill-9

Test3 - Expected behavior

• SAP HANA secondary on NODE2 crashes.
• The cluster detects the stopped secondary HANA database and marks the resource as failed.
• The cluster restarts the secondary HANA database.
• The cluster detects that the system replication is in sync again.

Test3 - Recovery procedure

Wait until the secondary HANA instance starts and syncs again (SOK), then cleanup the failed resource actions as shown in pcs status.

On NODE2, run the following command.

pcs resource refresh SAPHana_${SID}_${INSTNO}

pcs status --full

Test4 - Description

Use cluster commands to move the primary instance to the other node for maintenance purposes.

Test4 - Prerequisites

• A functional two-node RHEL HA Add-On cluster for HANA system replication.
• Both nodes are active.
• Cluster is started on NODE1 and NODE2.
• Cluster Resource SAPHana_${SID}_${INSTNO} is configured with AUTOMATED_REGISTER=true.
• Check SAP HANA System Replication status:
  • Primary SAP HANA database is running on NODE1
  • Secondary SAP HANA database is running on NODE2
  • HANA System Replication is activated and in sync

Test4 - Test procedure

Move SAP HANA primary to other node by using the pcs resource move command.

On NODE1, run the following command.

pcs resource move SAPHana_${SID}_${INSTNO}-clone

Test4 - Expected behavior

• The cluster creates location constraints to move the resource.
• The cluster triggers a takeover to the secondary HANA database.
• If an application, such as SAP NetWeaver, is connected to a tenant database of SAP HANA, the application automatically reconnects to the new primary.

Test4 - Recovery procedure

The automatically created location constraints must be removed to allow automatic failover in the future.

Wait until the primary HANA instance is active and remove the constraints.

The cluster registers and starts the HANA database as new secondary instance.

On NODE1, run the following command.

pcs constraint

pcs resource clear SAPHana_${SID}_${INSTNO}-clone

pcs constraint

pcs status --full