Setting extra environment variables for implementing a single zone

Review the information in Reserving virtual IP addresses and reserve a virtual IP address for the SAP HANA System Replication cluster. Set the VIP environment variable to the reserved IP address.

Activating the srConnectionChanged() hook on all SAP HANA instances

1. Stop the cluster.

   On NODE1, run the following command.

   crm cluster stop --all
   

   Then, follow the steps that are described in Setting up SAP HANA HA/DR providers.

2. Verify that the hook functions.

   • Restart both HANA instances and verify that the hook script works as expected.
   • Perform an action to trigger the hook, such as stopping a HANA instance.
   • Check whether the hook logged anything in the trace files.

   On both nodes, run the following commands.

   Stop the HANA instance.

   sudo -i -u ${sid}adm -- HDB stop
   

   Start the HANA instance.

   sudo -i -u ${sid}adm -- HDB start
   

   Check that the hook logged messages to the trace files.

   sudo -i -u ${sid}adm -- sh -c 'grep "ha_dr_SAPHanaSR.*crm_attribute" $DIR_INSTANCE/$VTHOSTNAME/trace/nameserver_* | cut -d" " -f2,3,5,17'
   

   After you verify that the hooks function, you can restart the HA cluster.

3. Start the cluster.

   On NODE1, run the following commands.

   Start the cluster.

   crm cluster start --all
   

   Check the status of the cluster.

   crm status --full
   

Test 1 - Description

Simulate a crash of the primary HANA database instance that is running on NODE1.

Test 1 - Prerequisites

• A functional two-node SLES HA Extension cluster for HANA system replication.
• Both cluster nodes are active.
• Cluster that is started on NODE1 and NODE2.
• Cluster Resource SAPHana_${SID}_${INSTNO} that is configured with AUTOMATED_REGISTER=false.
• Check SAP HANA System Replication status:
  • Primary SAP HANA database is running on NODE1
  • Secondary SAP HANA database is running on NODE2
  • HANA System Replication is activated and in sync

A variation of Test 1 is described in Test cases for semi-automation.

Test 1 procedure

Use the following command to run Test 1.

Crash SAP HANA primary by sending a SIGKILL signal as the user ${sid}adm.

On NODE1, run the following command.

sudo -i -u ${sid}adm -- HDB kill-9

Test 1 - Expected behavior

You can expect the following behavior from the test.

• SAP HANA primary instance on NODE1 crashes.
• The cluster detects the stopped primary HANA database and marks the resource as failed.
• The cluster promotes the secondary HANA database on NODE2 to take over as the new primary.
• The cluster releases the virtual IP address on NODE1, and acquires it on the new primary on NODE2.
• If an application, such as SAP NetWeaver, is connected to a tenant database of SAP HANA, the application automatically reconnects to the new primary.

Test 1 - Recovery procedure

As the cluster resource SAPHana_${SID}_${INSTNO} is configured with AUTOMATED_REGISTER=false, the cluster doesn't restart the failed HANA database and doesn't register it against the new primary. Which means that the status on the new primary (NODE2) also shows the secondary in status 'CONNECTION TIMEOUT'.

To reregister the previous primary as a new secondary use the following commands.

On NODE1, run the following command.

sudo -i -u ${sid}adm -- <<EOT
    hdbnsutil -sr_register \
      --name=${DC1} \
      --remoteHost=${NODE2} \
      --remoteInstance=00 \
      --replicationMode=sync \
      --operationMode=logreplay \
      --online
EOT

Verify the system replication status by using the following command.

sudo -i -u ${sid}adm -- <<EOT
    hdbnsutil -sr_state
    HDBSettings.sh systemReplicationStatus.py
EOT

After the manual register and resource refreshes, the new secondary instance restarts and shows a synced (SOK) status.

On NODE1, run the following command.

crm resource refresh SAPHana_${SID}_${INSTNO}

Test 2 - Description

Simulate a crash of the node that is running the primary HANA database.

Test 2 - Prerequisites

See the following prerequisites before you perform Test 2.

• You need a functional two-node SLES HA Extension cluster for HANA system replication.
• Make sure that both nodes are active.
• Confirm that the cluster is started on NODE1 and NODE2.
• Check SAP HANA System Replication status.
  • Primary SAP HANA database is running on NODE2.
  • Secondary SAP HANA database is running on NODE1.
  • HANA System Replication is activated and in sync.

Test 2 - Preparation

Make sure that the cluster resource SAPHana_${SID}_${INSTNO} is configured with AUTOMATED_REGISTER=true.

On NODE1, run the following commands.

crm resource update SAPHana_${SID}_${INSTNO} AUTOMATED_REGISTER=true

crm resource config SAPHana_${SID}_${INSTNO}

Test 2 - Test procedure

Crash primary on NODE2 by sending a crash system request.

On NODE2, run the following command.

sync; echo c > /proc/sysrq-trigger

Test 2 - Expected behavior

You can expect the following behavior from the test.

• NODE2 shuts down.
• The cluster detects the failed node and sets its state to OFFLINE.
• The cluster promotes the secondary HANA database on NODE1 to take over as the new primary.
• The cluster acquires the virtual IP address on NODE1 on the new primary.
• If an application, such as SAP NetWeaver, is connected to a tenant database of SAP HANA, the application automatically reconnects to the new primary.

Test 2 - Recovery procedure

Use the following information to recover from Test 2.

1. Log in to the IBM Cloud® Console and start the NODE2 instance.

2. Wait until NODE2 is available again, then restart the cluster framework.

   • On NODE2, run the following commands.

   crm cluster start
   

   crm status --full
   

As the cluster resource SAPHana_${SID}_${INSTNO} is configured with AUTOMATED_REGISTER=true, SAP HANA restarts when NODE2 rejoins the cluster and the former primary reregisters as a secondary.

Test 3 - Description

Simulate a crash of the secondary HANA database.

Test 3 - Prerequisites

See the following prerequisites before you perform Test 3.

• A functional two-node SLES HA Extension cluster for HANA system replication.
• Both nodes are active.
• Cluster is started on NODE1 and NODE2.
• Cluster Resource SAPHana_${SID}_${INSTNO} is configured with AUTOMATED_REGISTER=true.
• Check SAP HANA System Replication status.
  • Primary SAP HANA database is running on NODE1.
  • Secondary SAP HANA database is running on NODE2.
  • HANA System Replication is activated and in sync.

Test 3 - Test Procedure

Crash SAP HANA secondary by sending a SIGKILL signal as the user ${sid}adm.

On NODE2, run the following command.

sudo -i -u ${sid}adm -- HDB kill-9

Test 3 - Expected behavior

You can expect the following behavior from the test.

• SAP HANA secondary on NODE2 crashes.
• The cluster detects the stopped secondary HANA database and marks the resource as failed.
• The cluster restarts the secondary HANA database.
• The cluster detects that the system replication is in sync again.

Test 3 - Recovery procedure

Use the following information to recover from Test 2.

1. Wait until the secondary HANA instance starts and syncs again (SOK), then cleanup the failed resource actions as shown in crm status.

2. On NODE2, run the following command.

   crm resource refresh SAPHana_${SID}_${INSTNO}
   

   crm status --full
   

Test 4 - Description

Use cluster commands to move the primary instance to the other node for maintenance purposes.

Test 4 - Prerequisites

See the following prerequisites before you perform Test 4.

• A functional two-node SLES HA Extension cluster for HANA system replication.
• Both nodes are active.
• Cluster is started on NODE1 and NODE2.
• Cluster Resource SAPHana_${SID}_${INSTNO} is configured with AUTOMATED_REGISTER=true.
• Check SAP HANA System Replication status:
  • Primary SAP HANA database is running on NODE1
  • Secondary SAP HANA database is running on NODE2
  • HANA System Replication is activated and in sync

Test 4 - Test procedure

Move SAP HANA primary to other node by using the crm resource move command.

On NODE1, run the following command.

crm resource move SAPHana_${SID}_${INSTNO}-clone

Test 4 - Expected behavior

You can expect the following behavior from the test.

• The cluster creates location constraints to move the resource.
• The cluster triggers a takeover to the secondary HANA database.
• If an application, such as SAP NetWeaver, is connected to a tenant database of SAP HANA, the application automatically reconnects to the new primary.

Test 4 - Recovery procedure

Use the following information to recover from Test 2.

The automatically created location constraints must be removed to allow automatic failover in the future.

Wait until the primary HANA instance is active and remove the constraints.

The cluster registers and starts the HANA database as a new secondary instance.

On NODE1, run the following command.

crm constraint

crm resource clear SAPHana_${SID}_${INSTNO}-clone

crm constraint

crm status --full