Deploying a dedicated private network subnet

Create a dedicated private network subnet for each SAP system that you install in the workspace. For detailed steps, see Configuring a private network subnet.

Planning the Power Virtual Server instances

Select the appropriate compute profiles for the Power Virtual Server instances. For information about available profiles, see SAP HANA certified instances on Power Virtual Server and SAP Application Server certified instances on Power Virtual Server.

You can choose one of the following options for the OS image subscription:

IBM provided subscription

IBM Cloud® offers full subscriptions for boot images for AIX, IBM i, and Linux operating systems. Choose Linux for SAP HANA, or Linux for SAP Netweaver.

Client supplied subscription ("Bring Your Own License" - BYOL)

If you use a client supplied subscription, either:

• Import your own image as a custom image for deployment, or
• Select an image from the client supplied subscription section of the IBM stock images (identified with the BYOL suffix) for virtual server instance deployment.

See more details about the operating system versions in OS for IBM Power Virtual Servers. See Using RHEL within the Power Virtual Server and Using SLES within IBM Power Virtual Server for more details about the subscriptions for RHEL and SLES.

The following steps describe deploying virtual server instances by using an IBM provided subscription.

IBM Cloud® for SAP provides SAP-certified infrastructure to run SAP workloads. The Boot image selection offers the following operating systems with an IBM provided subscription for both RHEL and SLES:

• Linux for SAP (HANA) for SAP HANA systems. The image names have the suffix -SAP.
• Linux for SAP (NetWeaver) for SAP application servers. The image names have the suffix -SAP-NETWEAVER.

Pay attention to the choice of operating system. Linux is different than Linux for SAP (HANA) and Linux for SAP (NetWeaver). Plain Linux images do not include the specific extensions that are required to run SAP systems for production environments.

Deploying an Power Virtual Server instance for the SAP HANA system

Deploy a virtual server instance for the SAP HANA system.

1. Select a workspace from the list of Workspaces on the left navigation page.
2. Click View virtual servers on the right page. The list of virtual server instances is shown.
3. Click Create instance to create a new virtual server instance.

Complete the fields in the General section.

Expand the Advanced Configurations section, and keep Virtual server pinning disabled.

Click Continue.

Complete the fields in the Boot image section.

Advanced configurations allow influencing the profile selection for subsequent steps. When you select Custom SAP (HANA) Deployment, you can modify parameters such as CPU cores, memory size, and storage tier. Custom profiles are supported only on Power9 systems, and are intended exclusively for nonproduction environments. When you select SAP RISE Deployment, you can choose from predefined profiles that are aligned with the RISE with SAP on Power Virtual Server offering.

Click Continue.

Complete the fields in the Profile section.

Click Continue.

Complete the fields in the Storage volumes section.

For the SAP HANA system, you create storage volumes that belong to different storage tiers 'Tier 1' and 'Tier 3'. You cannot mix storage tiers during the instance creation process, so you need to create and attach the storage volumes later. Do not specify the storage volumes now.

Click Continue.

Complete the fields in the Networking section.

• Do not specify any Public networks.
• Click Attach existing network to attach the subnet.
  • Select the name of the subnet from the list of existing subnets.
  • Select Manually specify an IP address from IP range, and enter the IP address <HANA_PVS_IP> that you reserved before in your DNS setup. If you select Automatically assign an IP address from IP range, then the IP address is assigned dynamically. In that case, you must retrieve the <HANA_PVS_IP> after deployment and update your DNS configuration by using this IP address for the hostname of the instance.
  • Click Attach

Click Finish, accept the terms and conditions, and then click Create.

It takes some time for the virtual server instance for SAP HANA to become available.

Wait for the instance to become active, then log in to it by using the VPC access host. Use the following SSH command to login as root user to the virtual server instance:

ssh -A -o ServerAliveInterval=60 -o ServerAliveCountMax=600 -o ProxyCommand="ssh -W %h:%p root@<ACCESS_HOST_FLOATING_IP>" root@<HANA_PVS_IP>

<ACCESS_HOST_FLOATING_IP> is the public IP address of the jump host. <HANA_PVS_IP> is the IP address of the virtual server instance.

An alternative way to connect to the VPC instance is to use a VPN server. For more information, see the tutorial Connect by using a client-to-site VPN.

Deploying a Power Virtual Server instance for SAP NetWeaver

To deploy a virtual server instance for SAP NetWeaver, go to your Workspaces and create a new instance as described in Configuring a Power Virtual Server instance. Use the information in Deploying SAP HANA on Power Virtual Server to complete the configuration for General, Boot Image, Profile, Storage volumes, and Networking sections. For the Boot image section, specify the IBM provided subscription 'Linux for SAP (NetWeaver)' in the selection.

Wait for the instance to become active. Then log on to the instance by using the following SSH command:

ssh -A -o ServerAliveInterval=60 -o ServerAliveCountMax=600 -o ProxyCommand="ssh -W %h:%p root@<ACCESS_HOST_FLOATING_IP>" root@<NETWEAVER_PVS_IP>

<ACCESS_HOST_FLOATING_IP> is the public IP address of the jump host. <NETWEAVER_PVS_IP> is the IP address of the virtual server instance.

Alternatively, connect by using a VPN client, as described in the tutorial Connect by using a client-to-site VPN.

Creating storage volumes

To plan storage volumes for SAP HANA, refer to the tables provided in SAP HANA certified instances on IBM Power Virtual Server. Locate the compute profile applicable to your deployment and review the Log File System, Data File System, and Shared File System tabs. These tabs specify the required quantity of storage volumes, the storage tier, and the corresponding volume sizes for the file systems /hana/log, /hana/data, and /hana/shared.

Create Power Virtual Server storage volumes to host the /hana/data and /hana/log file systems. For a single-host SAP HANA installation, allocate a storage volume for /hana/shared as well. For a SAP HANA multi-host installation, refer to the SAP HANA scale-out section.

To plan storage volumes for the SAP application server, refer to the table in SAP Application Server certified instances on IBM Power Virtual Server. Use the SAP Application Server Filesystem tab to determine the storage tier, and the size of the storage volume for the /usr/sap file system.

Modify the Power Virtual Server virtual server instances and attach the volumes to the virtual server instance for SAP HANA or the SAP application server. See Managing your storage volumes.

Make sure that the "Shareable" option remains "off" for all the block storage volumes.

Continue with the configuration of virtual server instances for SAP applications by choosing one of the following options:

• Configuring Power Virtual Server instances manually.

• Configuring Power Virtual Server instances by using Ansible automation playbooks.

Distributed SAP system configuration

A distributed SAP system installation requires file systems that are shared between the multiple virtual server instances. IBM Cloud File Storage for VPC provides NFS based file storage services. See the detailed information in the document File Storage for VPC.

See Accessing file storage shares for VPC for more details about creating, accessing, and mounting the file shares on the virtual server instances.

SAP HANA multiple-host system configuration

A SAP HANA multiple-host system, also known as SAP HANA scale-out system, is a system with more than one host, which can be configured as active worker hosts or idle standby hosts. See detailed information about the Scaling SAP HANA.

Install all hosts of the SAP HANA system by using the same operating system version of Red Hat Enterprise Linux (RHEL) to ensure seamless operation and compatibility.

Ensure that the hostnames of the virtual server instances comply with SAP HANA naming conventions to avoid any configuration issues.

Use Power Virtual Server storage volumes for the /hana/data and /hana/log file systems. Do not share the storage volumes between different virtual server instances to maintain data integrity and performance. See the General storage configurations on IBM Power Virtual Server Infrastructure for more detailed explanation about storage configurations of different SAP workloads on IBM® Power® Virtual Server.

In a multiple-host SAP HANA installation, the shared directory /hana/shared has to be mounted by all hosts that are part of the system. Follow the instructions in Accessing VPC zonal file storage shares from IBM Power Virtual Server Instances to provision a file share.

A distributed SAP system installation requires also file systems that are shared between the virtual server instances of the system, for example the file systems /sapmnt/<SID> or /usr/sap/trans. Use the same procedure for the shared file systems of a SAP system installation.

To successfully mount the NFS share on the SAP HANA nodes, add the following line to the /etc/fstab file:

<IP-of-NFS-service>:/<mount_target> /hana/shared nfs vers=4,rw,sec=sys,hard,rsize=65536,wsize=65536,namlen=255

Verify that the /hana/shared directory is mounted on all nodes before starting with the SAP HANA installation. Run following command to mount the file share.

mount /hana/shared

Configuring a proxy endpoint

The virtual server instances are not attached to a public network, and do not have direct access to the internet. The SQUID proxy server in IBM VPC acts as an intermediary between the virtual server instance and the internet. In a Linux environment, environment variables such as http_proxy, https_proxy, and no_proxy configure the proxy settings.

Run the following commands to export a SQUID_PROXY_SERVER proxy endpoint:

export http_proxy=http://<SQUID_PROXY_SERVER>:3128
export https_proxy=http://<SQUID_PROXY_SERVER>:3128
export HTTP_PROXY=http://<SQUID_PROXY_SERVER>:3128
export HTTPS_PROXY=http://<SQUID_PROXY_SERVER>:3128

The commands set the environment variables temporarily, they do not persist across multiple terminal sessions, and do not survive a restart. To keep the environment variables persistent, add these entries to file /etc/bash.bashrc (on SLES) or /etc/bashrc (on RHEL).

Checking IBM subscription for an operating system image

To help ensure that the OS subscription is set up correctly, check for a subscription by running:

RHEL:

subscription-manager release

subscription-manager list

yum repolist

Optional:

subscription-manager status

SLES:

SUSEConnect --status

RHEL and SLES:

Enable the IBM Power Tools repository by using:

/opt/ibm/lop/configure

Check whether the repo is enabled:

RHEL:

yum repolist enabled

SLES:

zypper lr

Run a system update. This process also updates the installed IBM Power Tools.

RHEL:

yum -y update

SLES:

zypper update -y

Configuring a Domain Name System (DNS) client

The DNS servers of IBM Cloud are configured as default, you do not need to modify the DNS configuration for them. If you create a private DNS server for your environment, configure it after the virtual server instance is created.

Configuring a Network Time Protocol client

Install the chrony package by using the following command.

RHEL:

yum install -y chrony

SLES:

zypper install -y chrony

Check the status of chrony service:

systemctl status chronyd

Adapt the /etc/chrony.conf configuration. In Power Virtual Server workspaces enabled for Power Edge Router (PER), the virtual server instances can directly access IBM Cloud NTP server time.adn.networklayer.com.

See the following sample configuration for /etc/chrony.conf:

server time.adn.networklayer.com iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
logdir /var/log/chrony

Then, restart the chronyd service on the node.

systemctl restart chronyd.service

Configuring a Network File System (NFS) client

For the shared SAP file systems, create file storage for VPC and mount it on the virtual server instances. Use the following steps to manually configure the NFS client.

1. Install the nfs-client package and enable the NFS client by using the systemctl command.

   • RHEL:

   yum install -y nfs-utils
   

   • SLES:

   zypper install -y nfs-utils
   

2. To enable NFS client service, use the following command.

   systemctl start nfs-client
   

3. After the NFS client service starts, you can mount the shared NFS directory by using the mount command.

   mount -t nfs4 -o sec=sys <NFS_SERVER_IP>:<NFS_DIRECTORY_PATH>
   

   NFS_SERVER_IP is the IP address of the NFS server. NFS_DIRECTORY_PATH is the path of the NFS file storage share.

Creating file systems manually

For an ABAP application server, create a file system /usr/sap to store instance-specific data.

To install SAP HANA, three file systems are required:

• HANA Data
• HANA Log
• HANA Shared

By default, these file systems are named:

• /hana/data
• /hana/log
• /hana/shared

However, you can customize the names. You might configure other file systems, such as for the /usr/sap directory.

Stripe the /hana/data and /hana/log file systems across four or eight volumes. The /hana/shared file system and all other file systems are nonstriped, single-disk file systems.

Extra file systems can be created for other purposes.

/usr/bin/rescan-scsi-bus.sh -a -c -v

Newly discovered disks are listed with their details.

Identify the World Wide Names (WWNs) of the storage volumes by running the multipath command.

multipath -ll

The output of the multipath -ll command details the World Wide Names of the volumes. The WWNs are also visible on the IBM Cloud console. Log on to the IBM Cloud® console and go to the Storage volumes, select the workspace, and then select the virtual server instance for which the storage volumes are defined.

Notice that the WWNs on the IBM Cloud® are in uppercase, while they are in lowercase in the output of the multipath -ll command. In the output of the multipath -ll command, a prefix 3 is added at the beginning of each WWN.

The example illustrates the steps to create a /hana/data file system. Repeat similar steps for /hana/log, /hana/shared, and all the other file systems.

For example, a storage volume that is named dm-6 has the WWN 6005076813810214200000000000a7a8 and a size of 60G.

multipath -ll
36005076813810214200000000000a72d dm-0 IBM,2145
size=100G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 1:0:1:0 sdj  8:144  active ready running
| |- 2:0:1:0 sdab 65:176 active ready running
| |- 3:0:1:0 sdat 66:208 active ready running
| `- 4:0:1:0 sdbl 67:240 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  |- 1:0:0:0 sda  8:0    active ready running
  |- 2:0:0:0 sds  65:32  active ready running
  |- 3:0:0:0 sdak 66:64  active ready running
  `- 4:0:0:0 sdbc 67:96  active ready running
36005076813810214200000000000a7a8 dm-6 IBM,2145
size=60G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 1:0:1:4 sdn  8:208  active ready running
| |- 2:0:1:4 sdaf 65:240 active ready running
| |- 3:0:1:4 sdax 67:16  active ready running
| `- 4:0:1:4 sdbp 68:48  active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  |- 1:0:0:4 sde  8:64   active ready running
  |- 2:0:0:4 sdw  65:96  active ready running
  |- 3:0:0:4 sdao 66:128 active ready running
  `- 4:0:0:4 sdbg 67:160 active ready running
...

One or more device names need to be extracted to create the physical volume devices, the volume group, and the logical volume.

In the example, multipath aliases are used. Run the following commands to create the /hana/data file system.

Export the following variables:

export PV_SIZE="60G"

export LV_NAME="hana_data_lv"

export VG_NAME="hana_data_vg"

export MOUNT="/hana/data"

If you are using multipath aliases, use the following commands:

DEVICES=$(multipath -ll | grep -B 1 ${PV_SIZE} | grep dm- | awk '{print "/dev/"$2}' | tr '\n' ' ')

STRIPES=$(multipath -ll | grep -B 1 ${PV_SIZE} | grep dm- | awk '{print "/dev/"$2}' | wc | awk '{print $1}')

pvcreate ${DEVICES}

vgcreate ${VG_NAME} ${DEVICES}

lvcreate -i${STRIPES} -I64 -l100%VG -n ${LV_NAME} ${VG_NAME}

mkfs.xfs /dev/mapper/${VG_NAME}-${LV_NAME}

mkdir -p ${MOUNT}

mount /dev/mapper/${VG_NAME}-${LV_NAME} ${MOUNT}

Run the same commands to create storage volumes for /hana/log and /hana/shared. Change LV_NAME to hana_log_lv, VG_NAME to hana_log_vg and MOUNT to /hana/log.

Export these variables:

export LV_NAME=hana_log_lv

export VG_NAME=hana_log_vg

export MOUNT=/hana/log

Use the same approach for /hana/shared.

You must have different storage volume sizes and so different values for the PV_SIZE for /hana/data, /hana/log, and /hana/shared. Otherwise, the approach fails.

If you don't use multipath aliases, adjust the DEVICES=$() command that is used to identify the disk devices. Use the following DEVICES=$(multipath -ll | grep -B 1 ${PV_SIZE} | grep dm- | awk '{print "/dev/"$2}' | tr '\n' ' ') instead.

Checking the storage volumes

After you create the storage volumes, verify that they were created correctly by running the following commands:

lvscan

The command output shows the status of the created logical volumes (LV). Verify that the status is active.

Verify that the file system is mounted:

mount | grep hana

See a sample output of the command:

/dev/mapper/hana_shared_vg-hana_shared_lv on /hana/shared type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/mapper/hana_log_vg-hana_log_lv on /hana/log type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=64k,sunit=128,swidth=256,noquota)
/dev/mapper/hana_data_vg-hana_data_lv on /hana/data type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=64k,sunit=128,swidth=512,noquota)

Add the file systems /hana/data, /hana/log, and /hana/shared to the file systems table /etc/fstab, unless an entry for each of them exists.

cat /etc/fstab

Sample output:

...
/dev/mapper/datavg-datalv /hana/data xfs defaults,nofail 0 0
/dev/mapper/logvg-loglv /hana/log xfs defaults,nofail 0 0
/dev/mapper/sharedvg-sharedlv /hana/shared xfs defaults,nofail 0 0

On RHEL 8.8 and RHEL 8.10 run the following command as a workaround to prevent the logical volumes remain closed after a restart:

dracut --force --verbose

It might be necessary to restart the operating system. After the restart, check that the LV setup is correct.

Continue with Preparing for SAP software installation.

Configuring network management services

The configure_network_management_services Ansible role installs and configures a proxy endpoint, NTP, DNS, and NFS network services in a virtual server instance. For more details, see the configure_network_management_services on Ansible Galaxy.

Configure the variable file in playbooks/vars/sample-variables-configure-network-services-client.yml first to run an Ansible playbook with the configure_network_management_services role. Update the sample with your own values. See an example how it might look like:

client_config:
  squid:
    enable: true
    squid_server_ip_port: "<SQUID_PROXY_SERVER_IP>:3128"
    no_proxy_hosts: "161.0.0.0/8"
  ntp:
    enable: true
    ntp_server_ip: "time.adn.networklayer.com"
  nfs:
    enable: true
    nfs_server_path: "<NFS_SERVER_IP>:/nfs"
    nfs_client_path: "/nfs"
    opts: sec=sys,nfsvers=4.1,nofail
    fstype: nfs4
  dns:
    enable: true
    dns_server_ip: "<DNS_SERVER_IP>"

A detailed description of the parameters SQUID_PROXY_SERVER_IP, NTP_SERVER_IP, NFS_SERVER_IP, DNS_SERVER_IP, that are used for the preceding example, can be found in the section Edit parameters in the configuration file.

Then, run the following command:

ansible-playbook --connection=local -i "localhost," playbooks/sample-configure-network-services-client.yml

Creating the file systems

The Ansible role, powervs_storage_and_swap_setup is used to create file systems for both SAP HANA, and the SAP application server. This role runs tasks such as creating file systems for /hana/data, /hana/log and /hana/shared, mounting these file systems on provided mount points, adding an entry to /etc/fstab for automount on restart, and other tasks. For more details, see the description of powervs_storage_and_swap_setup on Ansible Galaxy.

Identify the WWNs as described in the Manually creating file systems section first to run a playbook for storage setup. Add the WWNs to the sample-variables-powervs-storage-setup.yml variable file. For an explanation of the variable file, see the Edit parameters in the configuration file section.

Next, run an Ansible playbook to create the file systems as:

ansible-playbook --connection=local -i "localhost," playbooks/sample-powervs-storage-setup.yml

To verify that all file systems are created and mounted correctly, use the Checking of storage volumes section. When /hana/data, /hana/log, and /hana/shared are mounted correctly then continue with Preparing for SAP installation.

Configuring SLES for SAP applications

SLES

Use the saptune tool to apply the recommended operating system settings for SLES to the SAP HANA system (HANA) or the application server (NETWEAVER). For more information about saptune, see SAP Note 1275776 - Linux: Preparing SLES for SAP environments.

1. Verify that the saptune package is installed.

   zypper info saptune
   

2. Verify that the saptune version is at least 3.

   saptune version
   

3. List all available solutions. Numbered entries represent integrated SAP Notes for each of the solutions.

   saptune solution list
   

4. Get an overview of saptune options.

   saptune --help
   

5. Enable and start the saptune.service. The command disables sapconf and tuned, which are no longer used.

   saptune service takeover
   

6. Simulate the changes first before they are applied (optional).

   On the virtual server instance for the SAP HANA system:

   saptune solution simulate HANA
   

   For SAP NetWeaver:

   saptune solution simulate NETWEAVER
   

7. Apply the saptune solution.

   On the virtual server instance for the SAP HANA system:

   saptune solution apply HANA
   

   On the virtual server instance for the application server:

   saptune solution apply NETWEAVER
   

8. Check the saptune status.

   saptune status
   

9. Verify that saptune is set up correctly.

   saptune check
   

Configuring RHEL for SAP applications

RHEL

RHEL System Roles for SAP are a collection of Ansible roles that help you configure a RHEL system for installing SAP HANA or SAP NetWeaver. Red Hat distributes and maintains the Ansible roles for SAP configuration, so the tasks and required parameters can differ depending on the version of the rhel-system-roles-sap package. The RHEL image that is provided by IBM includes the Ansible execution engine, SAP-related system roles, and the Ansible execution files.

The RHEL system roles for preparing the operating system settings for SAP HANA or SAP NetWeaver are available in directory /root.

Use the following command to prepare the virtual server instance for SAP HANA workload.

ansible-playbook -i /root/inventory /root/sap-hana.yml

Use the following command to prepare the virtual server instance for SAP NetWeaver workload.

ansible-playbook -i /root/inventory /root/sap-netweaver.yml

For more information about customizing the operating system, see the following documentation.

• SAP Note 2772999 "Red Hat Enterprise Linux 8.x: Installation and Configuration"
• SAP Note 2777782 "SAP HANA DB: Recommended OS Settings for RHEL8"
• SAP Note 3108316 "Red Hat Enterprise Linux 9.x: Installation and Configuration"
• SAP Note 3108302 "SAP HANA DB: Recommended OS Settings for RHEL9"
• SAP Note 2382421 "Optimizing the Network Configuration on HANA- and OS-Level"
• Red Hat Enterprise Linux System Roles for SAP

Configuring jumbo frames

Enable jumbo frames by setting MTU=9000 on both RHEL (nmcli) and SLES (wicked).

RHEL

Identify the connection name for the network manager. The example shows the commands for connection System env2.

• Verify the current MTU size for the connection

   nmcli con show "System env2" | grep -E "802-3-ethernet.mtu|ethtool.feature-tso"
   802-3-ethernet.mtu:                     1500
  

• Enable jumbo frames by altering the MTU size to 9000. Activate TCP segmentation offload (TSO).

  nmcli connection modify "System env2" \
    802-3-ethernet.mtu 9000 \
    ethtool.feature-tso on
  

• Verify the new settings for the connection

  nmcli con show "System env2" | grep -E "802-3-ethernet.mtu|ethtool.feature-tso"
  802-3-ethernet.mtu:                     9000
  ethtool.feature-tso:                    on
  

SLES

Identify the device name for the network adapter. The example shows the commands for device name env2.

• Edit file /etc/sysconfig/network/ifcfg-env2, and check that it contains the parameters

  MTU='9000'
  ETHTOOL_OPTS='-K env2 tso on'
  

• Afterward, restart the interface

  wicked ifdown env2 && wicked ifup env2
  

Checking the nonuniform memory access layout

Check that the CPU and memory placement is optimized for SAP HANA by running the chk_numa_lpm.py script.

The script runs the following checks:

• NUMA Layout Validation Verifies the nonuniform memory access (NUMA) configuration according to SAP HANA requirements.

  • Verifies that for each CPU core local memory is available.
  • Verifies that the memory distribution across the cores does not exceed a 50% variance.
  • Generates error messages if a core lacks memory, and warning messages if the memory distribution exceeds the margin.

• Live Partition Mobility (LPM) Check Detects whether an LPM operation occurred since the last system restart. An LPM operation can alter the NUMA layout from its boot-time configuration.

  • Searches system logs for the most recent LPM event.
  • Issues a warning if an LPM operation is detected.

Steps to run the Check

1. Review the details in SAP Note 2923962.

2. Download the chk_numa_lpm.py script that is attached to the SAP Note and copy it to your virtual server instance.

3. Set executable permissions for the script:

   chmod +x ./chk_numa_lpm.py
   

4. Run the script:

   ./chk_numa_lpm.py