Deploying SAP applications on Power Virtual Server

Deploying an Power Virtual Server instance for SAP HANA

To deploy an Power Virtual Server instance for SAP HANA database, select a previously created workspace from the list of Workspaces on the left navigation page. Then click Virtual server instances on the left page. To create a new instance, click Create instance on the right side. Complete the following steps:

1. In the General section, make the following selections:

   Table 1. SAP HANA general selections

   Field	Details
   Instance name	Enter a unique name for the instance.
   Number of instances	Enter '1'.
   Add to server placement group	Optional and can be skipped.
   Add to a shared processor pool	Optional and can be skipped.
   Virtual server pinning	Optional, can be set to None as default selection.
   SSH key	Choose an existing SSH key that was created previously.

2. In the Boot image section, make the following selections:

   Table 2. SAP HANA boot image selections

   Field	Details
   Operating system	Select the IBM provided subscription 'Linux for SAP (HANA)'. See the explanation of differences in the section Deploying Power Virtual Server instances for SAP on IBM Cloud®
   Image	Select an operating system and a version. Make sure that you use the same operating system and version for all deployments.
   Tier	Choose a tier that best meets your needs, for more information, see Storage tiers.
   Storage pool	Select the storage pool that you need.
   Advanced configurations	Enable toggle buttons to support more settings.

3. In the Profile section, make the following selection:

   • Select a profile that meets your needs. For more information, see SAP HANA profiles.

4. In the Storage volumes section, make the following selection:

   • For SAP HANA, the attached volumes are on different storage tiers 'Tier 1' and 'Tier 3'. You can't mix storage tiers during the instance creation process, so you need to attach storage volumes later. Leave this list empty.

5. In the Networking section, make the following selections:

   • Leave Public networks deactivated.
   • Attach both private networks (management and backup) and any separate private networks. Enter the IP addresses as entered in the DNS configuration for the corresponding host names. If the IP addresses are assigned dynamically, you need to adapt the DNS entries for the host names of this system.

It takes some time for the Power Virtual Server instance for SAP HANA to become available. When the deployment is completed, you can log in to the instance via the VPC access host. Use the SSH command below to login as root user to the virtual server instance:

ssh -A -o ServerAliveInterval=60 -o ServerAliveCountMax=600 -o ProxyCommand="ssh -W %h:%p root@<ACCESS_HOST_FLOATING_IP>" root@<HANA_PVS_IP>

ACCESS_HOST_FLOATING_IP is a public IP address of a jump host, HANA_PVS_IP is the virtual server instance IP address in the management subnet.

An alternative solution for connecting to the VPC instance is to use a VPN server. See more details in the tutorial Connect by using a client-to-site VPN.

Deploying an Power Virtual Server instance for SAP NetWeaver

To deploy an Power Virtual Server instance for SAP NetWeaver, go to your workspace for Power Virtual Server and create an Power Virtual Server instance as described in Configuring a Power Virtual Server instance. Use the information in Deploying SAP HANA on Power Virtual Server to complete the configuration for General, Profile, Storage volumes, Networking sections. For the Boot image section, specify the IBM provided subscription 'Linux for SAP (NetWeaver)' selection.

After creating an Power Virtual Server instance for SAP NetWeaver, wait for the instance to become active. Then log on to the SAP NetWeaver instance by using the following SSH command:

ssh -A -o ServerAliveInterval=60 -o ServerAliveCountMax=600 -o ProxyCommand="ssh -W %h:%p root@<ACCESS_HOST_FLOATING_IP>" root@<NETWEAVER_PVS_IP>

ACCESS_HOST_FLOATING_IP is a public IP address of a jump host, NETWEAVER_PVS_IP is the virtual server instance IP address in the management subnet.

Alternatively, connect by using a VPN client, as described in the tutorial Connect by using a client-to-site VPN.

Deploying an optional Power Virtual Server instance for SAP shared file systems

Each deployment of an SAP NetWeaver-based application server, contains file systems that are shared between multiple application server instances. It is a good practice to set up a separate virtual service instance for the shared file systems. A single Power Virtual Server shared file systems instance can be used by multiple SAP systems. Your security requirements determine how many Power Virtual Server shared file systems instances you might need.

To deploy an Power Virtual Server instance for shared file systems, use the Workspace where the SAP HANA and SAP NetWeaver instances were created. Create a shared file system as described in Configuring a Power Virtual Server instance.

After deploying the Power Virtual Server instance for shared file systems, you can use it as Network File System (NFS) storage, where an NFS server will run. How to set up the NFS server is described in the Deploying an NFS server for RHEL and Sharing file systems with NFS for SLES.

Creating extra storage volumes for SAP HANA Power Virtual Server

Modify Power Virtual Server SAP HANA instance and attach extra storage volumes as described in Managing your storage volumes.

In accordance with the SAP sizing guidelines, you should add the following volumes:

• A storage volume for SAP HANA shared file system with a size of MIN (1 x RAM; 1 TB). Storage tier "Tier 3" is sufficient. The "Shareable" switch should remain "off".

• Four or more equal-size storage volumes for SAP HANA log file system with size MIN (1/2xRAM; 512 GB). Divide the file system size by the number of volumes to determine the size for each storage volume. Select "Tier 1", "Tier 0", or "Fixed IOPS Tier"  as the storage tier. Ensure that your configuration provides a total of at least 12,000 IOPS (input/output operations per second). The "Shareable" flag should remain "off".

• Four or more equal-sized storage volumes for the SAP HANA data file system with a size of 1.5 times the RAM. Divide the file system size by the number of volumes to determine the size for each storage volume. Select "Tier 1" or "Tier 0" as the storage tier. Ensure that your configuration provides a total of at least 8,000 IOPS (input/output operations per second). The "Shareable" flag should remain "off".

• An extra storage volume for other data (such as "/usr/sap" file system). The Storage tier "Tier 3" is sufficient. The "Shareable" flag should remain "off".

• You can add more volumes for backup or export.

For more information, see Block Storage for VPC profiles.

Continue with the configuration of Power Virtual Server instances for SAP applications, which can be done by manual or automated setup. You can choose one of the following setup sections:

• Configuring Power Virtual Server instances manually.

• Configuring Power Virtual Server instances by using Ansible automation playbooks.

Checking IBM subscription for an operating system image

To ensure that the OS subscription is set up correctly, check a subscription manager by running:

RHEL:

subscription-manager release

subscription-manager list

SLES:

yum repolist

Optional:

subscription-manager status

Then run a system update.

RHEL:

yum -y update

SLES:

zypper update -y

Configuring a proxy endpoint

To use proxy and cache services for HTTP, FTP, and other popular network protocols, you must to export a proxy server endpoint. Run the following commands to export a SQUID_PROXY_SERVER proxy endpoint:

export http_proxy=http://<SQUID_PROXY_SERVER>:3128

export https_proxy=http://<SQUID_PROXY_SERVER>:3128

export HTTP_PROXY=http://<SQUID_PROXY_SERVER>:3128

export HTTPS_PROXY=http://<SQUID_PROXY_SERVER>:3128

To keep these exported variables persistent across multiple sessions, these entries must be added to /etc/bash.bashrc (on SLES) or /etc/bashrc (on RHEL). These files ensure that new sessions use the exported variables as environment variables.

Configuring a NTP client

If an NTP server is already configured, you must install the chrony package by using the following command.

RHEL:

yum install -y chrony

SLES:

zypper install -y chrony

Check the status of chrony service:

systemctl status chronyd

Adapt the chrony.conf configuration according to the description in the section Setting up chrony for a system in an isolated network.

The following is a sample configuration for `chrony.conf':

server <NTP_SERVER_IP> iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
logdir /var/log/chrony

Then, restart the chronyd service on the node.

systemctl restart chronyd.service

Configuring a DNS client

A DNS server should be configured after the Power Virtual Server instance is created. The /etc/resolv.conf file contains all the DNS server configurations, there is no need to modify this file.

Configuring an NFS client

Use the following steps to manually configure an NFS client.

1. Install the nfs-client package and enable the NFS client by using the systemctl command.

   • RHEL:

   yum install -y nfs-utils
   

   • SLES:

   zypper install -y nfs-utils
   

2. To enable NFS client service, use the following command.

   systemctl start nfs-client
   

3. After the NFS client service starts, you can mount the shared NFS directory by using the mount command.

   mount -t nfs4 -o sec=sys,nfsvers=4.1 <NFS_SERVER_IP>:<NFS_DIRECTORY_PATH>
   

   NFS_SERVER_IP is an IP address of a loadbalancer, NFS_DIRECTORY_PATH is the path of the NFS file storage share.

Creating file systems manually

For shared SAP file systems, you must create a file system to store SAP data and distribute them to all SAP instances. You can use extra file systems for other purposes.

For an SAP NetWeaver instance, you must create a file system to store instance-specific data.

Three file systems are required to install SAP HANA: data, log, and shared. According to the default installation catalog, these file systems are /hana/data, /hana/log and /hana/shared, but you can customize the file system names. You might also need file systems for other purposes (/usr/sap directory). The /hana/data and /hana/log file systems are striped across four or eight disks, depending on the number of volumes that you created. /hana/shared and all other file systems are non-striped 1-disk file systems.

Perform a disk discovery by running the following script:

/usr/bin/rescan-scsi-bus.sh -a -c -v

Newly discovered disks are listed with their details.

To identify the Word Wide Names (WWNs) that will be used to set up storage volumes, run:

multipath -ll

The output of the multipath -ll command corresponds to the World Wide Names that are also listed on the IBM Cloud® console. Log on to the IBM Cloud® console and go to the Storage volumes, select a workspace and a Virtual server instance where storage volumes are defined. You might notice that the WWNs on the IBM Cloud® are in uppercase, while they are in lowercase in the operating system.

To create a file system, use the /hana/data example that is decribed below. The same procedure is repeated for /hana/log and /hana/shared and all other file systems. For example, a storage volume named dm-6 has a WWN of 6005076813810214200000000000a7a8 and a size of 60G. The 3 at the beginning is ignored. The device name is needed to create a logical volume and a volume group.

multipath -ll
36005076813810214200000000000a72d dm-0 IBM,2145
size=100G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 1:0:1:0 sdj  8:144  active ready running
| |- 2:0:1:0 sdab 65:176 active ready running
| |- 3:0:1:0 sdat 66:208 active ready running
| `- 4:0:1:0 sdbl 67:240 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  |- 1:0:0:0 sda  8:0    active ready running
  |- 2:0:0:0 sds  65:32  active ready running
  |- 3:0:0:0 sdak 66:64  active ready running
  `- 4:0:0:0 sdbc 67:96  active ready running
36005076813810214200000000000a7a8 dm-6 IBM,2145
size=60G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 1:0:1:4 sdn  8:208  active ready running
| |- 2:0:1:4 sdaf 65:240 active ready running
| |- 3:0:1:4 sdax 67:16  active ready running
| `- 4:0:1:4 sdbp 68:48  active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  |- 1:0:0:4 sde  8:64   active ready running
  |- 2:0:0:4 sdw  65:96  active ready running
  |- 3:0:0:4 sdao 66:128 active ready running
  `- 4:0:0:4 sdbg 67:160 active ready running
...

Multipath aliases are used for this setup. Run the following commands to create the /hana/data file system.

Export the following variables:

export pv_size=60G

export lv_name=hana_data_lv

export vg_name=hana_data_vg

export mount=/hana/data

If you are using multipath aliases, use the following commands:

devices=$(multipath -ll | grep -B 1 $pv_size | grep dm- | awk '{print "/dev/"$2}' | tr '\n' ' ')

stripes=$(multipath -ll | grep -B 1 $pv_size | grep dm- | awk '{print "/dev/"$2}' | wc | awk '{print $1}')

pvcreate $devices

vgcreate ${vg_name} ${devices}

lvcreate -i${stripes} -I64 -l100%VG -n ${lv_name} ${vg_name}

mkfs.xfs /dev/mapper/${vg_name}-${lv_name}

mkdir -p ${mount}

Run the same commands to create storage volumes for /hana/log and /hana/shared. Change lv_name to hana_log_lv, vg_name to hana_log_vg and mount to /hana/log.

Export these variables:

export lv_name=hana_log_lv

export vg_name=hana_log_vg

export mount=/hana/log

Use the same approach for /hana/shared.

Make sure that the variable pv_size has different values for /hana/data, /hana/log and /hana/shared, before running the export command, otherwise the whole setup will not work.

If you don't use multipath aliases, replace the line (starting with devices=$() that is used to identify devices. Instead, use following line: devices=$(multipath -ll | grep -B 1 $pv_size | grep dm- | awk '{print "/dev/"$2}' | tr '\n' ' ')

Checking the storage volumes

After you create the storage volumes, verify that they were created correctly by running the following commands:

lvscan

The command output shows the status of the created logical volumes (LV). The status should be active.

To check if the file system is mounted:

mount | grep hana

The following is a sample output of this command:

/dev/mapper/hana_shared_vg-hana_shared_lv on /hana/shared type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/mapper/hana_log_vg-hana_log_lv on /hana/log type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=64k,sunit=128,swidth=256,noquota)
/dev/mapper/hana_data_vg-hana_data_lv on /hana/data type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=64k,sunit=128,swidth=512,noquota)

Check if the entries for /hana/data, /hana/log and /hana/shared exist in the /etc/fstab file:

cat /etc/fstab

Sample output:

...
/dev/mapper/datavg-datalv /hana/data xfs defaults,nofail 0 0
/dev/mapper/logvg-loglv /hana/log xfs defaults,nofail 0 0
/dev/mapper/sharedvg-sharedlv /hana/shared xfs defaults,nofail 0 0

It might be necessary to reboot the operating system. Then check that the LV setup is still correct and proceed to the Preparing for SAP Installation section.

Configuring network management services

The configure_network_management_services Ansible role installs and configures a proxy endpoint, NTP, DNS, and NFS network services in a virtual server instance. For more details, see the full role description on Ansible Galaxy.

Before executing an Ansible playbook where the configure_network_management_servicesrole is defined, configure the variable file in playbooks/vars/sample-variables-configure-network-services-client.yml by updating the downloaded sample with your own values, see an example how it might look like:

client_config:
  squid:
    enable: true
    squid_server_ip_port: "SQUID_PROXY_SERVER_IP:3128"
    no_proxy_hosts: "161.0.0.0/8"
  ntp:
    enable: true
    ntp_server_ip: "NTP_SERVER_IP"
  nfs:
    enable: true
    nfs_server_path: "NFS_SERVER_IP:/nfs"
    nfs_client_path: "/nfs"
    opts: sec=sys,nfsvers=4.1,nofail
    fstype: nfs4
  dns:
    enable: true
    dns_server_ip: "DNS_SERVER_IP"

A detailed description of the parameters SQUID_PROXY_SERVER_IP, NTP_SERVER_IP, NFS_SERVER_IP, DNS_SERVER_IP, that are used for the above example, can be found in the section Edit parameters in the configuration file.

After updating the variable file, run the following command:

ansible-playbook --connection=local -i "localhost," playbooks/sample-configure-network-services-client.yml

Creating the file systems

The Ansible role, powervs_storage_and_swap_setup is used to create file systems for SAP HANA, SAP NetWeaver, or for a SAP shared file systems instance. This role performs tasks such as creating file systems for /hana/data, /hana/log and /hana/shared, mounting these file systems on provided mount points, adding an entry to /etc/fstab for automount on reboot, and other tasks. For more details, see the description of powervs_storage_and_swap_setup on Ansible Galaxy.

Before running a playbook for a storage setup, identify the WWNs as described in the Manually creating file systems section. Add the WWNs to the sample-variables-powervs-storage-setup.yml variable file. For an explanation of the variable file, see the Edit parameters in the configuration file section.

Next, run an Ansible playbook to create the file systems as:

ansible-playbook --connection=local -i "localhost," playbooks/sample-powervs-storage-setup.yml

To see if all file systems have been created and mounted correctly, use the Checking of storage volumes section. Continue to Preparing for SAP installation, if the /hana/data, /hana/log, and /hana/shared are mounted correctly.

Configuring SLES for SAP applications

SLES

Use the saptune tool to apply recommended operating system settings for SAP HANA or SAP NetWeaver on SUSE Linux® Enterprise Server. On IBM Power Systems Virtual Servers, the same SUSE Linux® Enterprise Server image is used for SAP NetWeaver and SAP HANA.

The following workflow shows how you can use the saptune tool to apply the SAP solution to your server. For more information about saptune, see SAP Note 1275776 - Linux: Preparing SLES for SAP environments.

1. Verify that the package status is current.

   zypper info saptune
   

2. Verify that the saptune version is at least 3.

   saptune version
   

3. List all available solutions. Numbered entries represent integrated SAP Notes for each of the solutions.

   saptune solution list
   

4. Get an overview of saptune options.

   saptune --help
   

5. Enable and start the saptune.service. This command also disables sapconf and tuned, which isn't used since saptune version 3.

   saptune service takeover
   

6. Simulate the changes before applying them (optional).

   For SAP HANA:

   saptune solution simulate HANA
   

   For SAP NetWeaver:

   saptune solution simulate NETWEAVER
   

7. Apply the saptune solution.

   For SAP HANA:

   saptune solution apply HANA
   

   For SAP NetWeaver:

   saptune solution apply NETWEAVER
   

8. Check the saptune status.

   saptune status
   

9. Verify if the saptune is set up correctly.

   saptune check
   

Configuring RHEL for SAP applications

RHEL

RHEL System Roles for SAP are a collection of Ansible roles that help you configure a RHEL system for installing SAP HANA or SAP NetWeaver. Ansible roles for SAP configuration are distributed and updated directly by Red Hat, so the task performed and parameters required might vary depending on the version of the rhel-system-roles-sap package. The RHEL image that is provided by IBM includes the Ansible execution engine, SAP-related system roles, and the Ansible execution files.

Starting with rhel-system-roles-sap-3.2.0-1.el8_4, the role names have changed. The /root/sap-preconfigure.yml, /root/sap-netweaver.yml, and /root/sap-hana.yml files in the OS images for RHEL 8.1 or RHEL 8.4 must be adapted. For more information, see following Red Hat article.

The RHEL system roles for setting up SAP application are available in the root directory.

Use the following command to prepare the operating system for an SAP HANA workload.

• For RHEL 8.4 and previous versions:

ansible-playbook /root/sap-hana.yml

• For RHEL versions RHEL 8.6 and greater:

ansible-playbook -i /root/inventory /root/sap-hana.yml

Use the following command to prepare the operating system for an SAP NetWeaver workload.

• For RHEL 8.4 and previous versions:

ansible-playbook /root/sap-netweaver.yml

• For RHEL versions RHEL 8.6 and greater:

ansible-playbook -i /root/inventory /root/sap-netweaver.yml

For more information about customizing the operationg system, see the following documentation.

• SAP Note 2772999 "Red Hat Enterprise Linux 8.x: Installation and Configuration"
• SAP Note 2777782 "SAP HANA DB: Recommended OS Settings for RHEL8"
• SAP Note 2382421 "Optimizing the Network Configuration on HANA- and OS-Level"
• Red Hat Enterprise Linux System Roles for SAP

Configuring jumbo frames

Jumbo frames should be enabled by setting MTU='9000'.

RHEL

In the /etc/sysconfig/network-scripts directory, check that the ifcfg-env0, ifcfg-env2, ifcfg-env(...) files contain the MTU='9000' parameter.

SLES

In the /etc/sysconfig/network directory, check the content of the files such as ifcfg-eth0, ifcfg-eth1, etc.

Checking the NUMA layout

Check that the CPU and memory placement is optimized for SAP HANA by running the chk_numa_lpm.py script. The chk_numa_lpm.py script performs the following actions.

• Checks the nonuniform memory access (NUMA) layout according to SAP HANA rules. The script verifies that there are no cores without memory and that the memory distribution between the cores doesn't exceed a margin of 50%. In the first case, the script generates an error; in the second case, the script generates a warning.
• Checks if a Live Partition Mobility (LPM) operation has been performed. After LPM, the NUMA layout might be different from the configuration at boot time. The script searches the system log for the last LPM operation. A warning is generated if there has been an LPM operation since the last system boot.

1. Check the information in SAP Note 2923962.

2. Download the chk_numa_lpm.py script that is attached to this SAP Note and copy it to your Power Virtual Server instance.

3. Set executable permissions for the script:

   chmod +x ./chk_numa_lpm.py
   

4. Run the script:

   ./chk_numa_lpm.py