Deploying SAP applications on Power Virtual Server
The following information explains how to deploy Power Virtual Server instances for SAP HANA database and SAP application server (NetWeaver) on Red Hat Enterprise Linux (RHEL) and SUSE Linux Enterprise Server (SLES).
An infrastructure, that was deployed on IBM Cloud® and runs on IBM® Power® Virtual Server consists of the following components:
- A separate private network for SAP system.
- A Power Virtual Server instance for shared file systems.
- A Power Virtual Server instance for SAP HANA.
- A Power Virtual Server instance for SAP NetWeaver (on Linux).
- An operating system, such as RHEL or SLES, that is configured to use management services that are configured on IBM Cloud® Virtual Private Cloud (SQUID proxy, NFS, NTP, DNS).
- A registered RHEL or SLES operating systems (OS) with IBM provided subscription, which includes SAP-specific network performance tuning, file system setup, and packages.
The following diagram shows SAP applications already deployed on Power Virtual Server infrastructure.
Deploying a separate private network for SAP systems
For each SAP system that you deploy, create a separate private network for communication between the virtual server instances. Follow Configuring a private network subnet to configure a separate private network for the SAP system.
Deploying Power Virtual Server instances for SAP on IBM Cloud®
It is recommended to perform a memory sizing of SAP applications before deploying them on Power Virtual Server instances. See the Sizing process for SAP Systems and IBM Power Virtual Server certified profiles for SAP HANA.
Before deploying of Power Virtual Server instances, you should be aware of the various options for choosing an operating system (OS) image, that is based on a subscription. The available subscription alternatives are:
-
IBM provided subscription, where IBM Cloud® provides a full subscription to IBM stock OS images, such as RHEL and SLES, Linux® for SAP applications (RHEL and SLES for SAP workloads), AIX, and IBM i.
-
Client supplied subscription, where you use your own subscription with IBM stock OS images or custom images. This feature is called "Bring Your Own License" (BYOL). Custom images are imported by users as boot images into Power Virtual Server. Therefore, if you plan to use your own subscription, select the OS image that has a suffix of -BYOL under the "Client supplied subscription" when deploying Power Virtual Server instances in the step Deploying an Power Virtual Server instance for SAP HANA.
For deploying and setting up the Power Virtual Server instances for SAP applications, we focus on the IBM provided subscription for Red Hat Enterprise Linux (RHEL) and SUSE Linux Enterprise Server (SLES) images.
More details about the subscription for RHEL and SLES could be found in Using RHEL within the Power Virtual Server and Using SLES within IBM Power Virtual Server.
IBM Cloud® for SAP provides SAP-certified infrastructure to run SAP workloads, which includes the following operating systems with OS images under IBM provided subscription:
-
Linux® for SAP HANA: Red Hat Enterprise Linux for SAP HANA and SUSE Linux Enterprise Server for SAP HANA. The images have a
SAP
suffix. -
Linux® for SAP application server (NetWeaver): Red Hat Enterprise Linux for SAP NetWeaver and SUSE Linux Enterprise Server for SAP NetWeaver. The images have a
NETWEAVER
suffix.
Pay attention to the choice of operating system. Linux® OS is not the same as Linux® for SAP applications. Linux® OS does not have specific pre-configured settings for SAP workloads.
See more details about Linux® versions for SAP applications in OS for IBM Power Virtual Servers.
It is recommended to perform a memory sizing of SAP applications before deploying them on Power Virtual Server instances. See the Sizing process for SAP Systems and IBM Power Virtual Server certified profiles for SAP HANA.
Deploying an Power Virtual Server instance for SAP HANA
To deploy an Power Virtual Server instance for SAP HANA database, select a previously created workspace from the list of Workspaces on the left navigation page. Then click Virtual server instances on the left page. To create a new instance, click Create instance on the right side. Complete the following steps:
-
In the General section, make the following selections:
Table 1. SAP HANA general selections Field Details Instance name Enter a unique name for the instance. Number of instances Enter '1'. Add to server placement group Optional and can be skipped. Add to a shared processor pool Optional and can be skipped. Virtual server pinning Optional, can be set to None as default selection. SSH key Choose an existing SSH key that was created previously. -
In the Boot image section, make the following selections:
Table 2. SAP HANA boot image selections Field Details Operating system Select the IBM provided subscription 'Linux for SAP (HANA)'. See the explanation of differences in the section Deploying Power Virtual Server instances for SAP on IBM Cloud® Image Select an operating system and a version. Make sure that you use the same operating system and version for all deployments. Tier Choose a tier that best meets your needs, for more information, see Storage tiers. Storage pool Select the storage pool that you need. Advanced configurations Enable toggle buttons to support more settings. -
In the Profile section, make the following selection:
- Select a profile that meets your needs. For more information, see SAP HANA profiles.
-
In the Storage volumes section, make the following selection:
- For SAP HANA, the attached volumes are on different storage tiers 'Tier 1' and 'Tier 3'. You can't mix storage tiers during the instance creation process, so you need to attach storage volumes later. Leave this list empty.
-
In the Networking section, make the following selections:
- Leave Public networks deactivated.
- Attach both private networks (management and backup) and any separate private networks. Enter the IP addresses as entered in the DNS configuration for the corresponding host names. If the IP addresses are assigned dynamically, you need to adapt the DNS entries for the host names of this system.
It takes some time for the Power Virtual Server instance for SAP HANA to become available. When the deployment is completed, you can log in to the instance via the VPC access host. Use the SSH command below to login as root
user
to the virtual server instance:
ssh -A -o ServerAliveInterval=60 -o ServerAliveCountMax=600 -o ProxyCommand="ssh -W %h:%p root@<ACCESS_HOST_FLOATING_IP>" root@<HANA_PVS_IP>
ACCESS_HOST_FLOATING_IP
is a public IP address of a jump host, HANA_PVS_IP
is the virtual server instance IP address in the management subnet.
An alternative solution for connecting to the VPC instance is to use a VPN server. See more details in the tutorial Connect by using a client-to-site VPN.
Deploying an Power Virtual Server instance for SAP NetWeaver
To deploy an Power Virtual Server instance for SAP NetWeaver, go to your workspace for Power Virtual Server and create an Power Virtual Server instance as described in Configuring a Power Virtual Server instance. Use the information in Deploying SAP HANA on Power Virtual Server to complete the configuration for General, Profile, Storage volumes, Networking sections. For the Boot image section, specify the IBM provided subscription 'Linux for SAP (NetWeaver)' selection.
After creating an Power Virtual Server instance for SAP NetWeaver, wait for the instance to become active. Then log on to the SAP NetWeaver instance by using the following SSH command:
ssh -A -o ServerAliveInterval=60 -o ServerAliveCountMax=600 -o ProxyCommand="ssh -W %h:%p root@<ACCESS_HOST_FLOATING_IP>" root@<NETWEAVER_PVS_IP>
ACCESS_HOST_FLOATING_IP
is a public IP address of a jump host, NETWEAVER_PVS_IP
is the virtual server instance IP address in the management subnet.
Alternatively, connect by using a VPN client, as described in the tutorial Connect by using a client-to-site VPN.
Creating extra storage volumes for SAP HANA Power Virtual Server
Modify Power Virtual Server SAP HANA instance and attach extra storage volumes as described in Managing your storage volumes.
In accordance with the SAP sizing guidelines, you should add the following volumes:
-
A storage volume for SAP HANA shared file system with a size of MIN (1 x RAM; 1 TB). Storage tier "Tier 3" is sufficient. The "Shareable" switch should remain "off".
-
Four or more equal-size storage volumes for SAP HANA log file system with size MIN (1/2xRAM; 512 GB). Divide the file system size by the number of volumes to determine the size for each storage volume. Select "Tier 1", "Tier 0", or "Fixed IOPS Tier" as the storage tier. Ensure that your configuration provides a total of at least 12,000 IOPS (input/output operations per second). The "Shareable" flag should remain "off".
-
Four or more equal-sized storage volumes for the SAP HANA data file system with a size of 1.5 times the RAM. Divide the file system size by the number of volumes to determine the size for each storage volume. Select "Tier 1" or "Tier 0" as the storage tier. Ensure that your configuration provides a total of at least 8,000 IOPS (input/output operations per second). The "Shareable" flag should remain "off".
-
An extra storage volume for other data (such as "/usr/sap" file system). The Storage tier "Tier 3" is sufficient. The "Shareable" flag should remain "off".
-
You can add more volumes for backup or export.
For more information, see Block Storage for VPC profiles.
Continue with the configuration of Power Virtual Server instances for SAP applications, which can be done by manual or automated setup. You can choose one of the following setup sections:
Configuring Power Virtual Server instances manually
Complete the following steps on your Power Virtual Server instances.
Checking IBM subscription for an operating system image
To ensure that the OS subscription is set up correctly, check a subscription manager by running:
RHEL:
subscription-manager release
subscription-manager list
SLES:
yum repolist
Optional:
subscription-manager status
Then run a system update.
RHEL:
yum -y update
SLES:
zypper update -y
Configuring a proxy endpoint
To use proxy and cache services for HTTP, FTP, and other popular network protocols, you must to export a proxy server endpoint. Run the following commands to export a SQUID_PROXY_SERVER
proxy endpoint:
export http_proxy=http://<SQUID_PROXY_SERVER>:3128
export https_proxy=http://<SQUID_PROXY_SERVER>:3128
export HTTP_PROXY=http://<SQUID_PROXY_SERVER>:3128
export HTTPS_PROXY=http://<SQUID_PROXY_SERVER>:3128
To keep these exported variables persistent across multiple sessions, these entries must be added to /etc/bash.bashrc
(on SLES) or /etc/bashrc
(on RHEL). These files ensure that new sessions use the exported variables as environment variables.
Configuring a NTP client
If an NTP server is already configured, you must install the chrony package by using the following command.
RHEL:
yum install -y chrony
SLES:
zypper install -y chrony
Check the status of chrony service:
systemctl status chronyd
Adapt the chrony.conf
configuration according to the description in the section Setting up chrony for a system in an isolated network.
The following is a sample configuration for `chrony.conf':
server <NTP_SERVER_IP> iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
logdir /var/log/chrony
Then, restart the chronyd
service on the node.
systemctl restart chronyd.service
Configuring a DNS client
A DNS server should be configured after the Power Virtual Server instance is created. The /etc/resolv.conf
file contains all the DNS server configurations, there is no need to modify this file.
Configuring an NFS client
Use the following steps to manually configure an NFS client.
-
Install the
nfs-client
package and enable the NFS client by using the systemctl command.- RHEL:
yum install -y nfs-utils
- SLES:
zypper install -y nfs-utils
-
To enable NFS client service, use the following command.
systemctl start nfs-client
-
After the NFS client service starts, you can mount the shared NFS directory by using the mount command.
mount -t nfs4 -o sec=sys,nfsvers=4.1 <NFS_SERVER_IP>:<NFS_DIRECTORY_PATH>
NFS_SERVER_IP
is an IP address of a loadbalancer,NFS_DIRECTORY_PATH
is the path of the NFS file storage share.
Creating file systems manually
For shared SAP file systems, you must create a file system to store SAP data and distribute them to all SAP instances. You can use extra file systems for other purposes.
For an SAP NetWeaver instance, you must create a file system to store instance-specific data.
Three file systems are required to install SAP HANA: data, log, and shared. According to the default installation catalog, these file systems are /hana/data
, /hana/log
and /hana/shared
, but you can customize
the file system names. You might also need file systems for other purposes (/usr/sap
directory). The /hana/data
and /hana/log
file systems are striped across four or eight disks, depending on the number
of volumes that you created.
/hana/shared
and all other file systems are non-striped 1-disk file systems.
Perform a disk discovery by running the following script:
/usr/bin/rescan-scsi-bus.sh -a -c -v
Newly discovered disks are listed with their details.
To identify the Word Wide Names (WWNs) that will be used to set up storage volumes, run:
multipath -ll
The output of the multipath -ll
command corresponds to the World Wide Names that are also listed on the IBM Cloud® console. Log on to the IBM Cloud® console and go to the Storage volumes, select a workspace and a Virtual server instance where storage volumes are defined. You might notice that the WWNs on the IBM Cloud® are in uppercase, while they are in lowercase in the operating system.
To create a file system, use the /hana/data
example that is decribed below. The same procedure is repeated for /hana/log
and /hana/shared
and all other file systems. For example, a storage volume named
dm-6
has a WWN of 6005076813810214200000000000a7a8
and a size of 60G. The 3
at the beginning is ignored. The device name is needed to create a logical volume and a volume group.
multipath -ll
36005076813810214200000000000a72d dm-0 IBM,2145
size=100G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 1:0:1:0 sdj 8:144 active ready running
| |- 2:0:1:0 sdab 65:176 active ready running
| |- 3:0:1:0 sdat 66:208 active ready running
| `- 4:0:1:0 sdbl 67:240 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
|- 1:0:0:0 sda 8:0 active ready running
|- 2:0:0:0 sds 65:32 active ready running
|- 3:0:0:0 sdak 66:64 active ready running
`- 4:0:0:0 sdbc 67:96 active ready running
36005076813810214200000000000a7a8 dm-6 IBM,2145
size=60G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 1:0:1:4 sdn 8:208 active ready running
| |- 2:0:1:4 sdaf 65:240 active ready running
| |- 3:0:1:4 sdax 67:16 active ready running
| `- 4:0:1:4 sdbp 68:48 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
|- 1:0:0:4 sde 8:64 active ready running
|- 2:0:0:4 sdw 65:96 active ready running
|- 3:0:0:4 sdao 66:128 active ready running
`- 4:0:0:4 sdbg 67:160 active ready running
...
Multipath aliases are used for this setup. Run the following commands to create the /hana/data
file system.
Export the following variables:
export pv_size=60G
export lv_name=hana_data_lv
export vg_name=hana_data_vg
export mount=/hana/data
If you are using multipath aliases, use the following commands:
devices=$(multipath -ll | grep -B 1 $pv_size | grep dm- | awk '{print "/dev/"$2}' | tr '\n' ' ')
stripes=$(multipath -ll | grep -B 1 $pv_size | grep dm- | awk '{print "/dev/"$2}' | wc | awk '{print $1}')
pvcreate $devices
vgcreate ${vg_name} ${devices}
lvcreate -i${stripes} -I64 -l100%VG -n ${lv_name} ${vg_name}
mkfs.xfs /dev/mapper/${vg_name}-${lv_name}
mkdir -p ${mount}
Run the same commands to create storage volumes for /hana/log
and /hana/shared
. Change lv_name
to hana_log_lv
, vg_name
to hana_log_vg
and mount
to /hana/log
.
Export these variables:
export lv_name=hana_log_lv
export vg_name=hana_log_vg
export mount=/hana/log
Use the same approach for /hana/shared
.
Make sure that the variable pv_size
has different values for /hana/data
, /hana/log
and /hana/shared
, before running the export
command, otherwise the whole setup will not work.
If you don't use multipath aliases, replace the line (starting with devices=$()
that is used to identify devices. Instead, use following line: devices=$(multipath -ll | grep -B 1 $pv_size | grep dm- | awk '{print "/dev/"$2}' | tr '\n' ' ')
Checking the storage volumes
After you create the storage volumes, verify that they were created correctly by running the following commands:
lvscan
The command output shows the status of the created logical volumes (LV). The status should be active.
To check if the file system is mounted:
mount | grep hana
The following is a sample output of this command:
/dev/mapper/hana_shared_vg-hana_shared_lv on /hana/shared type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/mapper/hana_log_vg-hana_log_lv on /hana/log type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=64k,sunit=128,swidth=256,noquota)
/dev/mapper/hana_data_vg-hana_data_lv on /hana/data type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=64k,sunit=128,swidth=512,noquota)
Check if the entries for /hana/data
, /hana/log
and /hana/shared
exist in the /etc/fstab
file:
cat /etc/fstab
Sample output:
...
/dev/mapper/datavg-datalv /hana/data xfs defaults,nofail 0 0
/dev/mapper/logvg-loglv /hana/log xfs defaults,nofail 0 0
/dev/mapper/sharedvg-sharedlv /hana/shared xfs defaults,nofail 0 0
It might be necessary to reboot the operating system. Then check that the LV setup is still correct and proceed to the Preparing for SAP Installation section.
Configuring Power Virtual Server instances by using Ansible automation playbooks
To configure Power Virtual Server instances for SAP applications, use Ansible automation playbooks on both RHEL and SLES. Download and install the
ibm.power_linux_sap
Ansible Galaxy collection:
ansible-galaxy collection install ibm.power_linux_sap
This Ansible Galaxy collection is used on both operating systems, RHEL and SLES without changing any specific OS settings.
Configuring network management services
The configure_network_management_services
Ansible role installs and configures a proxy endpoint, NTP, DNS, and NFS network services in a virtual server instance. For more details, see the full role description on Ansible Galaxy.
Before executing an Ansible playbook where the configure_network_management_services
role is defined, configure the variable file in playbooks/vars/sample-variables-configure-network-services-client.yml
by updating
the downloaded sample with your own values, see an example how it might look like:
client_config:
squid:
enable: true
squid_server_ip_port: "SQUID_PROXY_SERVER_IP:3128"
no_proxy_hosts: "161.0.0.0/8"
ntp:
enable: true
ntp_server_ip: "NTP_SERVER_IP"
nfs:
enable: true
nfs_server_path: "NFS_SERVER_IP:/nfs"
nfs_client_path: "/nfs"
opts: sec=sys,nfsvers=4.1,nofail
fstype: nfs4
dns:
enable: true
dns_server_ip: "DNS_SERVER_IP"
A detailed description of the parameters SQUID_PROXY_SERVER_IP
, NTP_SERVER_IP
, NFS_SERVER_IP
, DNS_SERVER_IP
, that are used for the above example, can be found in the section Edit parameters in the configuration file.
After updating the variable file, run the following command:
ansible-playbook --connection=local -i "localhost," playbooks/sample-configure-network-services-client.yml
Creating the file systems
The Ansible role, powervs_storage_and_swap_setup
is used to create file systems for SAP HANA, SAP NetWeaver, or for a SAP shared file systems instance. This role performs tasks such as creating file systems for /hana/data
,
/hana/log
and /hana/shared
, mounting these file systems on provided mount points, adding an entry to /etc/fstab
for automount on reboot, and other tasks. For more details, see the description of
powervs_storage_and_swap_setup
on Ansible Galaxy.
Before running a playbook for a storage setup, identify the WWNs as described in the Manually creating file systems section. Add the WWNs to the sample-variables-powervs-storage-setup.yml
variable file. For an explanation of the variable file, see the Edit parameters in the configuration file section.
Next, run an Ansible playbook to create the file systems as:
ansible-playbook --connection=local -i "localhost," playbooks/sample-powervs-storage-setup.yml
To see if all file systems have been created and mounted correctly, use the Checking of storage volumes section. Continue to Preparing for SAP installation,
if the /hana/data
, /hana/log
, and /hana/shared
are mounted correctly.
Preparing for SAP software installation
Preparation for SAP software installation is required for SAP HANA and SAP NetWeaver, but is not needed on the Power Virtual Server instance for SAP shared file systems.
Configuring SLES for SAP applications
SLES
Use the saptune tool to apply recommended operating system settings for SAP HANA or SAP NetWeaver on SUSE Linux® Enterprise Server. On IBM Power Systems Virtual Servers, the same SUSE Linux® Enterprise Server image is used for SAP NetWeaver and SAP HANA.
The following workflow shows how you can use the saptune tool to apply the SAP solution to your server. For more information about saptune, see SAP Note 1275776 - Linux: Preparing SLES for SAP environments.
-
Verify that the package status is current.
zypper info saptune
-
Verify that the saptune version is at least 3.
saptune version
-
List all available solutions. Numbered entries represent integrated SAP Notes for each of the solutions.
saptune solution list
-
Get an overview of saptune options.
saptune --help
-
Enable and start the saptune.service. This command also disables sapconf and tuned, which isn't used since saptune version 3.
saptune service takeover
-
Simulate the changes before applying them (optional).
For SAP HANA:
saptune solution simulate HANA
For SAP NetWeaver:
saptune solution simulate NETWEAVER
-
Apply the saptune solution.
For SAP HANA:
saptune solution apply HANA
For SAP NetWeaver:
saptune solution apply NETWEAVER
-
Check the saptune status.
saptune status
-
Verify if the saptune is set up correctly.
saptune check
Configuring RHEL for SAP applications
RHEL
RHEL System Roles for SAP are a collection of Ansible roles that help you configure a RHEL system for installing SAP HANA or SAP NetWeaver. Ansible roles for SAP configuration are distributed and updated directly by Red Hat, so the task performed
and parameters required might vary depending on the version of the rhel-system-roles-sap
package. The RHEL image that is provided by IBM includes the Ansible execution engine, SAP-related system roles, and the Ansible execution
files.
Starting with rhel-system-roles-sap-3.2.0-1.el8_4
, the role names have changed. The /root/sap-preconfigure.yml
, /root/sap-netweaver.yml
, and /root/sap-hana.yml
files in the OS images for
RHEL 8.1 or RHEL 8.4 must be adapted. For more information, see following Red Hat article.
Previous role name | New role name |
---|---|
sap-preconfigure | sap_general_preconfigure |
sap-netweaver-preconfigure | sap_netweaver_preconfigure |
sap-hana-preconfigure | sap_hana_preconfigure |
The RHEL system roles for setting up SAP application are available in the root directory.
Use the following command to prepare the operating system for an SAP HANA workload.
- For RHEL 8.4 and previous versions:
ansible-playbook /root/sap-hana.yml
- For RHEL versions RHEL 8.6 and greater:
ansible-playbook -i /root/inventory /root/sap-hana.yml
Use the following command to prepare the operating system for an SAP NetWeaver workload.
- For RHEL 8.4 and previous versions:
ansible-playbook /root/sap-netweaver.yml
- For RHEL versions RHEL 8.6 and greater:
ansible-playbook -i /root/inventory /root/sap-netweaver.yml
For more information about customizing the operationg system, see the following documentation.
Configuring jumbo frames
Jumbo frames should be enabled by setting MTU='9000'
.
RHEL
In the /etc/sysconfig/network-scripts
directory, check that the ifcfg-env0
, ifcfg-env2
, ifcfg-env(...)
files contain the MTU='9000' parameter
.
SLES
In the /etc/sysconfig/network
directory, check the content of the files such as ifcfg-eth0
, ifcfg-eth1
, etc.
Checking the NUMA layout
Check that the CPU and memory placement is optimized for SAP HANA by running the chk_numa_lpm.py
script. The
chk_numa_lpm.py
script performs the following actions.
- Checks the nonuniform memory access (NUMA) layout according to SAP HANA rules. The script verifies that there are no cores without memory and that the memory distribution between the cores doesn't exceed a margin of 50%. In the first case, the script generates an error; in the second case, the script generates a warning.
- Checks if a Live Partition Mobility (LPM) operation has been performed. After LPM, the NUMA layout might be different from the configuration at boot time. The script searches the system log for the last LPM operation. A warning is generated if there has been an LPM operation since the last system boot.
-
Check the information in SAP Note 2923962.
-
Download the
chk_numa_lpm.py
script that is attached to this SAP Note and copy it to your Power Virtual Server instance. -
Set executable permissions for the script:
chmod +x ./chk_numa_lpm.py
-
Run the script:
./chk_numa_lpm.py
Next Steps
Your infrastructure is now ready to install the SAP software.