Custom OS image build process on RHEL for SAP solutions on IBM® Power® Virtual Server

The following information describes how to build a custom operating system (OS) image on Red Hat Enterprise Linux (RHEL) for SAP software on IBM® Power® Virtual Server.

Custom OS image build requirements

Complete the following requirements before you start the image build process:

Create an IBM Cloud account and obtain the necessary credentials to deploy an image build server on Power Virtual Server. Details about identity and access management (IAM) are described in Managing identity and access management (IAM) for IBM® Power® Virtual Server.
Ensure that you have access to IBM Cloud Object Storage so that you can upload a base image. See Getting started with IBM Cloud Object Storage for information on how to get started.
Download the Red Hat Enterprise Linux 9 KVM Guest Image rhel-9.4-ppc64le-kvm.qcow2 from Red Hat Enterprise Linux for Power little endian.

The following guidance for building a custom OS image is based on RHEL 9.4 OS image.
Create an image build server, where you configure an image build environment and build an OS image. It consists of an Power Virtual Server instance running RHEL 9.2 (RHEL9-SP2-SAP-NETWEAVER) or higher RHEL version, with the additional storage volume of 200 GByte. There are two methods how to set up the image build server:
- For a manual approach, refer to the detailed information in Getting started with Power Virtual Server and Creating an IBM Power Virtual Server.
- For an automated approach, refer to the Power Virtual Server with VPC Landing Zonedeployment guide.

Setup a build image environment

The following steps describe how to set up a build server environment for image creation.

Log in to your build server and check its registration status by executing the following command.
```
subscription-manager status
```
Update RHEL OS packages.
Check and adjust proxy settings for a build environment. Configure temporary HTTPS proxy settings by using your Proxy IP address or URL. If you deployed the automated Power Virtual Server with VPC landing zone from IBM Cloud catalog, use this proxy address http://10.30.40.7:3128/ for your configuration.
```
export https_proxy="http://10.30.40.7:3128/"  || die "Set HTTPS proxy failed (rc $?)"
```
Create a '/data' directory and export an environment variable that points to it.
```
mkdir -p /data
```
```
export MOUNT_PATH=/data
```

Attach an additional storage volume with a capacity of 200 GB and create the /data file system.

Run the multipath command to identify the available storage volumes and their World Wide Names (WWNs). Ensure that the command output only shows one 200 GB storage volume. Otherwise, creating the file system will not work properly..
```
multipath -ll
```
Export the following variables for physical and logical volumes, a volume group, and a mount directory.
```
export PV_SIZE=200G
```
```
export LV_NAME=image_lv
```
```
export VG_NAME=image_vg
```

Create the physical and logical volumes.

devices=$(multipath -ll | grep -B 1 $PV_SIZE | grep dm- | awk '{print "/dev/"$2}' | tr '\n' ' ')

stripes=$(multipath -ll | grep -B 1 $PV_SIZE | grep dm- | awk '{print "/dev/"$2}' | wc | awk '{print $1}')

pvcreate $devices

vgcreate ${VG_NAME} ${devices}

lvcreate -i${stripes} -I64 -l100%VG -n ${LV_NAME} ${VG_NAME}

mkfs.xfs /dev/mapper/${VG_NAME}-${LV_NAME}

Check if the created logical volume is active.
```
lvscan
```

Mount the file system.

mount -t xfs -o defaults,nofail --source /dev/mapper/${VG_NAME}-${LV_NAME} --target ${MOUNT_PATH}

Add the file system to the /etc/fstab file.

echo "/dev/mapper/${VG_NAME}-${LV_NAME}  ${MOUNT_PATH} xfs defaults,nofail 0 0" >> /etc/fstab

Ensure that the file system is present in the list of all mounted file systems.
```
df -h
```

Create the directory /data/image where the base image is saved.
Copy the image rhel-9.4-ppc64le-kvm.qcow2 to /data/image.
Create the /data/tmp directory, in which a temporary image is saved during the build process.
Complete the following tasks to configure the build server. Use the dnf RHEL package manager.
- Install the xfsprogs and lvm2 system packages.
- Install the make, git, libgcrypt-devel and go-toolset development packages.
- Install the qemu-img and cloud-utils-growpart packages.
- Configure a proxy by adding your proxy host in the file ~root/.bashrc.
```
export https_proxy='<your_https_proxy>'
```
Configure the NBD module to load at boot time by adding the following lines to the file /etc/modules-load.d/ndb.conf.
```
nbd
options nbd max_part=8
```
Download the pvsadm tool from the latest release of pvsadm-linux-ppc64le GitHub repository. Save it in the file named /usr/local/bin/pvsadm and change the mode to 0755. For more details about the tool, refer to pvsadm tool.

Building a custom OS image

The build process of a custom OS image for IBM Cloud PowerVS consists of the following tasks.

Check that the Red Hat Satellite subscription credentials are valid.
Create an empty file named image-prep.template and use it as the image preparation template .

The following tasks contain the shell commands that are needed for building a customized image. Add these commands to the template image-prep.template. The commands update the base OS image during image creation with the pvsadm tool to create the final SAP-ready OS image. The command order is important and should not be changed.

Copy the following lines, including error handling, to be used as the template header.

#!/usr/bin/env bash
set -o errexit
set -o nounset
set -o pipefail

die() {
  echo -e "\n${1}"
  set +o errexit
  set +o nounset
  exit 1
}

Add a temporary name server.

echo "nameserver 9.9.9.9" >> /etc/resolv.conf \
    || die "Add nameserver failed (rc $?)"

Create a work directory /tmp/work and export it as an environmental variable.

mkdir -p /tmp/work || die "Create work directory failed (rc $?)"

export WORK_DIR=/tmp/work

Create the 25GB swap file, add permissions and check that it has been created correctly.

fallocate -l 25G /swapfile || die "allocate swapfile failed (rc $?)"

chmod 600 /swapfile || die "chmod swapfile failed (rc $?)"

mkswap /swapfile || die "mkswap swapfile failed (rc $?)"

swapon /swapfile || die "swapon swapfile failed (rc $?)"

swapon --show

echo "/swapfile swap swap defaults 0 0" >> /etc/fstab || die "update fstab with swapfile failed (rc $?)"

echo "swapon -s"

Subscribe to Red Hat Satellite and enable the SAP repositories. Use your own values for the subscription user name, password and release version.

subscription-manager register \
    --force --auto-attach \
    --username=<SUBSCRIPTION_USER>  \
    --password=<SUBSCRIPTION_PASSWORD> \
    --release=<OS_VERSION> \
   || die "Register subscription failed (rc $?)"

subscription-manager repos \
    --disable="*" \
    --enable="rhel-9-for-$(uname -m)-baseos-e4s-rpms" \
    --enable="rhel-9-for-$(uname -m)-appstream-e4s-rpms" \
    --enable="rhel-9-for-$(uname -m)-sap-solutions-e4s-rpms" \
    --enable="rhel-9-for-$(uname -m)-sap-netweaver-e4s-rpms" \
    --enable="rhel-9-for-$(uname -m)-highavailability-e4s-rpms"\
    --enable="codeready-builder-for-rhel-9-$(uname -m)-rpms" \
    || die "Repository configuration failed (rc $?)"

Install a package group server.

dnf -y group install server \
    || die "Installing server group packages failed (rc $?)"

Install the system packages.

dnf -y install \
    cloud-init \
    device-mapper-multipath \
    libxcrypt-compat \
    glibc-langpack-en \
    || die "Install system packages failed (rc $?)"

Update grub2.

dnf -y reinstall grub2-common \
    || die "Reinstall system package failed (rc $?)"

dnf -y update grub2 \
    || die "Update grub2 package failed (rc $?)"

Install Ansible and the Red Hat system roles for SAP.

dnf -y install ansible-core rhel-system-roles rhel-system-roles-sap \
    || die "Install Ansible and RH System Roles failed (rc $?)"

Enable the EPEL repository.

dnf -y install "https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm" \
    || die "Installation EPEL configuration failed (rc $?)"

Configure the IBM Power tools repository.

dnf -y install "https://public.dhe.ibm.com/software/server/POWER/Linux/yum/download/ibm-power-repo-latest.noarch.rpm" \
    || die "Installation IBM Power Tools configuration failed (rc $?)"

echo 'y' | (/opt/ibm/lop/configure \
    || die "Power Tools configuration failed (rc $?)")

Install the IBM Power tools and disable the repository.

dnf config-manager --set-disabled Advance_Toolchain \
    || die "Disable Advance Toolchain repository failed (rc $?)"

dnf -y install ibm-power-managed-rhel9 \
   || die "Install IBM Power Tools failed (rc $?)"

dnf config-manager --set-disabled IBM_Power_Tools \
    || die "Disable IBM Power Tools repository failed (rc $?)"

Updating OS.

dnf -y update || die "OS update failed (rc $?)"

Remove the EPEL repository.

dnf -y remove epel-release \
    || die "Uninstall EPEL configuration failed (rc $?)"

Unregister from the Red Hat Satellite server.

subscription-manager unregister \
    || die "Unregister subscription failed (rc $?)"

subscription-manager clean \
    || die "Clean subscription failed (rc $?)"

Change the cryptographic policy (disable SHA1/CBC).

echo -e "cipher@ssh = -*-CBC\n" \
   > /etc/crypto-policies/policies/modules/NO-CBC.pmod \
   || die "Create of CBC disabling file failed (rc $?)"

update-crypto-policies --set DEFAULT:NO-SHA1:NO-CBC \
    || die "Update of cryptografic policy failed (rc $?)"

Create a basic multipath configuration.

echo -e "defaults {\n \
    user_friendly_names no\n \
    find_multipaths smart\n}\n" \
    > /etc/multipath.conf \
    || die "Create multipath configuration failed (rc $?)"

Define an Ansible inventory.

echo -e "localhost ansible_connection=local\n" > /root/inventory \
    || die "Create Ansible inventory file failed (rc $?)"

Create Ansible yml-files for SAP HANA and NetWeaver.

cat <<EOF > /root/sap-hana.yml \
  || die "Create SAP HANA Ansible file failed (rc $?)"
- hosts: localhost
  vars:
   sap_hana_preconfigure_min_rhel_release_check: false
   sap_hana_preconfigure_install_ibm_power_tools: false
   sap_hana_preconfigure_add_ibm_power_repo: false
  connection: local
  roles:
    - redhat.sap_install.sap_general_preconfigure
    - redhat.sap_install.sap_hana_preconfigure
EOF

cat <<EOF > /root/sap-netweaver.yml \
  || die "Create SAP NetWeaver Ansible file failed (rc $?)"
- hosts: localhost
  connection: local
  roles:
    - redhat.sap_install.sap_general_preconfigure
    - redhat.sap_install.sap_netweaver_preconfigure
EOF

cat <<EOF > /root/sap-preconfigure.yml \
  || die "Create SAP preconfiguration Ansible file failed (rc $?)"
- hosts: localhost
  connection: local
  roles:
    - redhat.sap_install.sap_general_preconfigure
EOF

Change the default kernel parameter in the file /etc/sysctl.conf.

cat <<EOF >> /etc/sysctl.conf \
 || die "Change of kernel parameter failed (rc $?)"
net.core.rmem_max = 56623104
net.core.wmem_max = 56623104
net.ipv4.tcp_rmem = 65536 262088 56623104
net.ipv4.tcp_wmem = 65536 262088 56623104
net.ipv4.tcp_mem = 56623104 56623104 56623104
EOF

Enable Receive Flow Steering (RFS) on ibmveth devices, adjust the file /etc/udev/rules.d/70-ibmveth-rfs.rules.

cat <<EOF >> /etc/udev/rules.d/70-ibmveth-rfs.rules \
 || die "Create udev rule for RFS failed (rc $?)"
# Enable Receive Flow Steering (RFS) on ibmveth devices
SUBSYSTEM=="net",ACTION=="add",DRIVERS=="ibmveth",RUN{program}+="/bin/bash -c 'echo 32768 > /sys/\$DEVPATH/queues/rx-0/rps_flow_cnt';"
EOF

Enable the Receive Flow Steering (RFS) in the file /etc/sysctl.d/95-enable-rfs.conf.

cat <<EOF >> /etc/sysctl.d/95-enable-rfs.conf \
 || die "Create kernel parameter file for RFS failed (rc $?)"
# Enable Receive Flow Steering (RFS)
net.core.rps_sock_flow_entries=32768
EOF

Configure the SSH daemon.

sed -i \
    -e 's/^\(#\)\?PermitRootLogin .*/PermitRootLogin yes/g' \
    -e 's/^\(#\)\?PasswordAuthentication .*/PasswordAuthentication no/g' \
    -e 's/^\(#\)\?MaxStartups .*/MaxStartups 10:30:60/g' \
    /etc/ssh/sshd_config \
    || die "Failed SSH daemon configuration (rc $?)"

Adapt GRUB.

Check that the PReP partition exists and update it.

prep_partition=$(fdisk -l | grep -i ppc | grep -i loop | awk '{print $1}')

# Check if a PReP partition was found
if [ -z "$prep_partition" ]; then
  echo "No PReP partition found with /dev/loop."
  exit 1
else
  echo "PReP partition found: $prep_partition"
fi

Install grub2on the PReP partition.

grub2-install "$prep_partition" \
    || die "Install GRUB update failed (rc $?)"

Execute the bootlist command.

bootlist -m normal -o || die "Setting boot list failed (rc $?)"

Change the GRUB default timeout option.

sed -i \
    -e 's/GRUB_TIMEOUT=.*/GRUB_TIMEOUT=20/' \
    /etc/default/grub || die "Fail to change default GRUB options (rc $?)"

Update the GRUB configuration.

grubby --update-kernel=ALL \
    --remove-args="net.ifnames=0" \
    --args="console=tty0 console=hvc0,115200n8 crashkernel=2G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G rd.shell rd.debug rd.driver.pre=dm_multipath log_buf_len=1M elevator=none" \
    || die "GRUB cmdline configuration update failed (rc $?)"

Enable multipath for all kernels.

echo -e 'force_drivers+=" dm-multipath "\n' >/etc/dracut.conf.d/10-mp.conf\
    || die "Create of multipath include file for dracut failed (rc $?)"

dracut --regenerate-all --force \
 || die "Regenerate of initramfs images failed (rc $?)"
for kernel in $(rpm -q kernel | sort -V | sed 's/kernel-//'); do
  dracut --kver ${kernel} --force --add multipath --include /etc/multipath\
  /etc/multipath --include /etc/multipath.conf /etc/multipath.conf \
     || die "Generate initramfs of ${kernel} failed (rc $?)"
done

Generate the GRUB configuration.

grub2-mkconfig -o /boot/grub2/grub.cfg \
    || die "Generate GRUB configuration failed (rc $?)"

Remove the temporary name server.

sed -i '/nameserver 9.9.9.9/d' /etc/resolv.conf \
    || die "Remove nameserver failed (rc $?)"

Enable SELinux relabeling on the next boot.

touch /.autorelabel || die "Create relabel file failed (rc $?)"

Delete the root user password.

usermod -p '!' root || die "Delete root password failed (rc $?)"

Clean up the file system.

rm -rf /etc/sysconfig/network-scripts/ifcfg-eth*

rm -rf  /tmp/work

rm -rf /root/.ssh

rm -rf /etc/pki/entitlement/

rm -rf /setup.sh

Deleting command history.
```
history -c
```

Before running the pvsadm tool, make sure that local swapping on a build server is disabled. Otherwise, the error mkswap: error: /swapfile is mounted; will not make swapspace occurs while executing pvsadm. To disable swapping, run the following command.
```
swapoff -a
```

Use the pvsadm image build tool with the qcow2ova command to convert the qcow2 image to the OVA format. Set your own customized values for the command flags according to their input formats. See the description of the flags below.

--image-name <string>     Name of the resultant OVA image
--image-url  <string>     URL or absolute local file path to the qcow2 image
--image-dist <string>     Image Distribution(supported: rhel, centos, coreos)
--target-disk-size <int>  Size (in GB) of the target disk volume where OVA will be copied (default 120)
--rhn-user <string>       RedHat Subscription username. Required when Image distribution is rhel
--rhn-password <string>   RedHat Subscription password. Required when Image distribution is rhel
--os-password  <string>   Root user password, will auto-generate the 12 bits password(applicable only for redhat and cento distro)
--temp-dir <string>       Scratch space to use for OVA generation
--prep-template <string>  Image preparation script template

For instance, use the rhel-9.4-ppc64le-image as a new OS image name, rhel-9.4-ppc64le-kvm.qcow2 is the base OS image located in /data/image, so the absolute local path for the qcow2 image is /data/image/rhel-9.4-ppc64le-kvm.qcow2, and the image distribution is rhel. Adjust the image and target disk sizes according to your requirements, 120 GB is set as the default value for the target disk volume. An image preparation script template is image-prep.template, and the name of the script is customizable as well. To simplify running the pvsadm image build tool, you can set up environment variables as well.

export IMAGE_NAME=rhel-9.4-ppc64le-image
export QCOW2_IMAGE_PATH=/data/image/rhel-9.4-ppc64le-kvm.qcow2
export IMAGE_SIZE=100
export TARGET_DISK_SIZE=100
export SUBSCRIPTION_USER=<RedHat subscription username>
export SUBSCRIPTION_PASSWORD=<RedHat Subscription password>
export IMAGE_BUILD_DIRECTORY=/data/tmp
export IMAGE_PREP_SCRIPT=image-prep.template

pvsadm image qcow2ova --image-name ${IMAGE_NAME} \
    --image-url ${QCOW2_IMAGE_PATH} \
    --image-dist rhel \
    --image-size ${IMAGE_SIZE} --target-disk-size ${TARGET_DISK_SIZE} \
    --rhn-user ${SUBSCRIPTION_USER} \
    --rhn-password ${SUBSCRIPTION_PASSWORD}  \
    --temp-dir ${IMAGE_BUILD_DIRECTORY} \
    --prep-template ${IMAGE_PREP_SCRIPT}

The pvsadm tool has a help menu with detailed descriptions of usage, commands and flags. Run the following command to access the help menu.

pvsadm --help

Upload the custom-created OS image to Cloud Object Storage (COS).

Importing OS image in the Power Virtual Server workspace

To import your custom boot image, see the guidance for Importing a boot image.

Creating an Power Virtual Server instance from the new imported OS image

To provision a new Power Virtual Server instance follow the steps that are described in Using a custom boot image to provision a new instance.