Deploying IBM Cloud VPC infrastructure for Power Virtual Server SAP workloads

Deploying access host in management VPC

Access host is a mandatory component in the SAP on Power Virtual Server best practices. By this virtual server instance, the floating IP address is activated. You can then log in to the environment by using SSH. You need to set up a VPN access as a separate step and disable the floating IP address for a more secure environment. Use the following steps to deploy the access host.

1. Create a security group for management services. For more information, see Configuring the security group for the instance. Specify following rules.

   Table 1. Overview of the inbound security group rules for the access host

   Protocol	Source type	Source	Value	Rule's purpose
   ALL	CIDR Block	10.0.0.0/8	n/a	Address block of IBM Cloud VPC and Power Virtual Server management subnets
   ALL	CIDR Block	161.26.0.0/16	n/a	Address block of OS registration servers
   TCP	CIDR Block	0.0.0.0/0	Port 22-22	Address block allowed for host administration over SSH

   Table 2. Overview of the outbound security group rules for the access host

   Protocol	Source type	Source	Value	Rule's purpose
   ALL	CIDR Block	0.0.0.0/0	n/a	Allow all connections initiated from the hosts

2. Create an ACL to restrict the access to the management VPC's subnets.

   Table 3. Overview of the inbound ACL rules for the access host

   Rule priority	Allow or deny	Protocol	Source	Destination	Value	Rule's purpose
   1	Allow	ALL	161.0.0.0/8	10.0.0.0/8	n/a	address block of OS registration servers
   2	Allow	TCP	Any IP, any port	10.10.10./24, ports 22-22	n/a	address block for host administration over SSH
   3	Allow	ALL	10.0.0.0/8	10.0.0.0/8	n/a	address block that is allowed for IBM Cloud VPC and Power Virtual Server management subnets
   4	Deny	ALL	Any IP	Any IP	n/a	Deny all other inbound traffic

   Table 4. Overview of the outbound ACL rules for the access host

   Rule priority	Allow or deny	Protocol	Source	Destination	Value	Rule's purpose
   1	Allow	ALL	10.0.0.0/8	161.0.0.0/8	n/a	address block of OS registration servers
   2	Allow	TCP	10.10.10./24, ports 22-22	Any IP, any port	n/a	address block for host administration over SSH
   3	Allow	ALL	10.0.0.0/8	10.0.0.0/8	n/a	address block that is allowed for IBM Cloud VPC and Power Virtual Server management subnets
   4	Deny	ALL	Any IP	Any IP	n/a	Deny all other outbound traffic

3. Create a virtual server instance with a Linux operating system. For more information, see Creating a virtual server instance.

   • You can select any available Linux image, but you need to use the same operating system release for all of the virtual server instances in the landscape. We verify the setup with newest versions of Red Hat Enterprise Linux (starting with RHEL 8.4) and of Suse Linux Enterprise Server (starting with SLES 15 SP3).
   • You don't have limitations as to which Compute profile, SAP certification, or storage and network performance is used. You can use the smallest profile with 2 virtual CPUs and with 4 GB memory.

4. Attach the security group to the virtual server instance that you created in the previous step. Detach the default security group.

5. Enable the floating IP address on the access host. For more information, see Reserving a floating IP address.

6. After instance status changes to Running, verify that you can successfully log in on the access host.

We recommend that you use two extra SSH client parameters to get a more reliable SSH connection: ServerAliveInterval=60 and ServerAliveCountMax=600. If you use a nondefault path to your SSH key, you must specify it by following SSH client parameter: -i \<path to your SSH private key\>.

SSH command example:

ssh -A -o ServerAliveInterval=60 -o ServerAliveCountMax=600 root@\<access_host_floating_ip\>

Optional steps on IBM Cloud VPC for management services

To create other VPC resources, such as ACL, load balancer, and other resources, see Using the IBM Cloud console to create VPC resources.

Deploying host for internet services in edge VPC

Host for internet services is a mandatory component in the SAP on Power Virtual Server best practices. This virtual server instance hosts mandatory proxy server. Use the following steps to deploy a host for basic management services.

The following example setup uses open ports for standard SQUID proxy server that is provided by Linux distributions. If you use another proxy software or custom configurations, setup might vary.

1. Create a security group for basic management services as described Configuring the security group for the instance. Specify following rules:

   Table 5. Overview of inbound security group rules for the internet services host

   Protocol	Source type	Source	Value	Rule's purpose
   ALL	CIDR Block	10.0.0.0/8	n/a	address block of IBM Cloud VPC and Power Virtual Server management subnets
   TCP	CIDR Block	0.0.0.0/8	Port 80-80	HTTP
   TCP	CIDR Block	0.0.0.0/8	Port 443-443	HTTPS
   TCP	CIDR Block	0.0.0.0/8	Port 8443-8443	HTTPS (used by OS registration)
   ALL	CIDR Block	161.26.0.0/16	n/a	address block of OS registration servers
   UDP	CIDR Block	0.0.0.0/0	Port 53-53	DNS

   Table 6. Overview of outbound security group rules for the internet services host

   Protocol	Source type	Source	Value	Rule's purpose
   ALL	CIDR Block	0.0.0.0/0	n/a	Allow all connections initiated from the hosts

2. Create an ACL to restrict the access to the edge IBM Cloud VPC's subnets.

   Table 7. Overview of ACL inbound rules for the internet services host

   Rule Priority	Allow or deny	Protocol	Source	Destination	Value	Rule's purpose
   1	Allow	ALL	161.0.0.0/8	10.0.0.0/8	n/a	address block of OS registration servers
   2	Allow	UDP	Any IP, ports 53-53	10.0.0.0/8, any port	n/a	DNS
   3	Allow	ALL	10.0.0.0/8	10.0.0.0/8	n/a	address block that is allowed for IBM Cloud VPC and Power Virtual Server management subnets
   4	Allow	TCP	Any IP, ports 8443-8443	10.0.0.0/8, any port	n/a	HTTPS for OS registration
   5	Allow	TCP	Any IP, ports 443-443	10.0.0.0/8, any port	n/a	HTTPS
   6	Allow	TCP	Any IP, ports 80-80	10.0.0.0/8, any port	n/a	HTTP
   7	Deny	ALL	Any IP	Any IP	n/a	Deny all other inbound traffic

   Table 8. Overview of ACL outbound rules for the internet services host

   Rule priority	Allow or deny	Protocol	Source	Destination	value	Rule's purpose
   1	Allow	ALL	10.0.0.0/8	161.0.0.0/8, any port	n/a	address block of OS registration servers
   2	Allow	UDP	10.0.0.0/8, any port	Any IP, ports 53-53	n/a	DNS
   3	Allow	ALL	10.0.0.0/8	10.0.0.0/8	n/a	address block that is allowed for IBM Cloud VPC and Power Virtual Server management subnets
   4	Allow	TCP	10.0.0.0/8, any port	Any IP, ports 8443-8443	n/a	HTTPS for OS registration
   5	Allow	TCP	10.0.0.0/8, any port	Any IP, ports 443-443	n/a	HTTPS
   6	Allow	TCP	10.0.0.0/8, any port	Any IP, ports 80-80	n/a	HTTP
   7	Deny	ALL	Any IP	Any IP	n/a	Deny all other outbound traffic

3. Provision a virtual service instance with Linux. For more information, see Creating a virtual server instance.

   • You can choose any of available Linux distributions. We recommend that you use the same OS release for all virtual server instances in the landscape. We verify the setup with the newest versions of Red Hat Enterprise Linux (starting with RHEL 8.4) and of Suse Linux Enterprise Server (starting with SLES 15 SP3).
   • One of the profiles with 2 or 4 virtual CPUs and with 4 GB or 8 GB of memory would be sufficient in general. If you have stronger performance requirements, by using SQUID cache, you can choose a profile with more virtual CPUs or memory.
   • We recommend that you attach extra storage to the instance, locate SAP installation files on this separate disk, and export them over NFS. The disk size must be large enough to host all the data from IBM Cloud Object Storage that is relevant for all the installations and setups.

4. Attach the security group to the virtual server instance that you created in the previous step. Detach the default security group.

5. After some time, instance becomes status 'running'.

Deploying host for workload services in workload VPC

Host for workload services is a mandatory component in the SAP on IBM Power Virtual Services best practices. This virtual server instance hosts optional NFS server, NTP forwarder, DNS forwarder, and LDAP server. If you have already some of these services in your environment that are accessible from Power Virtual Server networks, then you can use them and do not need to install here. Execute following steps to deploy host for private services.

By the using the following setup, we open ports for standard widely used management components: NFS service provided by Linux distributions, NTP service over 'chrony', DNS service over 'bind' and open LDAP. If you use another software solutions or custom configurations, set up might vary.

1. Create a security group for private services. For more information, see Configuring the security group for the instance. Specify following rules:

   Table 9. Overview of the inbound security group rules for the workload services host

   Protocol	Source type	Source	Value	Rule's purpose
   ALL	CIDR Block	10.0.0.0/8	n/a	address block of IBM Cloud VPC and Power Virtual Server management subnets
   ALL	CIDR Block	161.26.0.0/16	n/a	address block of OS registration servers
   UDP	CIDR Block	0.0.0.0/0	Port 53-53	DNS

   Table 10. Overview of the outbound security group rules for the workload services host

   Protocol	Source type	Source	Value	Rule's purpose
   ALL	CIDR Block	0.0.0.0/0	n/a	Allow all connections initiated from the hosts

2. Create an ACL to restrict the access to the Workload VPC's subnets.

   Table 11. Overview of the inbound ACL rules for the workload services host

   Rule priority	Allow or deny	Protocol	Source	Destination	Value	Rule's purpose
   1	Allow	ALL	161.0.0.0/8	10.0.0.0/8	n/a	address block of OS registration servers
   2	Allow	UDP	Any IP, ports 53-53	10.0.0.0/8, any port	n/a	DNS
   3	Allow	ALL	10.0.0.0/8	10.0.0.0/8	n/a	address block that is allowed for IBM Cloud VPC and Power Virtual Server management subnets
   4	Deny	ALL	Any IP	Any IP	n/a	Deny all other inbound traffic

   Table 12. Overview of the outbound ACL rules for the workload services host

   Rule priority	Allow or deny	Protocol	Source	Destination	Value	Rule's purpose
   1	Allow	ALL	10.0.0.0/8	161.0.0.0/8, any port	n/a	address block of OS registration servers
   2	Allow	UDP	10.0.0.0/8, any port	Any IP, ports 53-53	n/a	DNS
   3	Allow	ALL	10.0.0.0/8	10.0.0.0/8	n/a	address block that is allowed for IBM Cloud VPC and Power Virtual Server management subnets
   4	Deny	ALL	Any IP	Any IP	n/a	Deny all other outbound traffic

3. Create a virtual server instance by using a Linux image. For more information, see Creating a virtual server instance.

   • You can choose any of available Linux distributions. We recommend that you use the same OS release for all virtual server instances in the landscape. We verify the setup with newest versions of Red Hat Enterprise Linux (starting with RHEL 8.4) and of Suse Linux Enterprise Server (starting with SLES 15 SP3).
   • Analyze your DNS and LDAP performance requirements and ensure you choose an appropriate profile with enough virtual CPUs and memory.
   • We recommend attaching an extra storage disk to the instance, locate SAP installation files on this separate disk and export them over NFS. The disk size must be large enough to host all the data from IBM Cloud Object Storage that is relevant for all the installations and setups in this landscape.

4. Attach the security group to the virtual server instance that you created previously. Detach the default security group.

5. After some time, instance becomes status 'running'.

Setup SQUID proxy server

1. Ensure all required ports in the security group in IBM Cloud VPC for edge services that are used by host for internet services are open. The needed ports are configured for internet services. Required ports for SQUID proxy are used by Power Virtual Server services. For more information, see Creating a proxy.

2. Log in to the internet services instance. SSH command example:

   ssh -A -o ServerAliveInterval=60 -o ServerAliveCountMax=600 -o ProxyCommand=\"ssh -W %h:%p root@\<access_host_floating_ip\>\" root@\<internet_services_host_private_ip\>
   

3. Ensure that the squid software is available. On SUSE: zypper update -y; zypper install -y squid. On RHEL: yum update -y; yum install epel-release; yum install -y squid.

4. Modify the SQUID configuration and add the rules relevant for OS update registration of virtual instances that run in Power Virtual Server. For more information, see proxy configuration documentation.

5. Enable and restart SQUID service: systemctl stop squid; systemctl start squid; systemctl enable squid.

Setup SQUID proxy server -- Ansible

To perform the previous steps through ansible automation, download ansible-galaxy collection ibm.power_linux_sap.

ansible-galaxy collection install ibm.power_linux_sap

After the ansible collection is installed, to set up proxy, update the variable file playbook/vars/sample_services_variable_file.yml

server_config: {
  squid: { enable: true }
}

After the file is updated, run the following ansible-playbook command.

ansible-playbook \--connection=local -i \"localhost,\" powervs-services.yml -e \@vars/sample_services_variable_file.yml

This ansible execution ensures that the squid proxy server is configured on host for workload services.

Setting up NFS server for SAP installation files

1. Ensure all required ports in the security group in IBM Cloud VPC for workload services that are used by host for private services are open. The needed ports are configured for private services. Required ports for NFS server are: 111 (TCP and UDP) and 2049 (TCP and UDP).

2. Log in to the workload services instance. SSH command example:

   ssh -A -o ServerAliveInterval=60 -o ServerAliveCountMax=600 -o ProxyCommand=\"ssh -W %h:%p root@\<access_host_floating_ip\>\" root@\<workload_services_host_private_ip\>
   

3. Make sure that the required NFS software is available. On SUSE: zypper update -y; zypper install -y nfs-utils. On RHEL: yum update -y; yum install epel-release; yum install -y nfs-utils.

4. Create a directory where NFS is mounted and export it.

5. Start NFS service and verify that the directory is exported. Use command showmount -e.

6. Make sure that the awscli software is available. This software is used later to download the software from S3 storage (IBM Cloud Object Storage). On SUSE: zypper update -y; zypper install -y aws-cli. On RHEL: yum update -y; yum install epel-release; yum install -y awscli.

Setting up NFS server for SAP installation files -- Ansible

To perform the previous steps through ansible automation, download ansible-galaxy collection ibm.power_linux_sap.

ansible-galaxy collection install ibm.power_linux_sap

After the ansible collection is installed, update the variable file playbook/vars/sample_services_variable_file.yml to set up proxy.

server_config: {
  nfs: { enable: true, nfs_directory: "/NFS;/hana/software" }
}

If you also want to install awscli, by using ansible, add update variable file playbook/vars/sample_services_variable_file.yml as

server_config: {
  nfs: { enable: true, nfs_directory: "/NFS;/hana/software" },
  awscli: { enable: true }
}

After file is updated, run the following ansible-playbook command.

ansible-playbook \--connection=local -i \"localhost,\" powervs-services.yml -e \@vars/sample_services_variable_file.yml

This ansible execution makes sure that the host for workload services acts as an NFS server and awscli package is installed.

Setting up NTP proxy and forwarder

1. Ensure all required ports in the security group in IBM Cloud VPC for workload services that are used by host for private services are open. The needed ports are configured for private services. Required port for NTP forwarder is: 123 (TCP).

2. Log in to the workload services instance. SSH command example:

   ssh -A -o ServerAliveInterval=60 -o ServerAliveCountMax=600 -o ProxyCommand=\"ssh -W %h:%p root@\<access_host_floating_ip\>\" root@\<workload_services_host_private_ip\>
   

3. Make sure that the required NTP software is available. On SUSE: zypper update -y; zypper install -y chrony. On RHEL: yum update -y; yum install epel-release; yum install -y chrony.

4. Modify file '/etc/chrony.conf' and add following lines. Replace \<pvs_mgmt_cidr\> with Power Virtual Server management CIDR block (network segment)

   local stratum 10
   manual
   allow \<pvs_mgmt_cidr \>
   

5. Enable and start chrony service: systemctl stop chronyd; systemctl start chronyd; systemctl enable chronyd;.

Setup NTP proxy or forwarder -- Ansible

To perform previous steps through ansible automation, download ansible-galaxy collection ibm.power_linux_sap.

ansible-galaxy collection install ibm.power_linux_sap

After the ansible collection is installed, to set up proxy, update the variable file playbook/vars/sample_services_variable_file.yml.

server_config: {
  ntp: { enable: true }
}

After the file is updated, run the following ansible-playbook command.

ansible-playbook \--connection=local -i \"localhost,\" powervs-services.yml -e \@vars/sample_services_variable_file.yml

This ansible execution ensures that the NTP proxy is configured on the host for workload services.

Setting up IBM Cloud-native DNS service (DNS option 1)

You can use IBM Cloud DNS service that is directly reachable from IBM Power Virtual Servers over custom resolver. For more information, see the following links.

• Setting up an instance
• Managing DNS zones
• Managing permitted networks
• Managing DNS records
• Configuring custom resolver. Specify VPE subnet of IBM Cloud VPC for workload services as location in the resolver.

Make sure that all required ports in the security group in IBM Cloud VPC for workload services that are used by host for private services are open. The needed ports are configured for private services. Required port for DNS is: 53 (TCP and UDP). As result, you get a private IP address that you can specify in Power® Virtual Server instances as DNS endpoints by private subnet configuration. These IP addresses are entered into /etc/resolf.conf in the operating system.

Setting up DNS forwarder (DNS option 2)

If you use your own DNS service, its IP must be reachable from VPC for workload services. If the IP is not directly reachable from IBM Power Systems Virtual Servers, a DNS forwarder on the host for critical management services is required. Use the following steps to complete the configuration:

1. Make sure that all required ports in the security group in IBM Cloud VPC for workload services that are used by host for private services are open. The needed ports are configured for private services. Required port for DNS is: 53 (TCP and UDP).

2. Log in to the private services instance. You can use the following SSH command example:

   ssh -A -o ServerAliveInterval=60 -o ServerAliveCountMax=600 -o ProxyCommand=\"ssh -W %h:%p root@\<access_host_floating_ip\>\" root@\<workload_services_host_private_ip\>
   

3. Make sure that the required DNS software is available. On SUSE: zypper update -y; zypper install -y bind. On RHEL: yum update -y; yum install epel-release; yum install -y bind.

4. Modify file /etc/named.conf:

   • Add the following lines at the beginning of the file direct after the starting comment (before the options section starts). Replace <pvs_mgmt_cidr> with Power Virtual Server management CIDR block (network segment)

     acl allowed_clients {
     localhost;
     10.10.0.0/16;
     };
     

   • Add the following lines at the beginning of the options section. Both 161.26.0.x IPs are the IBM Cloud DNS servers. 9.9.9.9 IP is the IBM public DNS server IP. Replace or extend this IP list with DNS servers of your choice.

     forwarders {
     161.26.0.7;
     161.26.0.8;
     9.9.9.9;
     };
     recursion yes;
     allow-query { allowed_clients; };
     forward only;
     

5. Enable and start DNS service: systemctl stop named; systemctl start named; systemctl enable named;.

Setup DNS forwarder (DNS option 2) -- Ansible

To perform the previous steps through ansible automation, download ansible-galaxy collection ibm.power_linux_sap.

ansible-galaxy collection install ibm.power_linux_sap

After the ansible collection is installed, to set up proxy, update the variable file playbook/vars/sample_services_variable_file.yml

server_config: {
  dns: { enable: false, dns_servers: "161.26.0.7; 161.26.0.8; 9.9.9.9;" }
}

After the file is updated, run the following ansible-playbook command.

ansible-playbook \--connection=local -i \"localhost,\" powervs-services.yml -e \@vars/sample_services_variable_file.yml

This ansible execution ensures that the DNS forwarder services are configured on Power Virtual Server.