Creating an advanced single-site VMware reference architecture

VMWare environment overview

The advanced representative VMware environment that is outlined, consists of one data center that manages two separate clusters: management and capacity. The configuration can be set up by using a single four node cluster that depending on requirements. The management cluster contains the following virtual machines (VMs) that are used for managing the infrastructure:

• VMware vCenter Server 5.5 or 6.0 Appliance
• Microsoft Windows 2012 R2 Standard with Active Directory and Domain Name System (DNS) roles

The capacity cluster contains the resources and infrastructure that are needed to create and run VMs. For networking, the environment consists of three private, internal VLANs, and a single, public VLAN that is used for external communication. Table 1 specifies the VLAN types and VLAN names are used throughout the environment.

For shared storage, you can use OS Nexus QuantaStor, a single-tenant shared storage server, or IBM Cloud Endurance or Performance storage services. In either case, the shared storage device is used to store the VMs on both the management and capacity clusters. For more information about storage options, see IBM Cloud storage solutions.

The storage environment is configured to support NFS volumes.

Step 1 Ordering primary public and private VLANs

Note: VLAN orders are subject to review and approval. No specific conditions or requirements are in place that flags a VLAN order to be auto-approved. VLAN orders can be denied for various reasons.

Four VLANs are created in this step: one for management, one for storage, one for VMs, and one for public access. It is recommended that you create these four VLANs before you order your servers. Ordering the VLANs first makes sure that the servers are placed on the correct VLANs and in the correct data center.

The environment consists of five ESXi hosts (two for the management cluster and three for the capacity cluster) and one virtual server. The private management VLAN consists of a subnet with 16 IP addresses to support it. The primary private VLAN for storage and VM traffic consists of eight addresses since the environment contains a single storage server and VMs use a portable subnet. If a larger subnet range is needed for the management network, adjust the range by calculating the number of ESXi hosts and multiplying by 2. Also, make sure that you specify the data center for which these VLANs are created.

After you log in to IBM Cloud, open a support ticket for the management and virtual machine VLANs by selecting Support > Add Ticket. Complete the fields with the information that is in Table 2:.

After the VLANs are provisioned, make note of the VLAN numbers, subnet range, and gateway, and assign them to logical vSphere networks. You can use the worksheet in Appendix A: VLAN worksheet to record the VLAN and the associated information. For example:

Note: Do not continue to the next step until the VLANs are provisioned.

Step 2 Ordering portable private IP addresses

In Step 2, a request is made to create portable private subnets for management VMs, VM kernel storage access, and VMs in the capacity cluster. You need to determine how many addresses you need for each VLAN subnet. In the representative environment, order the following addresses:

• Management VLAN - 1x8 addresses /29 – One address for vCenter Appliance; one address for DNS and Active Directory.
• Storage VLAN - 1x16 addresses /28 – Create two subnets on the same VLAN for storage and two VM kernel ports on each ESXi hosts by using different subnets to access the shared storage devices.
• VM VLAN - 1x32 addresses /27 – These addresses are used to assign IP addresses to VMs in the capacity cluster.

When you order an amount, make sure that you consider how many IP addresses are needed within the next 30 days and 6 months. It is also important to note that IBM Cloud reserves three to four IP addresses per portable subnet block. In order to get one IP address, you must order 4 IP addresses to account for the three to four reserved addresses, or zero if the pod supports Hot Standby Router Protocol.

Use the following steps to order a block of portable IP addresses for each VLAN for each subnet that you want to create:

1. Select Account > Place an Order.
2. In the pop-up window, go to Network > Subnets / IPs > Order.
3. From drop-down menu, select Portable Private.
4. Select XX Portable Private IP address and click Continue. Note: XX specifies the number of IP addresses.
5. Select the VLAN to associate with IP address block and click Continue.
6. Complete filling out the information on the screen and click Continue.

Creating IP addresses is fairly quick and is displayed by selecting Subnets from Network > IP Management. You can record these IP addresses in the worksheet that is found in Appendix A: VLAN worksheet.

Step 3 Ordering a virtual server

A Windows 2012 R2 Standard virtual server is provisioned to use as a utility server for ISOs and provide access to the environment in this step.

1. Open a browser window and log in to IBM Cloud.
2. Select Account > Place an Order.
3. Go to Virtual Server > Hourly or Monthly.
4. Select the appropriate data center (where the VLANs were created) to provision the virtual server and specify the following specifications for each option:
   • Computing instance - 1x2.0 GHz Cores
   • RAM: 4 GB
   • Operating System: Windows Server 2012 R2 Standard Edition (64-bit)
   • First Disk: 100 GB (SAN)
   • Uplink Port Speeds - 1 Gbps Public and Private Network Uplinks
5. Click Continue Your Order and select the backend and fronted VLANs on the Order Summary and Billing screen. Note: The selection of VLANs is important so the utility can be placed in the correct pod within the data center. For the example environment, the backend VLAN is the management VLAN (1101) and the front-end VLAN is the public VLAN (2101)
6. Enter a host name and domain name for the server and click Place Order.

Step 4 Ordering ESXi hosts and gateway / firewall

You need to order the ESXi hosts and Brocade vRouter (Vyatta) gateway and firewall appliance while the virtual server is provisioned. Order all of these servers at the same time so they are placed in the same pod at the same time. Ordering hardware at separate times can cause hosts and firewalls to be in separate pods within an IBM Cloud data center.

Step 5 Trunking VLANs on BCS Switches

By default, ports are placed on the backend customer switches (that is, private network switch in a pod, such as a Backend Customer Switch (BCS)) in access mode. As a result, you need to trunk the ports that are attached to the ESXi hosts so the hosts can access storage and so the VMs can communicate on the private network.

Before you open the ticket to trunk the VLANs, you need to determine which network interfaces are on the private network. To determine the interface, go to the Device Details for each ESXi server and go to Network > Private. For this environment, eth0 and eth2 are the private network adapters.

In addition to trunking of the VLANs for the ESXi hosts, you must also unbond the NICs for the management and capacity hosts. You unbond the NICs because the Link Aggregation Control Protocol (LACP) is not compatible with software iSCSI multipathing.

To trunk the VLANs and unbond the NICs, you need to open a ticket by following these steps:

1. Open a browser window and log in to IBM Cloud.
2. Select Support, Add Ticket.
3. Enter the following information:
   • Subject: Private network question
   • Title: Trunk VLANs and unbond NICs
   • Details: Trunk VLANs <Management VLAN>, <Storage VLAN>, and <VM VLAN> on eth0 and eth2 NIC pair for the following hosts [list each ESXi host]. Also, unbond (remove LACP) the private NICs (eth0 and eth2) on the following servers: [list each ESXi host]
4. Click Add Ticket.

Make sure to change the VLANs that are designated in the <> with your actual VLANs

Step 6 Configuring management host networking

After the servers are provisioned and the VLANs are trunked, you need to set up networking on the hosts in the management cluster. For this configuration, you use vSphere standard switches for the management cluster. Except for the vMotion and fault tolerance port groups, you use portable IP addresses to configure the VM kernel port groups. Note: You use your own IP scheme for vMotion and fault tolerance because the traffic does not need to be routed. However, all hosts need to be on the same subnet as other hosts in the cluster to use vMotion and fault tolerance capabilities. If the subnet needs to be routed, it is recommended that you use IBM Cloud Portable IP addresses.

To configure the port groups, the vSphere client must be installed on the utility virtual server or the workstation that accesses the hosts via IBM Cloud management VPN.

1. Open a browser window and log in to IBM Cloud.
2. After you connect to the utility server or IBM Cloud VPN, start the vSphere client.
3. Enter the IP address, user name, and password of one of the management ESXi hosts. (You can find the root password for the ESXi host by selecting Device Details > Passwords.
4. Go to Configurations > Networking and create or modify the following port groups on the vSphere standards switch (most likely vSwitch0).

Complete these steps for each host in the management cluster.

Step 7 Download or upload ISO images and vCenter Virtual Appliance

Now, the environment is ready to deploy the VMware vCenter Virtual Appliance and install a virtual machine for DNS, Active Directory, or BIND. However, before you deploy, you need to download the images. To do download the images, remote desktop to the virtual server previously provisioned and download the appropriate images on the virtual server for your environment:

• Active Directory / Windows DNS: Windows Server ISO Image (Your licensed Media)
• Linux BIND: CentOS install image
• [vCenter Virtual Appliance]](https://my.vmware.com/web/vmware/info/slug/datacenter_cloud_infrastructure/vmware_vsphere/5_5){: external} (requires a valid VMware subscription)

After you download the ISO, you need to upload the image to a management host data store so it can be attached to a virtual machine. Use the vSphere client to connect to a management host and create an ISO directory on the local data store.

Step 8 Deploying DNS

You need to deploy a VM (on a management host) and install DNS services. Use the traditional vSphere client to create a VM on the Management ESXi host where the Windows or Linux ISO is located. Connect the appropriate ISO (Windows or CentOS) to deploy a DNS server on the VM. Make sure to associate the VM to the VM port group (vmpg)-management port group that was created in a previous step.

After the VM is installed, assign an IP address and default gateway from the portable private subnet group. If you are using the VLAN worksheet in Appendix A, the Management VM subnet. Update the Notes section of each Portable IP address with the name of the VM assigned. The Notes section can be located by going to the IBM Cloud infrastructure customer portal and selecting Network > IP Management > Subnets > [Subnet].

Although it is beyond the scope of this document to detail the steps that are needed to enable DNS, the following guidance is provided:

1. Set DNS Forwarding to the service.softlayer.com local DNS hosts:
   • rs1.service.softlayer.com 10.0.80.11
   • rs2.service.softlayer.com 10.0.80.12
2. After you set up DNS, create a local DNS zone (dal06.mycompany.local) and a reverse lookup zone for all portable and primary subnets that are provisioned.
3. Add an A HOST records for each host’s Management IP address (vmk0 on vmkPG-management).
4. Add an A HOST record from the portable private subnet that is bound to the management VLAN for your vCenter Virtual Appliance.
5. Update the Notes section of the Portable IP Subnet that you assigned to vCenter.

Step 9 Deploying and configuring vCenter Virtual Appliance

Now that DNS is configured, you can deploy and configure the vCenter Server Appliance. To deploy the appliance, follow these steps:

1. Remote desktop to the virtual server and open the vSphere Client.
2. Connect to a management host and select File, Deploy OVF Template.
3. Follow the wizard to complete the deployment.

Because no Dynamic Host Configuration Protocol (DHCP) server is available to assign the vCenter virtual appliance an IP address upon power-on, you must use the root console to configure the appliance. Note: A NO NETWORKING DETECTED message is displayed on the vCenter Virtual Appliance Console.

Follow these steps to configure the appliance:

1. Log in to the console with a User name of 'root' and a Password of 'vmware'.
2. Run /opt/vmware/share/vami/vami_config_net and follow the text wizard to configure the IP, subnet, and DNS properties. Remember to use the IP address of the DNS or BIND server that was created in Step 8: Deploy DNS.
3. Save the network settings, exit the console, and open a browser on the virtual server.
4. Go to the IP address that you gave to the vCenter virtual appliance (VCVA) appended with port 5480.
5. Accept the EULA in the wizard, answer the Customer Improvement Experience Program question, and select Configure Options, Set custom configuration.
6. Click Next and enter the following values:

   VCVA setup wizard

   Wizard Menu	Option	Value
   Database settings	Database type	Embedded
   SSO settings	SSO deployment type	Embedded
   SSO settings	New administrator password (for administrator@vsphere.local)
   SSO settings	Retype the new password
   Time synchronization	NTP synchronization	servertime.service.softlayer.com

7. Click Start. The VCVA is configured.
8. Change the root password by using the options under Admin.
9. Log out of the VCVA configuration web page.
10. Go to the vSphere Web Client by entering the IP address of the VCVA appended with 9443/vsphere-client.
11. Log in to the root and select Administration, Licenses.
12. Enter your VMware vCenter license.

Step 10 Create vCenter clusters and distributed virtual switch

Now that VCVA is configured and licensed, you can create the data center and cluster constructs and distributed virtual switches for the capacity cluster.

Step 11 Migrating ESXi hosts in capacity cluster to distributed virtual switch

Now that the capacity hosts are added to the capacity cluster, you can migrate the virtual standard switch configuration to the distributed virtual switch that you created in Step 10: Create vCenter Clusters & Distributed Virtual Switch. Migrating is done for one host and you apply a host profile later to configure the rest of the cluster.

Before you begin adding VMkernel adapters, assign the vmnics to the uplinks on the distributed virtual switch.

1. Click vCenter Inventory Lists, Distributed Switches.
2. Select the appropriate distributed switch for the capacity hosts.
3. Click Add and manage hosts on the Getting Started page.
4. Use the following settings to add uplinks and migrate the existing VMkernel that are associated with management of the host.

   DVS add hosts

   Menu	Field	Value
   Select task	Select task	Add hosts
   Select host	Click New hosts	Click the Capacity Host
   Select network adapter tasks	Select network adapter tasks	Select Manage physical adapters and Manage VMkernel adapters

5. Select one of the private vmnics and click Manage physical network adapters > Assign uplink.
6. Select uplink1 on the pop-up screen and click OK.
7. Repeat these steps for the other private vmnic and assign it to uplink2.
8. Click Next, highlight the vmk0 VMkernel adapter, and click Assign port group.
9. Select dvpg-Private-VM Management on the pop-up window and click OK.
10. Click Next twice and then click Finish to complete the migration to the distributed virtual switch. Note: You might briefly lose network connectivity to the host. The connection reestablishes quickly. After you migrate the vmk0 adapter to the distributed virtual switch, you can add VM kernels to each port group in the DVS.
11. Click Manage > Networking on the host within vCenter.
12. Go to VM Kernel adapters > Add host networking and add the VM kernel adapters that are in Tables 19 and 21. To add these adapters, you migrate to the “VM Kernel adapters” menu that is in the Manage > Networking tab on the host within vCenter. Then, click the “Add host networking” icon and add the following VM Kernel adapters.

Add vmk1 for vMotion

Add vmk2 for fault tolerance

Add vmk3 for storage

Creating a host profile

To create a host profile, you must capture it from the single capacity host that you previously configured.

1. Go to the vSphere Web Client home screen click the Host Profiles icon.
2. Click the green plus (+) sign, Extract profile from a host, and select the previously configured capacity host on the pop-up window.
3. Click Next.
4. Give the host profile (Capacity01 Host Profile) an appropriate name, enter a description, and click Next.
5. Review the settings and click Finish.

After you create the host profile, you need to modify it so that it does not prompt for MAC addresses when the profile is applied to the rest of the hosts in the capacity cluster.

1. Right-click the host profile that you created and select Edit Settings > Edit Host Profile, Host virtual NIC.
2. In the right pane, change Determine how MAC address for vmknic is decided to User must explicitly choose the policy option.
3. Click Next and then click Finish.

Attaching host profile to capacity cluster

Now that you created a host profile, you need to attach the host profile to the cluster. Profiles are attached so that they can be applied to the capacity hosts.

1. Go to the Host and Clusters view in the vSphere Web Client.
2. Enter maintenance mode for each host in the cluster. Note: Profiles can be applied only to hosts that are in maintenance mode.
3. Right-click the capacity cluster and select Attach Host Profile from the pop-up menu.
4. Select the host profile that you created and click Next.
5. Enter the appropriate information on the Customize host screen and click Finish.
6. Remove the hosts from maintenance mode (non-maintenance mode).

Step 12 Order, configure, and attach shared storage

Before you continue, it is imperative that you order, configure, and attach shared storage for use with the management and capacity clusters and hosts within the environment. If you want to use the IBM Cloud multi-tenant shared storage solution (File Storage), see Architecture Guide for IBM File Storage for IBM Cloud with VMware.

Step 13 Enabling HA/DRS and svMotion vCVA

Enabling HA/DRS on management and capacity clusters

Now that shared storage is set up, you need to enable HA and DRS to provide extra protection and load-balancing capabilities to the VMs on the management cluster.

1. Go to the vSphere web client.
2. Select Manage, Settings for the management cluster.
3. Select vSphere DRS and click Edit. Ensure that the following settings are selected: Turn on vSphere DRS, Automation Level - Fully Automated, Migration threshold slider setting - midway, virtual machine automation - Enable individual virtual machine automation levels., Power Management - Off.
4. On the vSphere HA Settings screen, ensure that the following settings are selected: Turn on vSphere HA, Host Monitoring, VM restart priority - Medium, Host isolation response - Leave powered on.
   Storage vMotion for the vCenter Virtual Appliance

Now that storage is set up on the management cluster and HA and DRS are enabled, you need to set the vCenter Virtual Appliance to shared storage.

1. Right-click the appropriate appliance and select Migrate from the pop-up menu.
2. Select Change data store and click Next.
3. Select the iSCSI volume that you previously mounted to the management cluster and click Next.
4. Review the selections and click Next.

The advanced single-site VMware environment is complete.

Summary

You now have a VMware environment that is running in an IBM Cloud data center. Your VMware environment can run production workloads and supplementing an on-premises IBM Cloud deployment. The environment enacts VMware best practices and enables features such as VMware DRS, HA, Storage DRS, and networking redundancy. You can extend this reference architecture implementation with greater capacity or management hosts and more storage.

For more information about VMware, see Deploy VMware and VMware FAQ

Appendix A: VLAN Worksheet