Managing Virtualization Service clusters

Virtual Private Cloud 4.21 and later Bare metal worker nodes only RHCOS only

Learn how to manage your OpenShift Virtualization Service cluster, including working with pre-configured components, managing worker nodes, and performing maintenance tasks.

Understanding managed components

Virtualization Service clusters include several pre-configured components that are managed differently than in standard OpenShift clusters.

Core components (cannot be disabled)

The following components are essential to Virtualization Service and cannot be disabled:

OpenShift Virtualization add-on: The openshift-virtualization add-on is automatically enabled on all Virtualization Service clusters and cannot be disabled. This add-on manages the installation and updates of the OpenShift Virtualization, NMState, and Node Maintenance operators. For more information, see Managing the OpenShift Virtualization add-on.
OpenShift Virtualization Operator: Provides virtual machine management capabilities. This operator is automatically installed by the add-on and updated as part of the cluster lifecycle. Installation from Red Hat OperatorHub is blocked.
NMState Operator: Manages network configuration for virtual machines and nodes. This operator is automatically installed by the add-on.
Node Maintenance Operator: Handles node maintenance operations for virtual machine workloads. This operator is automatically installed by the add-on.
OpenShift Data Foundation (ODF): Provides storage for VM disks and enables live migration. ODF is pre-configured to use local NVME storage on bare metal nodes.

Viewing managed add-ons

List all add-ons in your cluster:

ibmcloud ks cluster addon ls --cluster CLUSTER_NAME

Example output:

Name                         Version   Health State   Health Status
ibm-storage-operator         1.0       normal         Addon Ready. For more info: http://ibm.biz/addon-state (H1500)
openshift-virtualization     4.21      normal         Addon Ready. For more info: http://ibm.biz/addon-state (H1500)

The openshift-virtualization add-on is automatically enabled and cannot be disabled on Virtualization Service clusters.

For detailed information about managing the OpenShift Virtualization add-on, including viewing details, checking versions, and updating, see Managing the OpenShift Virtualization add-on.

Managing worker nodes

Viewing worker nodes

List all worker nodes in your cluster:

ibmcloud ks workers --cluster CLUSTER_NAME

Or use the OpenShift CLI:

oc get nodes

Adding worker nodes

Add worker nodes to an existing worker pool:

ibmcloud ks worker-pool resize --cluster CLUSTER_NAME \
  --worker-pool default \
  --size-per-zone NUMBER_OF_WORKERS

All worker nodes in a Virtualization Service cluster must use supported bare metal flavors.

Replacing worker nodes

Replace a worker node:

ibmcloud ks worker replace --cluster CLUSTER_NAME --worker WORKER_ID

The replacement worker is provisioned with the same configuration as the original.

Reloading worker nodes

Before you reload a worker node, place the node into maintenance by using the Node Maintenance Operator or migrate running VMs to other nodes. For more information, see Placing a node into maintenance and Live migrating VMs.

Reload a worker node to apply updates or fix issues:

ibmcloud ks worker reload --cluster CLUSTER_NAME --worker WORKER_ID

Managing worker pools

Viewing worker pools

ibmcloud ks worker-pool ls --cluster CLUSTER_NAME

Creating additional worker pools

Create a new worker pool with a different bare metal flavor:

ibmcloud ks worker-pool create vpc-gen2 \
  --name POOL_NAME \
  --cluster CLUSTER_NAME \
  --flavor BARE_METAL_FLAVOR \
  --size-per-zone NUMBER_OF_WORKERS

All worker pools in a Virtualization Service cluster must use bare metal flavors that support the openshift-vs offering.

Adding zones to worker pools

Add a zone to an existing worker pool:

ibmcloud ks zone add vpc-gen2 \
  --cluster CLUSTER_NAME \
  --zone ZONE \
  --subnet-id SUBNET_ID \
  --worker-pool POOL_NAME

Updating the cluster

Checking for updates

Check if updates are available for your cluster:

ibmcloud ks cluster get --cluster CLUSTER_NAME | grep "Master Version"

View available versions:

ibmcloud ks versions --show-version openshift

Updating the cluster master

Update the cluster master to a new version:

ibmcloud ks cluster master update --cluster CLUSTER_NAME --version VERSION

The master update typically takes 30-60 minutes. During this time, you cannot access the Kubernetes API or OpenShift console.

Updating worker nodes

After updating the master, update worker nodes:

ibmcloud ks worker update --cluster CLUSTER_NAME --worker WORKER_ID

Or update all workers in a worker pool:

ibmcloud ks worker-pool update --cluster CLUSTER_NAME --worker-pool POOL_NAME

Before you update worker nodes, place each node into maintenance by using the Node Maintenance Operator or migrate running VMs to other nodes. For more information, see Placing a node into maintenance and Live migrating VMs.

Monitoring cluster health

Checking cluster status

ibmcloud ks cluster get --cluster CLUSTER_NAME

Look for:

State: Should be normal
Master Status: Should be Ready
Master Health: Should be normal

Monitoring component health

Check OpenShift Virtualization health:

oc get hyperconverged -n openshift-cnv

Check ODF health:

oc get storagecluster -n openshift-storage

Viewing cluster logs

View cluster activity:

ibmcloud ks cluster get --cluster CLUSTER_NAME --show-resources

For detailed logging, configure IBM Log Analysis. See Logging for clusters.

Managing virtual machines

Viewing virtual machines

List all VMs in the cluster:

oc get vms -A

View VMs in a specific namespace:

oc get vms -n NAMESPACE

Placing a node into maintenance

Before you perform maintenance actions such as updating, reloading, or replacing a bare metal worker node, put the node into maintenance mode. The Node Maintenance Operator cordons the node and automatically evicts or live-migrates all eligible virtual machine workloads to other nodes in the same zone without interrupting workloads.

If your cluster uses OpenShift Data Foundation (ODF), nodes that run ODF storage components must follow the ODF upgrade and maintenance procedures instead of this process. For more information, see Understanding OpenShift Data Foundation.

Starting node maintenance from the web console

You can initiate node maintenance directly from the Red Hat OpenShift web console.

In the OpenShift web console Administrator perspective, go to Compute > Nodes.
Find the bare metal worker node you want to perform maintenance on.
Click the actions menu (three vertical dots) for that node and select Start maintenance.
In the confirmation dialog, review the maintenance settings and click Start.
Verify that the node status displays as Scheduling disabled and that the actions menu shows Stop maintenance instead of Start maintenance. Wait until all VM instances migrate to other available nodes before proceeding with your node-level action (such as ibmcloud ks worker reload or ibmcloud ks worker update).
After your maintenance action completes and the node is healthy, return to Compute > Nodes, click the actions menu for the node, and select Stop maintenance.

Starting node maintenance from the CLI

You can also initiate node maintenance by creating a NodeMaintenance custom resource.

Create a YAML file named node-maintenance.yaml with the NodeMaintenance custom resource definition. Specify the target bare metal worker node name in the nodeName field.

apiVersion: nodemaintenance.medik8s.io/v1beta1
kind: NodeMaintenance
metadata:
  name: nodemaintenance-NODE_NAME
spec:
  nodeName: NODE_NAME
  reason: Node maintenance for update or reload

Apply the custom resource to put the node into maintenance mode:
```
oc apply -f node-maintenance.yaml
```
Monitor the status of the NodeMaintenance resource to verify that the drain operation succeeds:
```
oc get nodemaintenance nodemaintenance-NODE_NAME -o jsonpath='{.status.phase}'
```
Verify that the phase reports Succeeded before you proceed to reload, update, or replace the node.
Perform your planned node-level action, such as reloading or updating the worker node:
```
ibmcloud ks worker reload --cluster CLUSTER_NAME --worker WORKER_ID
```
After the node reload or update is complete and the node status in oc get nodes is Ready, remove the node from maintenance by deleting the NodeMaintenance resource:
```
oc delete nodemaintenance nodemaintenance-NODE_NAME
```

Live migrating VMs manually

If you want to manually trigger a live migration for an individual virtual machine instead of using the Node Maintenance Operator:

List the virtual machine instances in the namespace to identify the name of the VM you want to migrate:
```
oc get vmi -n NAMESPACE
```
Trigger a live migration for the VM:
```
virtctl migrate VM_NAME -n NAMESPACE
```

If a VM has a Virtual Network Interface (VNI) attached, live migration is supported only within the same zone. Migrating such a VM across zones succeeds, but the VM ends up with a broken network because VNIs cannot attach across zones.

Stopping and starting VMs

Stop a VM:

virtctl stop VM_NAME -n NAMESPACE

Start a VM:

virtctl start VM_NAME -n NAMESPACE

Storage management

Monitoring storage capacity

Check ODF storage capacity:

oc get cephcluster -n openshift-storage -o jsonpath='{.items[0].status.ceph.capacity}'

View storage usage:

oc get cephblockpool -n openshift-storage

Managing persistent volume claims

List PVCs used by VMs:

oc get pvc -A | grep virtualmachine

View PVC details:

oc describe pvc PVC_NAME -n NAMESPACE

Troubleshooting

For troubleshooting common issues with Virtualization Service clusters, see the following topics:

Troubleshooting clusters - Worker node issues, cluster access, and general cluster problems
Troubleshooting OpenShift Virtualization - Virtual machine issues, operator problems, and virtualization-specific errors
Troubleshooting storage - OpenShift Data Foundation and persistent volume issues

Managing Virtualization Service clusters

Managing Virtualization Service clusters

Understanding managed components

Core components (cannot be disabled)

Viewing managed add-ons

Managing worker nodes

Viewing worker nodes

Adding worker nodes

Replacing worker nodes

Reloading worker nodes

Managing worker pools

Viewing worker pools

Creating additional worker pools

Adding zones to worker pools

Updating the cluster

Checking for updates

Updating the cluster master

Updating worker nodes

Monitoring cluster health

Checking cluster status

Monitoring component health

Viewing cluster logs

Managing virtual machines

Viewing virtual machines

Placing a node into maintenance

Starting node maintenance from the web console

Starting node maintenance from the CLI

Live migrating VMs manually

Stopping and starting VMs

Storage management

Monitoring storage capacity

Managing persistent volume claims

Troubleshooting

Next steps