Storage design for Red Hat OpenShift Virtualization

Deploy OpenShift virtualization storage using Red Hat OpenShift Data Foundation with NVMe, File Storage for VPC, and IBM Cloud Object Storage for persistent volumes.

The key storage architecture elements are shown in the following diagram.

Red Hat OpenShift Virtualization on IBM Cloud Storage
Red Hat OpenShift Virtualization on IBM Cloud Storage

Storage options

Red Hat® OpenShift® Virtualization uses Kubernetes PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs) to manage storage. It supports multiple storage backends that include block storage (not available with bare metal servers), file storage, and local storage (only available with bare metal servers).

Red Hat OpenShift on IBM Cloud offers integrated add-ons for Red Hat OpenShift Data Foundation (ODF), block and file storage by using IBM Cloud resources.

Red Hat OpenShift Data Foundation (ODF)

Red Hat OpenShift Data Foundation (ODF) provides persistent, software-defined storage for containerized applications. It delivers highly available and scalable storage by combining object, block, and file storage under a unified platform. ODF offers features such as snapshots, replication, and scalable storage management and are integrated with the Red Hat OpenShift console and APIs. This integration help you manage storage across diverse workloads.

The primary storage option for Red Hat OpenShift Virtualization is Red Hat OpenShift Data Foundation. This highly available storage solution consists of several open source operators and technologies such as Ceph, NooBaa, and Rook. These operators are used to provision and manage file, block, and object storage for your clusters by using storage classes.

ODF abstracts your underlying storage, and you can use ODF to create file, block, or object storage claims from the same underlying raw block storage. In virtualization, ODF uses local NVMe disks on bare metal servers to create a performant virtualized storage layer, where your application data is replicated in multiples of 3 (typically) for high availability by default. ODF with Red Hat OpenShift Virtualization is especially critical if you want to use disaster recovery (DR) capabilities for your VM workloads.

For more information about ODF, see Understanding OpenShift Data Foundation.

ODF supports the following layers of encryption:

  • Cluster encryption - encrypts data at rest across the entire storage cluster.
  • In-transit encryption - secures data as it moves between nodes, pods, and clients.
  • Storage volume encryption - provides encryption for individual PersistentVolumes or storage volumes.

For more information about setting up storage volume encryption in Red Hat OpenShift Kubernetes Service, see Setting up encryption by using Hyper Protect Crypto Services.

IBM Cloud Object Storage

You can use IBM Cloud Object Storage with backup solutions, or other object storage needs. IBM Cloud Object Storage allows backup data to be stored outside of the ODF cluster in case a disaster occurs.

IBM Cloud Object Storage is managed by IBM Cloud, while object storage built in the ODF clusters is a self-managed on Red Hat OpenShift worker nodes. Depending on the use case, you can use both IBM Cloud Object Storage and Red Hat OpenShift.

File Storage for VPC

You can use File Storage for VPC, which is a network-attached storage with NFS support.

IBM Cloud File Storage for VPC is a persistent, fast, and flexible network-attached, NFS-based storage option. You can add IBM Cloud File Storage to your applications by using persistent volumes claims (PVCs). You can choose between predefined storage classes that provide the required capacity in GB and IOPS.

  • All file shares are provisioned with zonal availability.
  • All classes support cross-zone mounting.

Data on a file share is encrypted at rest with IBM-managed encryption by default. You can optionally use your own root keys to protect your file shares with customer-managed keys. For more information, see About File Storage for VPC and About File Storage for VPC > Securing your data.

For NFS-based file share needs, you can also use the NFS storage that is built into the ODF clusters. The key differences here are that File Storage for VPC is managed by IBM Cloud, while NFS storage is built in the ODF clusters and is on self-managed on Red Hat OpenShift managed worker nodes. The IOPS and GB settings are independent of your clusters. Depending on the use case, you can use both of these options.

For Red Hat OpenShift Virtualization workloads that use VPC File storage, keep the following considerations in mind:

  • Snapshots aren't supported.
  • Each Persistent Volume Claim (PVC) that was created with this provisioner provisions an NFS share and a mount target in your VPC.
  • A PVC can be mounted as a volume in multiple pods or virtual servers. However, it can't be shared across multiple virtual servers.
  • For virtual servers, one virtual server disk equals one PVC, which means that one NFS share per virtual server disk.

To deploy the File storage shares for VPC add-on on your Red Hat OpenShift cluster, see Enabling the IBM Cloud File Storage for VPC cluster add-on.

The add-on automatically installs the PersistentVolume provisioner vpc.file.csi.ibm.io and creates a set of StorageClasses that are named ibmc-vpc-file-*. Each option offers different IOPS tiers and varying reclaim and binding policies. For the full list of available StorageClasses and detailed explanations of their parameters, see Storage class reference.

Block Storage for VPC

Block storage for VPC is available for only virtual server worker nodes.

This add-on provisions hypervisor-mounted, high-performance, block-level data storage for your virtual server worker nodes by using Kubernetes persistent volume claims (PVCs). PVCs are used to store virtual server disks on IBM Cloud Block Storage volumes.

Data on a block volume is encrypted at rest with IBM-managed encryption by default. You can optionally use your own root keys to protect your file shares with customer-managed keys. For more information see, About Block Storage for VPC and About Block Storage for VPC > Securing your data.

Choosing your storage solution

Use the following table to select the right storage option based on your workload requirements. Multiple options can be combined within the same cluster.

Storage option selection by use case
Use case Recommended option Key constraints
Virtual machine boot and data disks, live migration, snapshots, DR ODF on bare-metal (local NVMe) Requires bare-metal worker nodes; minimum 3 nodes in the ODF storage pool (single-zone and flexible-scaling deployments can expand granularly; multi-zone deployments require multiples of 3); only replication pools supported on ROKS (erasure coding is not available on ROKS bare-metal)
Shared file system access across pods or VMs ODF NFS (CephFS-backed) or File Storage for VPC ODF NFS requires an enabled NFS gateway (adds ~3 CPU + 8 Gi RAM); File Storage for VPC does not support snapshots and cannot be shared across multiple virtual servers
Backup data or object workloads outside the ODF cluster IBM Cloud Object Storage Fully managed; billed independently; not suitable for block or file VM disk workloads
Block storage on virtual server (VSI) worker nodes Block Storage for VPC Available only on VSI worker nodes, not bare-metal
Shared read-only content or import sources File Storage for VPC One NFS share per PVC; supports cross-zone mounting; max 32 TB per PVC

ODF is the primary and recommended storage solution for Red Hat OpenShift Virtualization on IBM Cloud. Review the following requirements before you provision your cluster.

  • ODF requires bare-metal worker nodes that run Red Hat CoreOS. ODF storage clusters on Red Hat OpenShift Kubernetes Service do not support virtualized (VSI) worker nodes.
  • The ODF storage node pool requires a minimum of 3 nodes. Single-zone and flexible-scaling deployments can expand granularly beyond the initial 3. Multi-zone deployments must use multiples of 3 (3, 6, 9, …) to maintain balanced zone distribution. Non-multiples of 3 in a multi-zone topology create a zone imbalance that leads to uneven OSD weight distribution and suboptimal data placement.
  • Only replication pools (replica-3 and replica-2) are supported for production block storage on ROKS bare-metal. Erasure-coded pools for RBD are a developer preview feature (ODF 4.20+) and are not available for production use.
  • For optimal performance and stability, provision a dedicated storage worker pool for ODF and a separate compute worker pool for VM workloads. Colocating ODF daemons and VM workloads on the same nodes causes resource contention.

For step-by-step ODF configuration instructions after your cluster is provisioned, see Red Hat OpenShift Data Foundation (ODF) for virtual machine workloads.

Next steps

Now that you understand the storage design options for Red Hat OpenShift Virtualization, explore these related topics: