Replicated pools (default)

The default ODF configuration uses 3-way Replication. Every piece of data is stored as 3 copies across different nodes and protects against up to 2 simultaneous disk or node failures.

• Advantages: Simple architecture, fast reads, fast recovery, and predictable latency.
• Increased usage: 3x raw storage for each byte of usable data.

What happens when copies are lost (rep3):

The min_size parameter controls the minimum number of copies that must be available before Ceph allows I/O. ODF sets min_size=2 for rep3 pools by default and is enforced through requireSafeReplicaSize: true, which means that when 2 or 3 copies are available, reads and writes proceed normally. When only 1 copy is available, Ceph blocks I/O to prevent against further data inconsistency.

This behavior is a deliberate safety mechanism that prioritizes data integrity over availability.

The time that is between losing the second copy and completing rereplication is the most dangerous. During this time, a third failure causes permanent data loss. This reason is why capacity planning is important because Ceph rereplication speed depends on the available cluster bandwidth and free space. An overloaded or nearly full cluster takes longer to rereplicate which extends the vulnerability time.

For rep2 pools, the progression is more aggressive. If 1 copy is lost, only 1 copy remains. With min_size=2 (default), I/O is immediately blocked on affected PGs until the missing OSD returns or a new copy is replicated. With min_size=1, I/O continues on the single remaining copy, but a second failure results in permanent data loss.

The Red Hat OpenShift Kubernetes Service ODF add-on automatically creates only rep3 pools with ocs-storagecluster-cephblockpool. Rep2 pools are not created by the add-on. You must manually create a custom CephBlockPool with replicated.size: 2 and a corresponding StorageClass. For more information, see Creating a custom StorageClass for virtualization.

Because rep2 offers less fault tolerance than rep3, evaluate whether the storage savings justify the increased risk for your workload.

IBM recommends 3-way Replication for all production virtualization workloads to help ensure availability and data durability.

Erasure-coded pools

For environments where storage capacity efficiency is a priority, ODF also supports erasure-coded (EC) pools. EC splits data into k data chunks and m parity chunks reducing raw storage usage compared to replication while still providing fault tolerance.

Support status: Erasure coding for RBD and CephFS in ODF is a developer preview feature, first introduced in ODF 4.20. Developer preview features are not supported for production use. They are also not covered by Red Hat Customer Portal case management. Before ODF 4.20, only RGW (object storage) EC was available as a developer preview (from ODF 4.16). Although the underlying Ceph storage engine supports EC overwrites for RBD since the Luminous release (2017), the ODF operator and its managed deployment model do not yet certify EC pools for production block storage workloads. Plan to use replicated pools (rep2 or rep3) for all production VM storage, and evaluate EC pools only for nonproduction environments where the developer preview limitations are acceptable.

The following table compares all available pool types for reference.

The following lists show the key considerations for erasure coding.

• EC pools have a higher write latency than replicated pools because of parity computation and the need to write more chunks per operation.
• EC pools deliver competitive sequential read throughput, but might show reduced random write IOPS.
• The number of failure domains must be at least k+m. On a 3-node cluster, only rep2, rep3, and ec-2-1 are possible.
• For a 6-node cluster, all pool types that are listed previously are available.

Single replica pool (Nonresilient for dev and test only)

Starting with ODF add-on version 4.14, Red Hat OpenShift Kubernetes Service supports a single replica (rep1) pool through the addSingleReplicaPool parameter. This parameter creates a Ceph nonresilient block pool with no data replication with each block of data is stored one time.

To enable the single replica pool when you deploy ODF, use the following command

ibmcloud oc cluster addon enable openshift-data-foundation -c <cluster_name> \
  --version <version> \
  --param "addSingleReplicaPool=true"

This command creates an extra StorageClass:

• ocs-storagecluster-ceph-non-resilient-rbd: Single replica block storage with WaitForFirstConsumer volume binding.

The standard replica-3 pool (ocs-storagecluster-cephblockpool) is still created with it. The nonresilient pool is a separate, opt-in option.

A single replica pool has the following use cases.

• Development and test environments - where data durability is not critical and storage cost savings are prioritized.
• Applications with built-in replication - these applications manage their own data redundancy at the application layer. These applications maintain multiple copies across nodes, which means that storage-level replication is redundant.

The single replica pool provides zero fault tolerance. A single OSD or node failure results in permanent, unrecoverable data loss for all data on that OSD. No second copy is available for recovery. IBM Cloud documentation explicitly warns that this option increases the risk of data loss, data corruption, and potential system instability. For more information, see Ceph documentation.

Limitations:

• Block storage only: File storage is not supported by a single replica.
• Requires extra disks: At least one extra usable NVMe disk per node beyond what the replica-3 pool uses. Without this extra disk, the replica-1 OSDs do not start and the storage cluster remains in a progressing state.
• One pool per failure domain: ODF creates one nonresilient CephBlockPool per failure domain, with volumes that are bound by using WaitForFirstConsumer to validate data locality.
• Not recommended for virtual machine workload root disks: If the OSD that hosts a root disk fails, the virtual machine workload is permanently lost. Use the nonresilient pool for only disposable data disks in dev or test virtual machine workloads.

For production virtualization workloads, always use replica-3 (or at minimum replica-2) pools.

VMware vSAN migration comparison

For the teams that are migrating from VMware vSAN™, these Ceph pool types map to familiar vSAN storage policies:

Key differences from vSAN:

• Smaller host minimums: Ceph separates its cluster quorum from data placement. A rep3 pool needs only 3 hosts because each host stores one full copy. vSAN RAID-1 FTT=2 needs 5 hosts.
• rep2 matches standard vSAN RAID-1: Most vSAN deployments use FTT=1, which stores 2 copies. Ceph's rep2 is the direct equivalent.
• ec-3-1 matches vSAN RAID-5: Both use a 3+1 layout and achieve 1.33x usage, the most space-efficient option for single-failure tolerance.
• ec-4-2 matches vSAN RAID-6: Both use a 4+2 layout and achieve 1.5x usage with dual-failure tolerance.
• ec-2-1 and ec-2-2 have no direct vSAN equivalent: Smaller Ceph EC configurations with fewer data chunks, resulting in more usage per byte but requiring fewer hosts. They trade storage efficiency for smaller host count requirements.

Planning capacity

When you plan your ODF cluster, account for the raw storage usage of your chosen data protection policy. The usable capacity is less than the total raw NVMe capacity.

The usable capacity formula is Usable capacity = Total raw NVMe capacity / Replication or EC overhead factor.

See the following example of calculations for a 3-node cluster with 8 x 3.2 TB NVMe drives per node (76.8 TB raw total):

Virtual machine workload capacity estimation: A typical virtual machine workload with a 30 GB root disk and a 100 GB data disk uses 130 GB of usable storage. With rep3, that virtual machine workload requires 390 GB of raw storage. On the preceding 3-node cluster, you might provision approximately 196 virtual machine workloads of this size. In practice, keep Ceph usage less than 75% to maintain performance and support recovery operations.

Ceph performance degrades as cluster usage increases. ODF fires the CephOSDNearFull Prometheus alert when any OSD exceeds 75% usage. At 85%, Ceph sets the native nearfull OSD flag (mon_osd_nearfull_ratio) and ODF fires the CephOSDCriticallyFull alert. At 90%, Ceph stops backfill and recovery operations to the affected OSD (mon_osd_backfillfull_ratio). At 95%, Ceph marks the OSD full (mon_osd_full_ratio), blocks all writes, and issues HEALTH_ERR. Plan your capacity so that usage stays under 70% during normal operations, allowing headroom for data recovery and rebalancing during node maintenance or failures.

To check current cluster usage, run the following command:

oc exec -n openshift-storage $(oc get pods -n openshift-storage -l app=rook-ceph-tools -o name) -- ceph df

Worker node count and topology

The failure domain topology used by the IBM Cloud ODF add-on depends on your cluster configuration:

• Multi-zone clusters (3 availability zones): The failure domain is set to zone. Worker nodes are distributed across zones, and ODF must grow in multiples of 3 to keep zones balanced. For optimal availability, performance, and data safety, use 3, 6, or 9 nodes in the ODF storage cluster — each zone receives the same number of nodes.
• Single-zone clusters or clusters with fewer than 3 availability zones: Flexible scaling is automatically enabled and the failure domain is set to host. You can start with 3 nodes and add nodes one at a time.

All nodes that participate in the ODF storage cluster must be bare metal. Mixing virtualized and bare-metal nodes within the same ODF cluster is not supported.

For multi-zone clusters: adding nodes in counts that are not multiples of 3 creates a zone imbalance. With 4 nodes, one zone gets 2 nodes while the others get 1 each, resulting in uneven OSD weight distribution, suboptimal data placement, uneven usage, and partially idle OSDs.

For multi-zone clusters, always scale in multiples of 3 to maintain a balanced zone topology.

oc get nodes -l node-role.kubernetes.io/worker= \
  -o custom-columns='NAME:.metadata.name,RACK:.metadata.labels.topology\.kubernetes\.io/rack'

ODF subscription plans

Choose the plan that best fits your requirements:

• Essentials

  • Reduced cost
  • Internal-mode deployment only
  • Does not support disaster recovery, stretch clusters, or external-mode deployments
  • Best suited for test and development environments, proofs of concept, or small-scale deployments

• Advanced

  • Full feature set that includes disaster recovery, stretch clusters, external-mode deployment, advanced granular encryption, multi-cluster support
  • Recommended for production virtualization workloads with virtual machines

Both plans include BlueStore compression on block pools, thin provisioning, snapshots, and cloning. The differences between the plans relate to disaster recovery, encryption granularity, and deployment flexibility rather than storage efficiency features.

ODF supports deduplication only for object storage through the Multicloud Object Gateway (MCG). Block storage does not support deduplication. This support applies to both Essentials and Advanced plans. Upstream Ceph deduplication for RBD remains experimental and is not certified for use in ODF.

For more information, see ODF Essentials versus Advanced.

Storage type

• Select Local storage.
• Local storage uses the local NVMe instance storage available on bare metal worker nodes.
• NVMe drives provide reduced latency and high IOPS performance that is required for virtual machine workload disks.

ODF resource profile

ODF provides three resource allocation profiles that control the CPU and memory that is reserved for Ceph daemons.

• Lean: Minimum resource allocation. The Lean profile is suitable for resource-constrained environments, testing, development, and proofs of concept. Lean is not recommended for production virtualization workloads.
• Balanced: The default profile on Red Hat OpenShift Kubernetes Service. The Balanced profile provides a balance between resource consumption and performance for general-purpose workloads.
• Performance: Allocates more CPU and memory to Ceph daemons that reduce the risk of daemon-side bottlenecks. Best for high IOPS workloads, large numbers of virtual servers, and demanding applications.

For bare-metal deployments with local NVMe drives, use the Performance resource profile. Bare-metal nodes with 8 or more NVMe drives per node create a high OSD count per host. With the Balanced profile, Ceph daemon CPU and memory limits are often reached before the underlying NVMe hardware is saturated, which caps IOPS and increases latency. The Performance profile is the minimum recommended profile for production Red Hat OpenShift Virtualization workloads on bare-metal.

The resource requirements that are shown in the Red Hat OpenShift web console during ODF installation are dynamically computed based on the cluster OSD count. Therefore, clusters with more NVMe drives require proportionally more resources. The values are not fixed. Always verify the requirements displayed in the console for your specific cluster configuration.

The profile is selected during StorageSystem creation through the Configure Performance screen in the Red Hat OpenShift web console. Under-resourced Ceph daemons can become a hidden bottleneck, causing reduced IOPS or higher latency than the underlying storage hardware can deliver.

Match your bare metal server profiles to the resource requirements shown for your chosen profile: IBM Cloud VPC bare metal server profiles.

Slight oversubscription of resources is often acceptable on a bare metal server. However, never provision significantly less than the displayed minimum requirements, doing so degrades ODF performance and stability.

Number of OSD disks per node

• Determine the number of local NVMe drives that are available on each bare metal server.
• Ensure that the number of OSDs configured per node does not exceed the number of usable NVMe drives.
• The recommended pattern is 1 OSD per NVMe drive for optimal performance and failure isolation.
• Ensure that the number of OSD disks typically matches the number of NVMe drives per bare metal node. The storage capacity calculation that is shown in the UI does not reflect the actual usable capacity for local storage configurations and can be disregarded.

When you select nodes during StorageSystem creation, avoid selecting all nodes in the cluster. Selecting all nodes creates a LocalVolumeSet (LVS) with no nodeSelector. Any worker node that is added to the cluster in the future will be automatically detected by ODF, even if it is not intended for storage. Nodes that are detected unexpectedly must be manually removed from the LVS. To prevent this, select only the nodes in your dedicated storage worker pool, or ensure that those nodes carry the cluster.ocs.openshift.io/openshift-storage label before StorageSystem creation.

Default StorageClass for the cluster

After ODF deploys, the followingStorageClasses are typically created:

• ocs-storagecluster-ceph-rbd: Block storage
• ocs-storagecluster-cephfs: File storage
• ocs-storagecluster-ceph-rgw: Object storage

To allow workloads to automatically use high-performance ODF-backed persistent block storage without extra configuration, select Use Ceph RADOS block device (RBD) as the default storage class or manually set RBD (ocs-storagecluster-ceph-rbd) as the default StorageClass for the cluster after the ODF add-on is installed.

1. Mark RBD as default:

   oc patch storageclass ocs-storagecluster-ceph-rbd -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
   

2. If needed, remove default from the previously configured default:

   oc patch storageclass <previous-default-name> -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
   

ODF configuration checklist

• The cluster contains at least one bare metal server pool
• Red Hat OpenShift version 4.17 or greater
• The ODF add-on and operator are installed and running
• The bare metal servers have sufficient usable local NVMe drives
• The selected resource profile matches the node capacity
• ODF storage cluster uses a minimum of 3 nodes; multi-zone clusters must use multiples of 3 (3, 6, 9, …) to maintain zone balance
• All ODF participating nodes are bare metal
• The RBD StorageClass is created and preferably set as the default
• The ODF cluster status is Ready (oc get storagecluster -n openshift-storage)
• Ceph health is HEALTH_OK (oc -n openshift-storage rsh $(oc get pod -l app=rook-ceph-tools -o name) ceph status)

Prerequisites: Install the Red Hat OpenShift virtualization operator

Before you use Red Hat OpenShift Virtualization on IBM Cloud, verify that the Red Hat OpenShift virtualization operator is installed in your Red Hat OpenShift Kubernetes Service cluster.

The Red Hat OpenShift virtualization operator enables Kubernetes-native virtual machine workload management. It also provides the required controllers, CRDs, and integrations with storage and networking components.

For more information, see Red Hat OpenShift Virtualization on IBM Cloud.

Using ODF storage for virtual machine workloads

Red Hat OpenShift Data Foundation (ODF) provides persistent, software-defined storage for virtualization workloads that run on Red Hat OpenShift. When you use ODF as the storage backend for virtual machines, selecting the correct StorageClass is critical to ensure performance, stability, and full feature compatibility.

You must specify the appropriate StorageClass for the following situations:

• Virtual servers are created
• Virtual servers are imported or cloned
• Virtual servers are migrated to a Red Hat OpenShift Kubernetes Service cluster

Default virtualization StorageClass

When the Red Hat OpenShift Virtualization Operator is installed and an ODF cluster is available, a StorageClass optimized for virtualization workloads is automatically created:

• ocs-storagecluster-ceph-rbd-virtualization

This StorageClass is:

• Tuned for disk I/O patterns such as random reads, writes, and sustained throughput
• Validated for virtualization lifecycle operations such as start, stop, live migration, and snapshots
• Fully supported and recommended for production Red Hat OpenShift Virtualization environments

For most use cases, use this StorageClass without modification.

Live migration storage requirements

Live migration moves a running virtual machine workload from one worker node to another without downtime. For a successful live migration, the storage must be accessible from both the source and destination nodes simultaneously. Live migration requires the following configurations.

• ReadWriteMany access mode on virtual machine workload PVCs. Ceph RBD supports RWX in block mode, which is the default configuration for the ODF virtualization StorageClass.
• The ocs-storagecluster-ceph-rbd-virtualization StorageClass is pre-configured with ReadWriteMany support through RBD block mode. Virtual servers that use this StorageClass can live migrate without extra configuration.
• The generic ocs-storagecluster-ceph-rbd StorageClass uses the ReadWriteOnce access mode by default. Virtual servers that use RWO PVCs cannot perform live migration. The migration fails because the PVC cannot be mounted on the destination node while attached to the source.

If you create customStorageClasses for virtual servers that need live migration, verify that the PVCs are created with accessModes: [ReadWriteMany] and volumeMode: Block.

Live migration also requires:

• Configuring the Red Hat OpenShift Virtualization Operator with a suitable migration policy
• Verifying sufficient CPU and memory are available on the destination node

Virtualization-specific compared to generic RBD StorageClass

Although virtual machines can use a generic Ceph RBD StorageClass, the virtualization-specific StorageClass is optimized for the unique I/O and lifecycle characteristics of virtual machine workload disks.

Generic RBD StorageClasses remain suitable for container workloads, but virtualization-specific StorageClass is recommended for the production virtualization environments.

Creating a custom StorageClass for virtualization

In some scenarios, you might need a custom StorageClass to meet specific performance, resiliency, or capacity requirements.

To create a custom StorageClass requires, you must first create a custom CephBlockPool. When you create a custom pool, you must set targetSizeRatio on the pool. Without this setting, the Ceph placement group autoscaler assigns only 1 placement group to the pool. This assignment causes all I/O to bottleneck on a single OSD, resulting in worse performance than the default pool.

When you create a custom StorageClass for virtualization workloads, verify that the following parameters are configured correctly.

• Provisioner

  The StorageClass must use the Ceph RBD CSI provisioner-provided by ODF:

  openshift-storage.rbd.csi.ceph.com
  

  This provisioner enables dynamic provisioning of Ceph RBD volumes that are backed by the ODF cluster.

• Storage pool

  Specify the CephBlockPool that backs the virtual machine workload disks. You can choose one of the following options:

  • Default block pool. The default 3-way replicated Ceph block pool created by ODF:

    ocs-storagecluster-cephblockpool
    

  • Custom block pool. A user-defined CephBlockPool. The pool must include the following settings to avoid performance pitfalls:

      apiVersion: ceph.rook.io/v1
      kind: CephBlockPool
      metadata:
      name: my-custom-pool
      namespace: openshift-storage
     spec:
       failureDomain: zone          # Default for multi-zone clusters — data copies spread across zones; use host for single-zone flexible-scaling clusters
       deviceClass: ssd             # Match OSD device class
       enableCrushUpdates: true     # Keep CRUSH rules current on topology changes
       enableRBDStats: true         # Enable per-volume I/O monitoring
       replicated:
        size: 3
        requireSafeReplicaSize: true
            targetSizeRatio: 0.1       # CRITICAL — prevents 1-PG bottleneck
        ```
    
    The `targetSizeRatio` instructs the placement group autoscaler to proportionally preallocate placement groups based on the expected capacity share. Without it, the pool receives 1 PG and all I/O is funneled through a single OSD.
    
    

• Image features

  The StorageClass must include RBD image features that are critical for workload performance:

  imageFeatures: layering,deep-flatten,exclusive-lock,object-map,fast-diff
  

  RBD image features and their purpose

  Feature	Purpose
  exclusive-lock	Enables writeback caching and single-writer optimizations. Without this feature, write IOPS can be up to 7x worse.
  object-map	Enables bitmap tracking of allocated objects for sparse images.
  fast-diff	Accelerates snapshot diff and DataVolume clone operations for faster boot times.
  deep-flatten	Makes clones fully independent after they are flattened.
  layering	Enables copy-on-write cloning that is required for DataVolume cloning.

• Map options

  mapOptions: krbd:rxbounce
  

  This option fixes data corruption issues when you use the kernel RBD driver with Windows virtual servers. It forces the kernel to use a bounce buffer for received data to ensure compatibility. This option must be set on all workload StorageClasses.

• Complete custom StorageClass example

  apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    name: my-custom-virt-sc
  provisioner: openshift-storage.rbd.csi.ceph.com
  parameters:
    clusterID: <your-cluster-id>
    pool: my-custom-pool
    imageFormat: "2"
    imageFeatures: layering,deep-flatten,exclusive-lock,object-map,fast-diff
    mapOptions: krbd:rxbounce
    csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
    csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage
    csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
    csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage
    csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
    csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage
    csi.storage.k8s.io/fstype: ext4
  reclaimPolicy: Delete
  allowVolumeExpansion: true
  volumeBindingMode: Immediate
  

  To find the clusterID for your cluster, run the following command:

  oc get sc ocs-storagecluster-ceph-rbd -o jsonpath='{.parameters.clusterID}'
  

  For erasure-coded pools (developer preview only), see Understanding Data Protection, add dataPool that points to the EC pool, and keep pool pointing to the default-replicated pool:

  parameters:
    pool: ocs-storagecluster-cephblockpool   # Replicated pool for metadata
    dataPool: my-ec-pool                      # EC pool for data blocks
  

Compression

ODF supports BlueStore inline compression on Ceph block pools, which can reduce the raw storage that is used by the disks. Compression is applied transparently at the OSD layer, so the virtual machine workload and its guest OS are unaware that data is compressed.

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: compressed-block-pool
  namespace: openshift-storage
spec:
  failureDomain: zone          # Default for multi-zone clusters; use host for single-zone flexible-scaling clusters
  deviceClass: ssd
  enableCrushUpdates: true
  enableRBDStats: true
  replicated:
    size: 3
    requireSafeReplicaSize: true
    targetSizeRatio: 0.1
  parameters:
    compression_mode: "aggressive"

Encryption

ODF supports data-at-rest encryption at multiple layers that you can enable independently.

• IBM Cloud infrastructure encryption: Full-disk encryption on physical NVMe drives - managed by IBM Cloud.
• ODF cluster-wide encryption: All Ceph OSD disks are encrypted with dm-crypt at the device level. Enabled through encryption.clusterWide: true on the storage cluster CR. Protects against physical disk theft.
• ODF per-volume encryption: Individual RBD volumes are encrypted with LUKS2, each with its own data encryption key. Provides tenant isolation and granular key management.

On Red Hat OpenShift Kubernetes Service, ODF integrates with IBM Key Protect as the external key management service for both cluster-wide and per-volume encryption. When per-volume encryption is enabled, ODF automatically creates -encrypted StorageClass variants (for example, ocs-storagecluster-ceph-rbd-encrypted).

Ceph performance tuning for NVMe bare-metal

The default Ceph configuration is tuned for general workloads. The following parameter values are validated on the mx2d.metal.96x768 bare-metal profile and significantly increase IOPS and reduce latency for VM disk workloads on that profile. If you are using a different bare-metal profile, treat these as a starting reference and adjust the values based on the NVMe drive count and available CPU of your specific profile.

Apply each parameter by using the ceph config set command from the Ceph toolbox pod:

TOOLS_POD=$(oc get pod -n openshift-storage -l app=rook-ceph-tools -o name)

# OSD memory and shard tuning
oc exec -n openshift-storage ${TOOLS_POD} -- ceph config set osd osd_memory_target 8589934592
oc exec -n openshift-storage ${TOOLS_POD} -- ceph config set osd osd_op_num_shards_ssd 16
oc exec -n openshift-storage ${TOOLS_POD} -- ceph config set osd osd_op_num_threads_per_shard_ssd 2
oc exec -n openshift-storage ${TOOLS_POD} -- ceph config set osd bluestore_prefer_deferred_size_ssd 0

# RocksDB tuning
oc exec -n openshift-storage ${TOOLS_POD} -- ceph config set osd rocksdb_write_buffer_size 268435456
oc exec -n openshift-storage ${TOOLS_POD} -- ceph config set osd rocksdb_max_write_buffer_number 16
oc exec -n openshift-storage ${TOOLS_POD} -- ceph config set osd rocksdb_max_background_jobs 12

After applying, verify that the configuration was accepted:

oc exec -n openshift-storage ${TOOLS_POD} -- ceph config dump | grep -E "osd_memory_target|osd_op_num_shards|bluestore_prefer_deferred|rocksdb"

OSD pods do not need to be restarted for ceph config set changes. Ceph applies the configuration dynamically. However, BlueStore cache changes (osd_memory_target) take full effect only after each OSD pod is recycled. You can recycle OSD pods one at a time during a maintenance window without interrupting I/O.

Benchmark reference: Internal testing on a 3-node mx2d.metal.96x768 cluster with 200 VMs and unlimited IOPS showed that the combination of 16 shards, 8 GB OSD memory, and a 256 MB RocksDB write buffer achieved approximately 194,000 IOPS and 758 MB/s throughput. Increasing shards from 8 to 16 consistently produced the largest single improvement in IOPS across all test configurations.

Snapshot-based backup

ODF supports Kubernetes VolumeSnapshots for Ceph RBD volumes. To take a snapshot of a disk, use the following command:

oc apply -f - <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: my-vm-snapshot
spec:
  volumeSnapshotClassName: ocs-storagecluster-rbdplugin-snapclass
  source:
    persistentVolumeClaimName: my-vm-data-disk
EOF

VolumeSnapshots are copy-on-write and near-instant to create. You can use them to restore a virtual machine workload to a previous state or clone a disk. Red Hat OpenShift Virtualization also provides a built-in VM snapshot and restores API that captures the full virtual machine workload state that includes configuration and all disks, in a single operation.

Quiescing virtual machine workloads for application-consistent snapshots

When you take a snapshot of a running virtual machine workload, the data on disk must be in a consistent state. Without quiescing, the snapshot captures whatever is on disk at that instant, including partially written transactions, dirty buffers, and in-flight I/O. This process produces a crash-consistent snapshot, which might require application-level recovery on restore.

To achieve application-consistent snapshots, freeze the guest file system before the snapshot and thaw it afterward. Red Hat OpenShift Virtualization automates this process by using the QEMU guest agent.

The snapshot controller detects the QEMU guest agent. Before taking the snapshot, it issues a guest-fsfreeze-freeze command that halts all file systems I/O. The VolumeSnapshot is taken while the file system is frozen. After the snapshot completes, a guest-fsfreeze-thaw command resumes I/O. The snapshot status indicates the achieved consistency level. See the following table for the meaning of each status meaning.

Installing the QEMU guest agent is recommended for all production VMs. On Linux guests, use the following command.

# RHEL / CentOS / Fedora
sudo dnf install -y qemu-guest-agent
sudo systemctl enable --now qemu-guest-agent

# Ubuntu / Debian
sudo apt-get install -y qemu-guest-agent
sudo systemctl enable --now qemu-guest-agent

For Windows guests, install the VirtIO drivers package, which includes the QEMU guest agent service.

Custom freeze/thaw hooks for applications: For databases and other stateful applications that require extra quiescing beyond file system freeze, place custom hook scripts inside the guest virtual machine workload at /etc/qemu-ga/fsfreeze-hook.d/. These scripts are automatically run by the guest agent with a freeze argument before the file system is frozen and a thaw argument after the file system is thawed. Hook execution logs are written to /var/log/qga-fsfreeze-hook.log.

For example, the following PostgreSQL freeze hook can be placed at /etc/qemu-ga/fsfreeze-hook.d/postgresql.sh:

#!/bin/bash
case "$1" in
  freeze)
    sudo -u postgres psql -c "SELECT pg_backup_start('snapshot');" 2>/dev/null || true
    ;;
  thaw)
    sudo -u postgres psql -c "SELECT pg_backup_stop();" 2>/dev/null || true
    ;;
esac

VMware comparison: This example is analogous to the VMware pre-freeze and post-thaw script that is used with VMware Tools for application-consistent snapshots. The QEMU guest agent serves the same role as VMware Tools for snapshot quiescing.

Changed Block Tracking limitations

Changed Block Tracking enables incremental backups by identifying only the blocks that changed since the last backup. VMware's VADP (vStorage APIs for Data Protection) uses this mechanism to provide efficient incremental backups.

CBT is not available for ODF and Ceph RBD on Red Hat OpenShift Virtualization. Backups currently rely on full snapshots, which might result in longer backup windows and higher storage usage.

CBT development is in progress at multiple levels:

Although Ceph RBD supports the underlying rbd diff capability to identify changed blocks between snapshots, this capability is not yet exposed through the Kubernetes CSI Changed Block Tracking API. Until the full stack is in place (CSI CBT API + Ceph CSI driver support + KubeVirt integration), incremental backups at the block level are not available.

Backup solutions

Several backup vendors provide solutions for Red Hat OpenShift Virtualization that works within the current snapshot-based model:

• Veeam® Kasten: Kubernetes-native data protection with Red Hat OpenShift Virtualization support and incremental snapshot capabilities for ODF. For more information, see the Veeam Kasten reference architecture.
• Trilio for Kubernetes: Backup and recovery for Red Hat OpenShift Virtualization workloads with ODF integration. For more information, see Trilio Red Hat OpenShift Virtualization support.
• Veritas NetBackup: Enterprise backup with Red Hat OpenShift Virtualization support. For more information, see NetBackup - Comprehensive enterprise data protection.

Recommendations for VMware migrations

If your current VMware environment relies on CBT-based incremental backups, consider the following recommendations:

• Plan for full-snapshot backups. Evaluate your backup windows and storage requirements based on full VolumeSnapshots instead of incremental backups.
• Evaluate Kubernetes-native backup tools. Veeam Kasten and Trilio are designed for Kubernetes and Red Hat OpenShift Virtualization, and they work within the current snapshot model.
• Use Ceph snapshot efficiency. Ceph RBD snapshots are copy-on-write and use storage for only changed blocks after the snapshot is taken, which makes ongoing snapshot storage more efficient than full copies.

Monitor ODF and Ceph health

Regular monitoring of the ODF storage cluster is essential for maintaining availability and performance. The following section describes key commands and their outputs.

TOOLS_POD=$(oc get pods -n openshift-storage -l app=rook-ceph-tools -o name)
oc exec -n openshift-storage ${TOOLS_POD} -- ceph status

  cluster:
    id:     a1b2c3d4-...
    health: HEALTH_OK          ← What you want to see

  services:
    mon: 3 daemons             ← Should be 3 (quorum)
    mgr: 1 active              ← Manager daemon running
    osd: 24 osds: 24 up, 24 in ← All OSDs healthy (should match your NVMe count)

  data:
    pools:   4 pools, 353 pgs
    objects: 12.5k objects, 48 GiB
    usage:   152 GiB used, 69 TiB / 70 TiB avail   ← Cluster usage

oc exec -n openshift-storage ${TOOLS_POD} -- ceph health detail

oc exec -n openshift-storage ${TOOLS_POD} -- ceph osd tree

oc exec -n openshift-storage ${TOOLS_POD} -- ceph df

oc exec -n openshift-storage ${TOOLS_POD} -- ceph osd pool stats

Upgrading ODF on Red Hat OpenShift Kubernetes Service

The IBM Cloud Red Hat OpenShift Data Foundation (ODF) addon automatically applies z-stream updates within the same minor release. These updates are managed through IBM Cloud.

However, major and minor version upgrades (for example, 4.18 → 4.19) are not automatic. Follow the manual upgrade procedure to ensure data safety and cluster stability.

Updating ODF on a Red Hat OpenShift Kubernetes Service cluster consists of two main phases, both of which are required for a successful upgrade.

1. Upgrade or replace ODF worker nodes.

   • ODF relies on dedicated or labeled worker nodes to host storage components.
   • During a major or minor upgrade, these worker nodes must be upgraded or replaced to align with the target Red Hat OpenShift and ODF versions.
   • This process helps ensure that ODF pods (such as Ceph OSDs, MONs, and managers) are rescheduled correctly and continue functioning without data loss.
   • Ensure adequate capacity and node health before you start this step to maintain storage availability.

2. Update the ODF add-on.

   • After the worker nodes are upgraded or replaced, update the ODF add-on.
   • This step upgrades the ODF operators, CSI drivers, and related components to the target version.
   • After the add-on update completes, the cluster automatically reconciles the ODF resources and applies the required changes.

   Run post-upgrade validation to confirm:

   • ODF and Ceph cluster health
   • StorageClasses availability
   • Successful PVC read and write operations by applications

For more information, see Updating ODF on VPC clusters.

Expanding ODF Storage on Red Hat OpenShift Kubernetes Service

Accordingly, as your workloads grow and storage demands increase, it becomes essential to scale your storage infrastructure. Expansion in ODF is a key Day 2 operation that enables you to increase storage capacity, improve performance, and maintain resilience without disrupting running applications.

In IBM Cloud Red Hat OpenShift Kubernetes Service environments, expansion typically involves extending the storage worker pool. This operation is performed with minimal downtime, enabling seamless growth of your storage cluster.

1. Add worker nodes to your VPC cluster. For multi-zone clusters where the storage cluster spans 3 availability zones, add worker nodes in multiples of 3 to maintain zone balance (for example, 3, 6, or 9). For single-zone clusters with flexible scaling enabled, you can add nodes one at a time.

2. After the nodes are added, register them with ODF. If ODF runs on all worker nodes in your cluster, new nodes are added to the storage topology automatically. If ODF runs on only a subset of worker nodes, proceed to the next step.

3. If ODF runs on all of the worker nodes in your cluster, new worker nodes are added to the ODF storage cluster topology automatically. If ODF runs on only a subset of worker nodes, specify the private <workerNodes> parameters in your OcsCluster custom resource. Add the names of the new worker nodes to your ODF deployment by editing the custom resource definition. Modify OcsCluster custom resource as follows:

   • Find ocscluster

     oc get ocscluster
     

   • Edit ocscluster custom resource file and add new work nodes

     oc edit ocscluster <ocs cluster name> -o yaml
     

   • Save the OcsCluster custom resource file to reapply it to your cluster.

4. Increase the 'numOfOsd' value in your OcsCluster custom resource to enable OCS to deploy ODF components on newly added worker nodes and provision additional OSDs in the storage cluster.

   The adjustment to 'numOfOsd' depends on both the number of OSD disks per node and the number of nodes added. For example, if each node has 8 NVMe disks that are dedicated to OSDs, adding 3 nodes increases 'numOfOsd' by 8, while adding 6 nodes increases it by 16.

5. Verify the result by running the followwing command:

   oc exec -n openshift-storage ${TOOLS_POD} -- ceph osd tree
   

6. Verify that the new worker nodes are added and evenly distributed across each zone (for multi-zone clusters) or appear as individual host buckets (for single-zone flexible-scaling clusters), along with the corresponding number of OSDs assigned to each node.

For more information, see Expanding ODF by adding worker nodes to your VPC cluster.

Flexible scaling

The IBM Cloud ODF add-on uses different failure domain topologies depending on your cluster configuration:

• Multi-zone clusters (3 availability zones): The failure domain is set to zone. OSDs are provisioned in multiples of 3, one set per zone, to maintain data replication and high availability across zones. The storage cluster must grow in multiples of 3 to keep zones balanced.
• Single-zone clusters or clusters with fewer than 3 availability zones: Flexible scaling is automatically enabled. The failure domain is set to host, meaning each individual node is a distinct failure domain. You can add one node at a time and scale storage granularly.

Starting with ODF 4.21, flexible scaling behavior is determined automatically at initial deployment based on the cluster topology and cannot be changed afterward.

In a single-zone or flexible-scaling deployment, a replica-3 pool survives the loss of any single host. In a multi-zone deployment, a replica-3 pool survives the loss of an entire zone. Evaluate whether your cluster topology and its associated fault tolerance meet your resilience requirements before deploying ODF in production.

For the full set of add-on parameters and console-based installation steps, see Deploying OpenShift Data Foundation on VPC clusters.

Performance during node expansion: Adding a node to a flexible-scaling ODF cluster triggers Ceph data rebalancing. In the internal test described below, IOPS and throughput remained stable while write latency temporarily increased. In internal testing on a 3-node cluster with 100 VMs at 50,000 IOPS, the following results were observed:

Write latency returns to baseline after rebalancing completes. Plan node additions during periods of lower VM activity if your workloads are sensitive to write latency spikes.