Tuning ODF performance for virtualization workloads

Virtual Private Cloud 4.20 and later Bare metal worker nodes only

You can improve OpenShift Data Foundation (ODF) storage performance for virtualization workloads by selecting the appropriate performance profile, adjusting OSD pod resource limits, and configuring bulk data operations. These options apply to Red Hat OpenShift on IBM Cloud clusters with manually deployed OpenShift Virtualization and Red Hat OpenShift Virtualization Service clusters.

Before you begin

You must have cluster-admin access to the cluster.
Install or update the IBM Cloud CLI and the oc CLI.
ODF must be installed and in a healthy state before you adjust resource limits or configure storage pools. To verify ODF health, see Checking Ceph cluster health.

Selecting an ODF performance profile

ODF provides two performance profiles that control CPU and memory allocation for storage components. Choose the profile that best fits your workload.

Performance: Allocates more CPU and memory than the Balanced profile. Use this profile for VM workloads that need high throughput and low latency, such as databases or high-traffic applications.
Balanced: Allocates moderate CPU and memory. Use this profile for general-purpose workloads, mixed environments, or cost-optimized deployments.

When you deploy ODF, select Performance in the Resource profile field under Backing storage. On Virtualization Service clusters, Performance is selected by default.

You can set the profile from the console or CLI.

Console - Standard Red Hat OpenShift on IBM Cloud clusters: When you install the ODF add-on, select Performance in the Backing storage section.
Console - Virtualization Service clusters: Performance is selected by default. To change it, select Edit on the OpenShift Data Foundation card in the Virtualization integrations section during cluster creation.

This option applies only to standard Red Hat OpenShift on IBM Cloud clusters. For Virtualization Service clusters, you set the profile during cluster creation.

From the CLI, include --param "resourceProfile=performance" when you enable the add-on:

ibmcloud oc cluster addon enable openshift-data-foundation \
  -c <cluster-name> \
  --version <addon-version> \
  --param "odfDeploy=true" \
  --param "osdStorageClassName=localblock" \
  --param "autoDiscoverDevices=true" \
  --param "resourceProfile=performance" \
  --param "setDefaultStorageClassForVirtualization=true"

For all ODF add-on parameters, see OpenShift Data Foundation parameter reference.

Configuring OSD pod resource limits

Object Storage Daemon (OSD) pods store data and participate in data placement and replication. For virtualization workloads with high I/O, you can increase CPU and memory limits on OSD pods to help reduce bottlenecks.

Check current OSD resource limits

Before you modify resource limits, check the current CPU and memory settings for your OSD pods. Note the current requests and limits so that you can compare them with actual usage and the updated values later.

oc get pods -n openshift-storage -l app=rook-ceph-osd \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.containers[*]}  Container: {.name}{"\n"}    Requests - CPU: {.resources.requests.cpu}, Memory: {.resources.requests.memory}{"\n"}    Limits   - CPU: {.resources.limits.cpu}, Memory: {.resources.limits.memory}{"\n"}{end}{"\n"}{end}'

Review the output to identify the current CPU and memory requests and limits for each OSD container. Compare these values with actual usage in the next command. If CPU or memory usage consistently approaches the configured limits, increasing the limits might help reduce bottlenecks.

To check actual resource consumption, run the following command and compare CPU and memory usage with the limits you noted:

oc adm top pods -n openshift-storage -l app=rook-ceph-osd

Increase OSD resource limits

If the current limits are insufficient for your VM workload, update them by editing the ocs-storagecluster resource.

You can also modify limits for other Rook-Ceph pods, such as mon, mgr, and rgw, in the ocs-storagecluster configuration. For details, see Red Hat Solution 6959127.

Open the storage cluster resource for editing.

oc edit storagecluster ocs-storagecluster -n openshift-storage

In the relevant storageDeviceSets entry, add or update the resources field. The following partial example sets a limit of 4 CPUs and 24 Gi of memory, and a request of 2 CPUs and 24 Gi of memory:
```
storageDeviceSets:
  - name: ocs-deviceset
    resources:
      limits:
        cpu: "4"
        memory: "24Gi"
      requests:
        cpu: "2"
        memory: "24Gi"
```
CPU and memory requests must be less than or equal to the limits.
Save and exit the editor.

After you save the changes, the OSD pods restart automatically. Wait for the rolling restart to complete before you perform other storage operations.

Verify the updated resource limits

After the rolling restart completes, confirm that the updated limits are applied to all OSD pods.

Monitor the rolling restart to verify that all OSD pods return to a Running state. The restart is complete when all OSD pods show Running and none are Pending or Terminating.
```
oc get pods -n openshift-storage | grep osd | grep -v prepare | grep -v rotation
```

Confirm the new resource values are in effect.

oc get pods -n openshift-storage -l app=rook-ceph-osd \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.containers[*]}  Container: {.name}{"\n"}    Requests - CPU: {.resources.requests.cpu}, Memory: {.resources.requests.memory}{"\n"}    Limits   - CPU: {.resources.limits.cpu}, Memory: {.resources.limits.memory}{"\n"}{end}{"\n"}{end}'

Verify that the CPU and memory values match the values you configured.

Configuring the bulk flag for large data operations

For large data operations, such as VM migrations, bulk imports, or data archival, enabling the bulk flag on a Ceph block pool can improve initial data distribution across OSDs and reduce rebalancing overhead.

The bulk flag is commonly used for:

VM disk migrations and imports involving multiple TBs of data.
Backup and restore operations.
Initial data loads for new applications.
Data archival pools.

To configure a CephBlockPool resource with the bulk flag enabled, complete the following steps.

Create or update the CephBlockPool resource definition so that the parameters section includes bulk: "true".

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: <pool-name>
  namespace: openshift-storage
spec:
  replicated:
    size: 3
  parameters:
    bulk: "true"

Apply the configuration file.
```
oc apply -f <pool-config-file>.yaml
```
After you apply the configuration, Ceph can distribute new data more evenly across the pool from the start. This behavior can reduce rebalancing as the pool fills.

Checking Ceph cluster health

Regularly monitor your Ceph cluster to identify performance issues and ensure data integrity. Run health checks before and after you make configuration changes.

Run a basic health check

Run the following command to get an overall Ceph health summary. A healthy cluster returns HEALTH_OK.

oc rsh -n openshift-storage $(oc get pods -n openshift-storage -l app=rook-ceph-tools -o name) ceph status

To view active warnings or errors, run the following command:

oc rsh -n openshift-storage $(oc get pods -n openshift-storage -l app=rook-ceph-tools -o name) ceph health detail

Understand cluster states

The output of ceph status includes placement group (PG) states that indicate data health.

Active/clean: The ideal state. All placement groups are active, all data is replicated, and no data is moving. No action is required.
Active+remapped, active+backfilling, active+recovering: Data is being redistributed. These states are normal after an OSD resource change, node replacement, or scaling operation. Wait for the cluster to return to active/clean before making additional changes.

Example healthy output:

HEALTH_OK

Example output during rebalancing:

HEALTH_WARN
  Degraded data redundancy: 123/456 objects degraded (26.974%)
  Recovery 50/456 objects degraded (10.965%)

Check placement group and OSD status

For a more detailed view of data distribution and OSD health, complete the following checks.

Check placement group status to identify groups that are not in the active+clean state.

oc rsh -n openshift-storage $(oc get pods -n openshift-storage -l app=rook-ceph-tools -o name) ceph pg stat

Check individual OSD status to verify that the OSDs are up and in.

oc rsh -n openshift-storage $(oc get pods -n openshift-storage -l app=rook-ceph-tools -o name) ceph osd status

Tuning ODF performance for virtualization workloads

Tuning ODF performance for virtualization workloads

Before you begin

Selecting an ODF performance profile

Configuring OSD pod resource limits

Check current OSD resource limits

Increase OSD resource limits

Verify the updated resource limits

Configuring the bulk flag for large data operations

Checking Ceph cluster health

Run a basic health check

Understand cluster states

Check placement group and OSD status

Next steps