Monitoring NVIDIA GPUs with IBM Cloud Monitoring

You can monitor NVIDIA GPU performance in your IKS clusters, ROKS clusters, and VSI instances with IBM Cloud Monitoring. On IKS and ROKS clusters, GPU metrics are automatically collected by the Monitoring agent once the NVIDIA GPU Operator is installed. On VSI instances, a simple scrape configuration must be added to the agent. Metrics are sent to your Monitoring instance for analysis, troubleshooting, and alerting.

Metrics

With the NVIDIA DCGM Exporter, you can collect GPU performance metrics including:

GPU utilization and temperature
GPU memory usage and saturation
Power consumption
NVLink throughput
ECC memory errors
Per-GPU and per-workload breakdowns

Prerequisites

You must have an Monitoring instance provisioned in your account. For more information, see Getting started with IBM Cloud Monitoring.
You must have an IKS or ROKS cluster with GPU-enabled worker nodes, or a VSI instance provisioned with NVIDIA GPUs.
For IKS and ROKS clusters, you need kubectl and helm CLI tools installed and configured to access your cluster.
For ROKS clusters, you need oc CLI access.
Learn more about the NVIDIA GPU Operator.

Monitoring GPUs on IKS clusters

Step 1. Verify the Monitoring agent

The Monitoring agent is automatically deployed when creating IKS clusters. Verify the agent is running with the following command:

kubectl -n ibm-observe get pods

You should see the sysdig-agent pods in a Running state.

If the agent is not present, you can install it by following the Monitoring agent installation guide.

Step 2. Install the NVIDIA GPU Operator

Add the NVIDIA Helm repository and update it.

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

Create a values.yaml file with the following content:
```
nodeSelector:
  "ibm-cloud.kubernetes.io/gpu-enabled": "true"
dcgmExporter:
  serviceMonitor:
    enabled: false
```
In this file, the operator is configured to only be installed on worker nodes provisioned with NVIDIA GPUs. The serviceMonitor feature is disabled to ensure compatibility with IBM Cloud Monitoring.

Install the GPU Operator using Helm.

helm upgrade --install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  -f values.yaml

Step 3. Verify GPU metrics collection

After the GPU Operator is installed, the Monitoring agent automatically uses an out-of-the-box configuration sent from the backend to scrape all GPU metrics from the DCGM Exporter. No additional configuration is needed.

Restart the Monitoring agent to ensure the new configuration is picked up:

kubectl rollout restart daemonset sysdig-agent --namespace ibm-observe

After a few minutes, you will be able to see the Nvidia GPU Monitoring dashboard in your Monitoring instance.

Monitoring GPUs on ROKS clusters

Step 1. Verify the Monitoring agent

The Monitoring agent is automatically deployed when creating ROKS clusters. Verify the agent is running with the following command:

kubectl -n ibm-observe get pods

You should see the sysdig-agent pods in a Running state.

If the agent is not present, you can install it by following the Monitoring agent installation guide.

Step 2. Install the NVIDIA GPU Operator

For ROKS clusters, follow the official Red Hat guide:

Install the Node Feature Discovery Operator (NFD) and verify it.
Install the NVIDIA GPU Operator.

For detailed instructions, see the NVIDIA GPU Operator on OpenShift documentation.

Step 3. Verify GPU metrics collection

Once the GPU Operator is installed, the Monitoring agent automatically uses an out-of-the-box configuration sent from the backend to scrape all GPU metrics from the DCGM Exporter. No additional configuration is needed.

After a few minutes, you will be able to see the Nvidia GPU Monitoring dashboard in your Monitoring instance.

Monitoring GPUs on VSI instances

Step 1. Verify the Monitoring agent

Ensure the Monitoring agent is installed and running on your VSI instance. For more information, see Collecting default metrics by using the Monitoring agent.

Step 2. Install the NVIDIA DCGM Exporter

Run the NVIDIA DCGM Exporter as a Docker container:

docker run -d --gpus all --cap-add SYS_ADMIN --rm -p 9400:9400 nvcr.io/nvidia/k8s/dcgm-exporter:4.4.2-4.7.0-ubuntu22.04

It is recommended to create a systemd service or similar so the container runs automatically after a reboot.

Step 3. Configure the Monitoring agent

Add the following prometheus.yaml section to the Monitoring agent configuration file at /opt/draios/etc/dragent.yaml:

prometheus.yaml: |
  scrape_configs:
    - job_name: "nvidia-metrics"
      static_configs:
        - targets: ["localhost:9400"]

Restart the agent to apply the configuration:

service dragent restart

After a few minutes, you will be able to see the Nvidia GPU Monitoring dashboard in your Monitoring instance.

Dashboard and alerts

With this integration, the following are available in your Monitoring instance:

Nvidia GPU Monitoring dashboard with GPU performance data. The dashboard can be scoped by cluster, namespace, workload, and GPU device.
Alerts ready to be enabled in the Alerts Library:
- [Nvidia] Gpu Memory Exhaustion — GPU VRAM usage exceeded the configured threshold. The device is at risk of out-of-memory errors.
- [Nvidia] Gpu Temperature Too High — GPU temperature exceeded the configured threshold. Risk of thermal throttling or hardware damage.
- [Nvidia] Gpu Underutilization — GPU utilization has been below the configured threshold for too much time. The device may be idle or misconfigured.

You can customize the dashboard and enable the alerts from the Monitoring UI. For more information, see Working with dashboards and Working with alerts.