--- name: monitoring-nvidia title: Monitoring NVIDIA GPUs with IBM Cloud Monitoring description: You can monitor NVIDIA GPU performance in your IKS clusters, ROKS clusters, and VSI instances with IBM Cloud Monitoring. On IKS and ROKS clusters, GPU metrics are automatically collected by the Monitoring agent once the NVIDIA GPU Operator is installed. On VSI instances, a simple scrape configuration must be added to the agent. Metrics are sent to your Monitoring instance for analysis, troubleshooting, and alerting. last-updated: 2026-04-22 --- > ## Documentation Index > The table of contents for this documentation set is at https://cloud.ibm.com/docs/monitoring?format=markdown > The index for all IBM Cloud docs is at: https://cloud.ibm.com/docs/llms.txt > Use these files to discover more information as needed. # Monitoring NVIDIA GPUs with IBM Cloud Monitoring {: #nvidia-gpu-monitoring} You can monitor NVIDIA GPU performance in your IKS clusters, ROKS clusters, and VSI instances with IBM Cloud Monitoring. On IKS and ROKS clusters, GPU metrics are automatically collected by the Monitoring agent once the NVIDIA GPU Operator is installed. On VSI instances, a simple scrape configuration must be added to the agent. Metrics are sent to your Monitoring instance for analysis, troubleshooting, and alerting. {: shortdesc} ## Metrics {: #nvidia-gpu-monitoring-metrics} With the NVIDIA DCGM Exporter, you can collect GPU performance metrics including: - GPU utilization and temperature - GPU memory usage and saturation - Power consumption - NVLink throughput - ECC memory errors - Per-GPU and per-workload breakdowns ## Prerequisites {: #nvidia-gpu-monitoring-prereqs} - You must have an Monitoring instance provisioned in your account. For more information, see [Getting started with IBM Cloud Monitoring](https://cloud.ibm.com/docs/monitoring?topic=monitoring-getting-started&format=markdown). - You must have an IKS or ROKS cluster with GPU-enabled worker nodes, or a VSI instance provisioned with NVIDIA GPUs. - For IKS and ROKS clusters, you need `kubectl` and `helm` CLI tools installed and configured to access your cluster. - For ROKS clusters, you need `oc` CLI access. - [Learn more about the NVIDIA GPU Operator](https://github.com/NVIDIA/gpu-operator){: external}. ## Monitoring GPUs on IKS clusters {: #nvidia-gpu-monitoring-iks} ### Step 1. Verify the Monitoring agent {: #nvidia-gpu-monitoring-iks-step1} The Monitoring agent is automatically deployed when creating IKS clusters. Verify the agent is running with the following command: ```sh kubectl -n ibm-observe get pods ``` {: pre} You should see the `sysdig-agent` pods in a `Running` state. If the agent is not present, you can install it by following the [Monitoring agent installation guide](https://cloud.ibm.com/docs/monitoring?topic=monitoring-about-collect-metrics&format=markdown#about-collect-metrics-2). {: note} ### Step 2. Install the NVIDIA GPU Operator {: #nvidia-gpu-monitoring-iks-step2} 1. Add the NVIDIA Helm repository and update it. ```sh helm repo add nvidia https://helm.ngc.nvidia.com/nvidia helm repo update ``` {: pre} 2. Create a `values.yaml` file with the following content: ```yaml nodeSelector: "ibm-cloud.kubernetes.io/gpu-enabled": "true" dcgmExporter: serviceMonitor: enabled: false ``` {: codeblock} In this file, the operator is configured to only be installed on worker nodes provisioned with NVIDIA GPUs. The `serviceMonitor` feature is disabled to ensure compatibility with IBM Cloud Monitoring. 3. Install the GPU Operator using Helm. ```sh helm upgrade --install gpu-operator nvidia/gpu-operator \ --namespace gpu-operator \ --create-namespace \ -f values.yaml ``` {: pre} ### Step 3. Verify GPU metrics collection {: #nvidia-gpu-monitoring-iks-step3} After the GPU Operator is installed, the Monitoring agent automatically uses an out-of-the-box configuration sent from the backend to scrape all GPU metrics from the DCGM Exporter. No additional configuration is needed. Restart the Monitoring agent to ensure the new configuration is picked up: ```sh kubectl rollout restart daemonset sysdig-agent --namespace ibm-observe ``` {: pre} After a few minutes, you will be able to see the **Nvidia GPU Monitoring** dashboard in your Monitoring instance. ## Monitoring GPUs on ROKS clusters {: #nvidia-gpu-monitoring-roks} ### Step 1. Verify the Monitoring agent {: #nvidia-gpu-monitoring-roks-step1} The Monitoring agent is automatically deployed when creating ROKS clusters. Verify the agent is running with the following command: ```sh kubectl -n ibm-observe get pods ``` {: pre} You should see the `sysdig-agent` pods in a `Running` state. If the agent is not present, you can install it by following the [Monitoring agent installation guide](https://cloud.ibm.com/docs/monitoring?topic=monitoring-about-collect-metrics&format=markdown#about-collect-metrics-2). {: note} ### Step 2. Install the NVIDIA GPU Operator {: #nvidia-gpu-monitoring-roks-step2} For ROKS clusters, follow the official Red Hat guide: 1. Install the Node Feature Discovery Operator (NFD) and verify it. 2. Install the NVIDIA GPU Operator. For detailed instructions, see the [NVIDIA GPU Operator on OpenShift documentation](https://docs.nvidia.com/datacenter/cloud-native/openshift/latest/index.html){: external}. ### Step 3. Verify GPU metrics collection {: #nvidia-gpu-monitoring-roks-step3} Once the GPU Operator is installed, the Monitoring agent automatically uses an out-of-the-box configuration sent from the backend to scrape all GPU metrics from the DCGM Exporter. No additional configuration is needed. After a few minutes, you will be able to see the **Nvidia GPU Monitoring** dashboard in your Monitoring instance. ## Monitoring GPUs on VSI instances {: #nvidia-gpu-monitoring-vsi} ### Step 1. Verify the Monitoring agent {: #nvidia-gpu-monitoring-vsi-step1} Ensure the Monitoring agent is installed and running on your VSI instance. For more information, see [Collecting default metrics by using the Monitoring agent](https://cloud.ibm.com/docs/monitoring?topic=monitoring-about-collect-metrics&format=markdown#about-collect-metrics-2). ### Step 2. Install the NVIDIA DCGM Exporter {: #nvidia-gpu-monitoring-vsi-step2} Run the NVIDIA DCGM Exporter as a Docker container: ```sh docker run -d --gpus all --cap-add SYS_ADMIN --rm -p 9400:9400 nvcr.io/nvidia/k8s/dcgm-exporter:4.4.2-4.7.0-ubuntu22.04 ``` {: pre} It is recommended to create a `systemd` service or similar so the container runs automatically after a reboot. {: tip} ### Step 3. Configure the Monitoring agent {: #nvidia-gpu-monitoring-vsi-step3} Add the following `prometheus.yaml` section to the Monitoring agent configuration file at `/opt/draios/etc/dragent.yaml`: ```yaml prometheus.yaml: | scrape_configs: - job_name: "nvidia-metrics" static_configs: - targets: ["localhost:9400"] ``` {: codeblock} Restart the agent to apply the configuration: ```sh service dragent restart ``` {: pre} After a few minutes, you will be able to see the **Nvidia GPU Monitoring** dashboard in your Monitoring instance. ## Dashboard and alerts {: #nvidia-gpu-monitoring-dashboard} With this integration, the following are available in your Monitoring instance: - **Nvidia GPU Monitoring** dashboard with GPU performance data. The dashboard can be scoped by cluster, namespace, workload, and GPU device. - **Alerts** ready to be enabled in the Alerts Library: - **[Nvidia] Gpu Memory Exhaustion** — GPU VRAM usage exceeded the configured threshold. The device is at risk of out-of-memory errors. - **[Nvidia] Gpu Temperature Too High** — GPU temperature exceeded the configured threshold. Risk of thermal throttling or hardware damage. - **[Nvidia] Gpu Underutilization** — GPU utilization has been below the configured threshold for too much time. The device may be idle or misconfigured. You can customize the dashboard and enable the alerts from the Monitoring UI. For more information, see [Working with dashboards](https://cloud.ibm.com/docs/monitoring?topic=monitoring-dashboards&format=markdown) and [Working with alerts](https://cloud.ibm.com/docs/monitoring?topic=monitoring-alerts&format=markdown).