Monitoring cluster health

Understanding options for monitoring

To help understand when to use the built-in Red Hat OpenShift tools or IBM Cloud integrations, review the following information.

Monitoring limitation in private-only clusters with RHCOS worker nodes: The monitoring agent relies on kernel headers in the operating system, however RHCOS doesn't have kernel headers. In this scenario, the agent reaches back to sysdig.com to use the pre-compiled agent. In clusters with no public network access this process fails. To allow monitoring on RHCOS clusters, you must either allow outbound traffic or see the Sysdig documenation for installing the agent on air-gapped environments.

Forwarding cluster and app metrics to IBM Cloud Monitoring

Use the Red Hat OpenShift on IBM Cloud observability plug-in to create a monitoring configuration for IBM Cloud Monitoring in your cluster, and use this monitoring configuration to automatically collect and forward metrics to IBM Cloud Monitoring.

With IBM Cloud Monitoring, you can collect cluster and pod metrics, such as the CPU and memory usage of your worker nodes, incoming and outgoing HTTP traffic for your pods, and data about several infrastructure components. In addition, the agent can collect custom application metrics by using either a Prometheus-compatible scraper or a statsd facade.

Considerations for using the Red Hat OpenShift on IBM Cloud observability plug-in:

• You can have only one monitoring configuration for IBM Cloud Monitoring in your cluster at a time. If you want to use a different IBM Cloud Monitoring service instance to send metrics to, use the ibmcloud ob monitoring config replace command.
• Red Hat OpenShift clusters in Satellite can't currently use the Red Hat OpenShift on IBM Cloud console or the observability plug-in CLI to enable monitoring for Satellite clusters. You must manually deploy monitoring agents to your cluster to forward metrics to Monitoring.
• If you created a Monitoring configuration in your cluster without using the Red Hat OpenShift on IBM Cloud observability plug-in, you can use the ibmcloud ob monitoring agent discover command to make the configuration visible to the plug-in. Then, you can use the observability plug-in commands and functionality in the IBM Cloud console to manage the configuration.

Before you begin:

• Verify that you are assigned the Editor platform access role and Manager server access role for IBM Cloud Monitoring.
• Verify that you are assigned the Administrator platform access role and the Manager service access role for all Kubernetes namespaces in IBM Cloud Kubernetes Service to create the monitoring configuration. To view a monitoring configuration or launch the Monitoring dashboard after the monitoring configuration is created, users must be assigned the Administrator platform access role and the Manager service access role for the ibm-observe Kubernetes namespace in IBM Cloud Kubernetes Service.
• If you want to use the CLI to set up the monitoring configuration:
  • Install the Red Hat OpenShift on IBM Cloud observability CLI plug-in (ibmcloud ob).
  • Access your Red Hat OpenShift cluster.

To set up a monitoring configuration for your cluster:

1. Create an IBM Cloud Monitoring service instance and note the name of the instance. The service instance must belong to the same IBM Cloud account where you created your cluster, but can be in a different resource group and IBM Cloud region than your cluster.

2. Set up a monitoring configuration for your cluster. When you create the monitoring configuration, an Red Hat OpenShift project ibm-observe is created and a Monitoring agent is deployed as a Kubernetes daemon set to all worker nodes in your cluster. This agent collects cluster and pod metrics, such as the worker node CPU and memory usage, or the amount incoming and outgoing network traffic to your pods.

   In the console.

   1. From the Red Hat OpenShift clusters console, select the cluster for which you want to create a Monitoring configuration.
   2. On the cluster Overview page, click Connect.
   3. Select the region and the IBM Cloud Monitoring service instance that you created earlier, and click Connect.

   In the CLI.

   1. Create the Monitoring configuration. When you create the Monitoring configuration, the access key that was last added is retrieved automatically. If you want to use a different access key, add the --sysdig-access-key <access_key> option to the command.

      To use a different service access key after you created the monitoring configuration, use the ibmcloud ob monitoring config replace command.

      Version 4.15 and later: If your cluster has outbound traffic protection enabled, you must set up monitoring by using the private endpoint. To do this, specify the --private-endpoint option.

      ibmcloud ob monitoring config create --cluster <cluster_name_or_ID> --instance <Monitoring_instance_name_or_ID> [--private-endpoint]
      

      Example output

      Creating configuration...
      OK
      

   2. Verify that the monitoring configuration was added to your cluster.

      ibmcloud ob monitoring config list --cluster <cluster_name_or_ID>
      

      Example output

      Listing configurations...
      
      OK
      Instance Name                Instance ID                            CRN   
      IBM Cloud Monitoring-aaa     1a111a1a-1111-11a1-a1aa-aaa11111a11a   crn:v1:prod:public:sysdig:us-south:a/a11111a1aaaaa11a111aa11a1aa1111a:1a111a1a-1111-11a1-a1aa-aaa11111a11a::  
      

3. Optional: Verify that the Monitoring agent was set up successfully.

   1. If you used the console to create the Monitoring configuration, log in to your cluster.

   2. Verify that the daemon set for the Monitoring agent was created and all instances are listed as AVAILABLE.

      oc get daemonsets -n ibm-observe
      

      Example output

      NAME           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
      sysdig-agent   9         9         9       9            9           <none>          14m
      

      The number of daemon set instances that are deployed equals the number of worker nodes in your cluster.

   3. Review the ConfigMap that was created for your Monitoring agent.

      oc describe configmap -n ibm-observe
      

4. Access the metrics for your pods and cluster from the Monitoring dashboard.

   1. From the Red Hat OpenShift clusters console, select the cluster that you configured.
   2. On the cluster Overview page, click Launch. The Monitoring dashboard opens.
   3. Review the pod and cluster metrics that the Monitoring agent collected from your cluster. It might take a few minutes for your first metrics to show.

5. Review how you can work with the Monitoring dashboard to further analyze your metrics.

Viewing cluster states

Review the state of a Red Hat OpenShift cluster to get information about the availability and capacity of the cluster, and potential problems that might occur.

To view information about a specific cluster, such as its zones, service endpoint URLs, Ingress subdomain, version, and owner, use the ibmcloud oc cluster get --cluster <cluster_name_or_ID> command. Include the --show-resources option to view more cluster resources such as add-ons for storage pods or subnet VLANs for public and private IPs.

You can review information about the overall cluster, the IBM-managed master, and your worker nodes. To troubleshoot your cluster and worker nodes, see Troubleshooting clusters.

For more information about cluster and worker states, see:

• Cluster states.
• Worker node states.

Enabling remote health reporting

Telemetry is a remote health monitoring feature that collects aggregated data about your cluster, such as the health of your components and the number and types of resources in use. If you have a public cluster, you can elect to have your own Telemetry data visible in your account for your use. For more information, see Telemetry for remote health monitoring.