Using PromQL

Use the Prometheus Query Language (PromQL) to select and aggregate time-series metric data for Prometheus.

For more information about PromQL, see the open-source documentation.

Creating a PromQL query

Create a PromQL query for a metric in an existing dashboard.

From the Monitoring dashboard, click Open dashboard for your service instance.
From Dashboards, select the dashboard that contains the metrics that you want to query.
For one panel in the dashboard, click the edit icon.
Select the PromQL query type.
In the query form, enter Display data, such as the query name and the timeseries name.
Enter your PromQL Query, such as avg(avg_over_time(host_cpu_used_percent[$__interval])) by(kubernetes_cluster_name). Specify the following fields in your query:
- Metric: Specify the metric that you want to query, such as host_cpu_used_percent. Note: The generated values on the table are the latest metric.
- Time range: Specify a time range or time interval, such as 5m. To use the time range that is selected in the UI, specify $__range. To use the time interval that is based on the time range that is selected in the UI, specify $__interval. The $__range and $__interval variables in this query automatically update as the time range is changed in the UI. For more information, see Applying dashboard scopes to PromQL queries.
- Segmentation: Choose a value to segment the aggregated PromQL data, such as kubernetes_cluster_name.
Click Run Query.
Fine tune the results by configuring any additional Options, such as the units in which the data is returned and how the data is displayed.

Applying dashboard scopes

To scope a panel built from a PromQL query, you must use a scope variable within the query.

Two pre-defined variables exist for time-based scopes that you can specify in the query. These variables are automatically update as the time range for the dashed is changed in the UI.

$__range: Represents the time frame that is selected in the dashboard UI. For example, you might use this variable to calculate an average for the selected time frame.
$__interval: Represents the interval that is based on the time range that is selected in the dashboard UI. For example, you might use this variable to adapt the time range for different operations, such as the rate or average over time.

For example, say you use the following query to return the CPU used percent for all the hosts:

avg_over_time(host_cpu_used_percent[$__interval])

This query uses the $__interval variable, which scopes data to the dynamic time interval based on the time range that is selected in the dashboard UI.

You can also specify your own variables in the query to scope its output. For example, to further scope the previous query to a specific label such as hostname, you must first define a scope variable at the dashboard level. Then, you can specify that variable in the query:

avg_over_time(host_cpu_used_percent{host_name=$hostname}[$__interval])

Using labels

When you run PromQL queries, the data is returned with a minimum set of labels. To add more labels, such as a cluster name, use a vector matching operation.

Prometheus returns information metrics that have a value of 1 with several labels.Joining the labels of an information metric with a non-information metric can provide useful insights, such as the value of a metric across an application. You can use a vector matching operation, which is similar to an SQL join, to add the information labels to your metric data.

Example: Filtering an application metric by cluster

In this example, a metric that is returned by an application is filtered by cluster. The PromQL aggregates a value of that metric for a cluster by having only one cluster selected in the scope.

sum (myapp_metric * on (container_id) kube_pod_container_info{cluster=$cluster})

This query:

Filters the kube_pod_container_info information metric for only a specific cluster ($cluster, which is set as a dashboard variable) based on the cluster label and for a specific timeseries.
Matches the myapp_metric non-information metric to the filtered kube_pod_container_info information metric when the container_id label has the same value. The values are multiplied, but because the information metric has a value of 1, the result is unchanged. The cluster label is added to the resulting data.
Aggregates the timeseries of value myapp_metric by using the sum function, and returns the result.

Example: Filtering average CPU used by account and region

In this example, the average CPU used (percent) is calculated for a specific IBM Cloud account and region.

avg by(region,account_id) (host_cpu_used_percent * on (host_mac) group_left(region,account_id) sysdig_cloud_provider_info{account_id=~$account, region=~$region})

This query:

Filters the sysdig_cloud_provider_info information metric for only a specific region ($region) and account ($account) that are set as dashboard variables based on the region and account_id labels.
Matches the host_cpu_used_percent non-information metric to the filtered sysdig_cloud_provider_info information metric when the host_mac label has the same value. The values are multiplied, but because the information metric has a value of 1, the result is unchanged. The region and account labels are added to the resulting data.
Calculates the average of the new metrics by account and region.

cURL sample to extract latest metric

To get the latest value of a metric, specify only the metric name. The most recent value that was generated no more than 5 minutes ago is returned.

For instance, to get the latest value of host_cpu_used_percent:

curl <SYSDIG_REST_API_ENDPOINT>/prometheus/api/v1/query?query=sysdig_host_cpu_used_percent -H "Authorization: $AUTH_TOKEN" -H "IBMInstanceID: $GUID" -H "SysdigTeamID: $TEAM_ID" -H "content-type: application/json"

Where

<SYSDIG_REST_API_ENDPOINT>indicates the endpoint targetted by the REST API call. For more information, see Monitoring REST API endpoints. For example, the public endpoint for an instance that is available in us-south is the following: https://us-south.monitoring.cloud.ibm.com/api
You can pass multiple headers by using -H.

Authorization and IBMInstanceID are headers that are required for authentication.

SysdigTeamID is optional. When you specify this header, you limit the request to the data and resources available for the team specified.

To get an AUTH_TOKEN and the GUID see, Headers for IAM Tokens.

All dashboards support the full PromQL API. For more information about what’s possible with PromQL, see the Prometheus documentation.