Collecting metrics by using Prometheus remote write

Configuring Prometheus remote write

You can configure your Prometheus server to remote write to a Monitoring instance. For example,

• You can configure the Prometheus server that runs in a Kubernetes cluster in different ways. Depending on how you installed it, you can choose any of the following options to configure the remote write feature in Prometheus to send metrics to an IBM Cloud Monitoring instance:

  • Configure a Prometheus server that is managed by Prometheus Operator

  • Configure a Prometheus server by using Helm

  • Configure a Prometheus server through a Kubernetes ConfigMap

  For more information, see Collecting metrics from a Kubernetes cluster by using Prometheus remote write.

• You can configure Prometheus remote write on a Linux host. For more information, see Collecting metrics on Linux based infrastructures by using Prometheus remote write.

• You can monitor the health, behavior, and performance of a WebSphere Application Server 9.0.5.9 or later by using IBM Cloud Monitoring. The WebSphere Application Server can run in the IBM Cloud, outside the IBM Cloud, or on-prem.

  For more information, see Collecting metrics from WebSphere Application Server by using Prometheus remote write.

global:
  external_labels:
    [ <labelname>: <labelvalue> ... ]
remote_write:
- url: "https://<INGESTION_URL>/prometheus/remote/write"
  bearer_token: "<MONITOR_API_TOKEN>"

global:
  external_labels:
    [ <labelname>: <labelvalue> ... ]
remote_write:
- url: "https://<INGESTION_URL>/prometheus/remote/write"
  authorization:
    credentials: "<MONITOR_API_TOKEN>"

Managing metrics collected via remote write

By default, when you configure your Prometheus server to remote write to a Monitoring instance, all metrics are collected and include a remote_write: true label to easily identify them.

remote_write:
    - url: https://<INGESTION_URL>/prometheus/remote/write
      authorization:
        credentials_file: /etc/secrets/<SECRET_NAME>
      write_relabel_configs:
        - action: keep
          regex: my-namespace
          source_labels:
          - namespace

# Prometheus server 1
global:
  external_labels:
    provider: prometheus1
remote_write:
- url: ...

# Prometheus server 2
global:
  external_labels:
    provider: prometheus2
remote_write:
- url: ...

Rate limits

The following table lists the default rate limits per instance:

Ingestion endpoints for remote write

IBM Cloud Monitoring has deprecated Prometheus client support of endpoints in the form https://ingest.<region> and https://ingest.private.<region> and replaced them with new endpoints. Customers using Prometheus Write need to migrate to use the new endpoints by 31 May 2024.

The following table lists the public Monitoring ingestion endpoints that you can configure to collect metrics via Prometheus Remote Write:

Remote write ingestion uses TCP 443 port.

The following table lists the private Monitoring ingestion endpoints that you can configure to collect metrics via Prometheus Remote Write:

Limitations

The IBM Cloud Monitoring Prometheus Remote Write feature has the following limitations:

• Metrics that are sent to an instance can be accessed in Explore, but they are not compatible with the scope tree.

• Only labels that are collected at source can be used to filter metrics.

• The metadata of a metric is not sent via remote write.

• Metrics that have a name that ends with _total, _sum, or _count are stored as a counters, otherwise they are managed as gauge.

• You can set units in Dashboards manually.

• You cannot mix metrics with different sampling in a dashboard, for example, 10s and 1min.

  Consider configuring the scrape interval to be 10s so that you can combine Prometheus Remote Write metrics with Monitoring agent metrics.