Collecting availability metrics by using the Prometheus Blackbox exporter
The Prometheus Blackbox exporter allows blackbox probing of endpoints over HTTP, HTTPS, DNS, TCP and ICMP. The monitoring agent can be used in conjunction with the Blackbox exporter to collect availability metrics. The availability metrics can then be alerted upon within IBM Cloud Monitoring to alert users on the availability of the endpoints.
Complete the following steps to configure the Prometheus Blackbox Exporter:
Configure a monitoring agent to collect metrics
To monitor 1 or more hosts, you must configure a monitoring agent. The agent collects automatically a set of metrics that you can monitor through the monitoring UI.
See Install a monitoring agent to collect and forward metrics from a server to an IBM Cloud Monitoring instance and choose the monitoring agent that you want to configure to monitor a host.
Install the Prometheus Blackbox exporter
Complete the following steps to run the Blackbox exporter as a docker container:
-
Download the blackbox.yml file from Github.
Save the file to the directory
/config/prometheus/blackbox/
. -
Run the Blackbox exporter as a docker container:
docker run --rm -d -p 9115:9115 -l io.prometheus.scrape=true -l io.prometheus.port=9115 -l io.prometheus.path=/probe --name blackbox_exporter -v `pwd`:/config/prometheus/blackbox prom/blackbox-exporter:master --config.file=/config/prometheus/blackbox/blackbox.yml
-
Check the exporter is up. Run the following command:
docker container ls
You can see an output like the following one:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2480a0034bb4 prom/blackbox-exporter:master "/bin/blackbox_expor…" 36 minutes ago Up 36 minutes 0.0.0.0:9115->9115/tcp blackbox_exporter
To stop the blackbox container, run the following command:
docker container stop 2480a0034bb4
.
Configure network settings
If you want to collect metrics from remote servers, complete the following steps:
-
Enable the firewall to allow access to the
blackbox_exporter
. -
[Optional] Update the VPC rules
If you use private endpoints, add an inbound rule to the security group for port
9115
withsource type = Security Group
and choose the security group for the server.
Update the monitoring agent that is running in the server
Run the following command to edit the configmap and add information about the Blackbox targets that you want to monitor by using the Kubernetes monitoring agent:
kubectl edit configmap sysdig-agent -n ibm-observe
log:
file_priority: error
configmap: true
### Agent tags
# tags: linux:ubuntu,dept:dev,local:nyc
#### Monitoring Software related config ####
# Monitoring collector address
# collector: 192.168.1.1
# Collector TCP port
# collector_port: 6666
# Whether collector accepts ssl
# ssl: true
# collector certificate validation
# ssl_verify_certificate: true
#######################################
# new_k8s: true
# k8s_cluster_name: production
security:
k8s_audit_server_url: 0.0.0.0
k8s_audit_server_port: 7765
k8s_cluster_name: <CLUSTER_NAME>/<CLUSTER_ID>
tags: ibm.containers-kubernetes.cluster.id:<CLUSTER_ID>
collector: <INGESTION_ENDPOINT>
collector_port: 6443
ssl: true
ssl_verify_certificate: true
sysdig_capture_enabled: false
promscrape_fastproto: true
use_promscrape: true
prometheus:
enabled: true
prom_service_discovery: true
log_errors: true
max_metrics: 200000
max_metrics_per_process: 200000
max_tags_per_metric: 100
ingest_raw: true
ingest_calculated: false
prometheus.yaml:
----
global:
scrape_interval: 10s
scrape_configs:
- job_name: blackbox
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
- https://prometheus.io/
- https://promcat.io/
- https://api.promcat.io/apps
- https://www.google.com/search?q=promcat
- https://promcat.io/apps/harbor
- https://www.ibm.com/software/passportadvantage
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- source_labels: [__param_target]
target_label: blackbox_instance
- target_label: __address__
replacement: blackbox-exporter.ibm-observe:9115
Where
<IP_ADDRESS_OF_REMOTE_SERVER>
is the IP address of a server that you want to monitor.<INGESTION_ENDPOINT>
is the IBM Cloud Monitoring instance ingestion endpoint, for example,ingest.us-south.monitoring.cloud.ibm.com
. See Collector endpoints.
When you save the file, changes are applied.
Configure the default dashboard and alerts to analyze the Blackbox status of your server
Complete the following steps:
-
Create a dashboard.
-
Add a panel for each of the following queries:
Metric | Query | Options |
---|---|---|
Status code by Instance | probe_http_status_code{instance=$instance} |
number (auto) |
SSL Cert Expiry in Days | probe_ssl_earliest_cert_expiry{instance=$instance}-time() |
time (auto) |
DNS Lookup | probe_dns_lookup_time_seconds{instance=$instance} |
time (auto) |
Probe duration | probe_duration_seconds{instance=$instance} |
time (auto) |
For example, you can create a dashboard that looks as follows:
![Sample Blackbox dashboard Sample Blackbox dashboard](images/blackbox-ui-1.png)