Setting up and configuring a monitoring host

A monitoring agent collects the metrics from an SAP system and forwards them to the dashboard of the monitoring instance.

Deploying VPC virtual server instance host for a monitoring agent

Use Prometheus exporters to collect the metrics. The section about Collecting metrics explains details about metrics, labels, and monitoring agents.

The Prometheus exporter is based on the GitHub open source project SAP HANA DB exporter.

Before you begin, check out the IBM Cloud® documentation about Creating virtual server instances to create a monitoring host by using theIBM Cloud® UI.

Monitoring host consideration

Consider the following settings for the monitoring host.

The monitoring host must be deployed in the same region and zone as the monitored SAP system.
Make sure that you select the correct resource group that corresponds to your deployable architecture (DA).
You must deploy the ibm-sles-15-5-amd64-sap-applications-<version> image - SUSE Linux Enterprise Server 15 SP5 for SAP Applications (AMD64) - with the smallest available size of 2 cores.
Add your appropriate SSH keys as described in Managing SSH keys.

Verifying ACLs and security groups

After you create your monitoring agent host, go to your ACLs and security groups to verify that the settings are correct.

If you are using ingestion endpoints other than the country location of the monitoring host, you must adjust the IBM Cloud® network ACL settings for a corresponding Private REST API endpoint. For more information, see Collecting metrics by using Prometheus remote write.

Exporting SAP variables as bash variables

Use the following command to export SAP variables as Bash variables.

export sap_monitoring_nr=<sap_monitoring_nr>
export sap_hana_ip=<sap_hana_ip>
export sap_ascs_ip=<sap_ascs_ip>
export sap_app_server_ip_01=<sap_app_server_ip_01>
export sap_app_server_port_01=<sap_app_server_port_01>
export sap_ascs_http_port=<sap_ascs_http_port>
export sap_hana_http_port=<sap_hana_http_port>
export sap_hana_sql_systemdb_port=<sap_hana_sql_systemdb_port>
export sap_hana_sql_systemdb_user=<sap_hana_sql_systemdb_user>
export sap_hana_sql_systemdb_password=<sap_hana_sql_systemdb_password>
export ibmcloud_monitoring_instance_url=<ibmcloud_monitoring_instance_url>
export ibmcloud_monitoring_authorization_credentials=<ibmcloud_monitoring_authorization_credentials>

Checking connectivity for the database port and SAP instance ports

Install the netcat tool by using the following command.

sudo zypper install netcat

Use netcat to check connectivity to the database port and SAP instance ports on the monitoring host.

nc –vz ${sap_hana_ip} ${sap_hana_sql_systemdb_port}

Check connectivity from the SAP HANA database server to the HTTP port of the sapstartsrv web service on the SAP HANA server.

nc –vz ${sap_hana_ip} ${sap_hana_http_port}

Check connectivity from the SAP ASCS server to the HTTP port of the sapstartsrv web service of the ASCS on the SAP ASCS server.

nc –vz ${sap_ascs_ip} ${sap_ascs_http_port}

Check connectivity from IP address of the first SAP (primary) application server to the HTTP port of the sapstartsrv web service on the first application server.

nc –vz ${sap_app_server_ip_01} ${sap_app_server_port_01}

Checking the status of SLES operating system repositories

Make sure the appropriate SLES OS repositories are enabled by using the following command.

SLE-Module-SAP-Applications15-SP5-Updates
SLE-Module-Packagehub-Subpackages15-SP5-Pool

If the repositories are not enabled, use the following commands to enable them.

sudo SUSEConnect -p sle-module-sap-applications/15.5/x86_64

sudo SUSEConnect -p PackageHub/15.5/x86_64

If you are unable to activate these repositories, then you have installed the wrong version of the SLES operating system image.

Installing SLES Prometheus packages

Use the following steps to install SLES Prometheus packages.

Update all operating system packages to the latest version. Restart the host if necessary.
```
sudo zypper update
```

Install the following Prometheus packages.

sudo zypper install prometheus-sap_host_exporter

sudo zypper install prometheus-hanadb_exporter

sudo zypper install golang-github-prometheus-prometheus

Create the Prometheus data storage directory.
```
sudo mkdir /opt/prometheus
```
Change the user and group IDs of the Prometheus data storage directory to the Prometheus user.
```
sudo chown -R prometheus:prometheus /opt/prometheus
```
```
sudo chmod u+rwx /opt/prometheus
```

Copy the metric file metrics.json to the configuration directory /etc/hanadb_exporter.

sudo cp /usr/share/doc/packages/prometheus-hanadb_exporter/metrics.json  /etc/hanadb_exporter/metrics.json

Installing the SAP Python driver hdbcli

Use the following steps to install the SAP Python driver hdbcli.

Access the SAP Download Center by using the SAP S-user authentication.
Retrieve the SAP HANA Python driver by downloading the official SAP HANA Client. For more detailed information, see Install the SAP HANA Client.
Install the SAP Python driver hdbcli as described in the Installing the Python driver.

Choose the following SAP HANA Client package with the current <version> and <patch>.

SUPPORT PACKAGES & PATCHES - SAP HANA PLATFORM EDITION/ - SAP HANA PLATFORM EDITION 2.0/ - SAP HANA CLIENT 2.0 - LINUX ON X86_64 64BIT - IMDB_CLIENT20_<version>_<patch>-80002082.SAR

Download the SAPCAR_*.EXE package from the SAP download center.
Copy the two files SAPCAR_*.EXE and IMDB_CLIENTXXX.SAR to an arbitrary directory on the monitoring host.
As the root user, run the following command.
```
chmod 755 SAPCAR_*.EXE
```
Extract the SAP HANA Client from the SAR-file with SAPCAR.
```
./SAPCAR_*.EXE -xvf IMDB_CLIENT20*.SAR
```
The file name SAPCAR_*.EXE is a Linux binary file.
Use hdbinst to install the SAP HANA Client.
```
cd SAP_HANA_CLIENT
```
```
./hdbinst
```
Enter the installation path: [/usr/sap/hdbclient]: /usr/sap/hdbclient.
Install the required pip package for Python 3.x.
```
sudo zypper install python3-pip
```

Install the SAP HANA Python driver.

pip install /usr/sap/hdbclient/hdbcli-*.tar.gz

Creating the hdbuserstore key

Use the following steps to create the hdbuserstore key.

Create the stored user key with the same Python user that runs the hanadb_exporter. Use the ReadOnly SQL user as <sap_hana_sql_systemdb_user>.
```
/usr/sap/hdbclient/hdbuserstore SET MONITORING-${sap_monitoring_nr}-KEY \
"${sap_hana_ip}:${sap_hana_sql_systemdb_port}@SYSTEMDB" \
${sap_hana_sql_systemdb_user} ${sap_hana_sql_systemdb_password}
```
For more information, see SAP HANA User Store (hdbuserstore) and hdbuserstore Commands.

For the next steps, the userkey is added to the configuration file for the hanadb_exporter as "userkey": "MONITORING-<sap_monitoring_nr>-KEY".
Verify that your key was created successfully by listing all keys.
```
/usr/sap/hdbclient/hdbuserstore list
```

To list only your key, use the following command.

/usr/sap/hdbclient/hdbuserstore list MONITORING-${sap_monitoring_nr}-KEY

Configuring the hanadb_exporter and activating the daemon

Use the following steps to configure the hanadb_exporter and activate the daemon.

The hanadb_exporter collects metrics from the SAP HANA database. All metrics from the database tenant SYSTEMDB are forwarded to the monitoring ingestion endpoint through Prometheus.

The GitHub repository and original documentation can be found at the following websites.

The configuration consists of two configuration files:

metrics.json - contains all the SQL statements for querying the SAP HANA database
config.json contains sap_hana_ip, sap_hana_sql_systemdb_port, and hdbuserkey

To avoid needing administrator privileges on the database, use the ReadOnly SQL user from the hdbuserstore. The specific parameters are added to the configuration file by replacing the placeholders <…>.

Create the configuration file as user root with the file name /etc/hanadb_exporter/config-<sap_monitoring_nr>-SQL.json by using the following command:

{
   "listen_address": "0.0.0.0",
   "exposition_port": 5<sap_monitoring_nr>02,
   "multi_tenant": false,
   "_comment_multi_tenant": "true, if you want another 1000 metrics",
   "timeout": 30,
   "hana": {
     "host": "<sap_hana_ip>",
     "port": <sap_hana_sql_systemdb_port>,
     "userkey": "MONITORING-<sap_monitoring_nr>-KEY",
     "ssl": true,
     "ssl_validate_cert": false
     }
}

For the value of the userkey parameter, use the hdbuserkey name that you created earlier.

You can list the keys by using the following command.

/usr/sap/hdbclient/hdbuserstore list

AChange the permissions of the configuration files so that only the root user can read them.
```
chmod 600 /etc/hanadb_exporter/config*
```
Before you define the configuration of sap_host_exporters for the systemd service, test the exporters manually by using the following command.
```
hanadb_exporter -c /etc/hanadb_exporter/config-${sap_monitoring_nr}-SQL.json -m /etc/hanadb_exporter/metrics.json
```
The process is blocking one terminal, so open a second terminal and verify that an exporter has correctly collected the metric. Run the following curl command.
```
curl http://localhost:5${sap_monitoring_nr}02/metrics
```

If you see the metrics, you should also see a status message. If the metrics are listed correctly, stop the exporter process.

Configure the systemd application for the hanadb_exporter.

The systemd configuration file for the hanadb_exporter is included in the installation package. No changes to this configuration file are required. The systemd service can start any process with the configuration name (without the JSON extension as the file name).

To enable the systemd service for the hanadb_exporter, use the following commands.

sudo systemctl start prometheus-hanadb_exporter@config-${sap_monitoring_nr}-SQL

sudo systemctl enable prometheus-hanadb_exporter@config-${sap_monitoring_nr}-SQL

To show the status of the hanadb_exporter run the command below.

sudo systemctl status prometheus-hanadb_exporter@config-${sap_monitoring_nr}-SQL

Configuring the sap_host_exporter and activating the daemon

Use the following information to configure the sap_host_exporter and activate the daemon.

Prometheus sap_host_exporter uses stateless HTTP protocol to collect metrics from the SAP system instances. For more information, see the official GitHub repository.

The SAP exporter configuration contains all the parameters for the connection and the HTTP port. Note that you need to create a separate configuration file for each connection.

Create the configuration files in the /etc/sap_host_exporter/ directory. Name the configuration files according to the SAP instance by using the following command.

Use the following templates for the configuration files.

As the root user, create the SAP HANA configuration file with the file name /etc/sap_host_exporter/sap_host_exporter-${sap_monitoring_nr}-HANA.yaml.

# The listening TCP/IP address and port.
address: "0.0.0.0"
port: "5<sap_monitoring_nr>03"
log-level: "info"
sap-control-url: "http://<sap_hana_ip>:<sap_hana_http_port>"

As the root user, create the SAP ASCS configuration file with the file name /etc/sap_host_exporter/sap_host_exporter-${sap_monitoring_nr}-ASCS.yaml.

# The listening TCP/IP address and port.
address: "0.0.0.0"
port: "5${sap_monitoring_nr}04"
log-level: "info"
# ASCS instance
sap-control-url: "http://<sap_ascs_ip>:<sap_ascs_http_port>"

As the root user, create the first SAP application server (PAS) configuration file with the name /etc/sap_host_exporter/sap_host_exporter-${sap_monitoring_nr}-DI-01.yaml.

# The listening TCP/IP address and port.
address: "0.0.0.0"
port: "5<sap_monitoring_nr>05"
log-level: "info"
# DI instance (metrics dispatcher server + resources)
sap-control-url: "http://<sap_app_server_ip_01>:<sap_app_server_port_01>"

As the root user, create the second SAP application server (AAS) configuration file with the name /etc/sap_host_exporter/sap_host_exporter-<sap_monitoring_nr>-DI-02.yaml.
```
# The listening TCP/IP address and port.
address: "0.0.0.0"
port: "5<sap_monitoring_nr>06"
log-level: "info"
# DI instance (metrics dispatcher server + resources)
sap-control-url: "http://<sap_app_server_ip_02>:<sap_app_server_port_02>"
```
If there are multiple application servers, then create more configuration files for each SAP application server. Make sure that you increment the number of each configuration file.

Test each sap_host_exporter configuration.

sap_host_exporter -c /etc/sap_host_exporter/sap_host_exporter-<sap_monitoring_nr>-HANA.yaml

The process is blocking the terminal, so open a second terminal and run the following command.
```
curl http://localhost:5<sap_monitoring_nr>03/metrics
```
Verify that the metric data is displayed. If no metrics are displayed, you should see some error messages.

Repeat the same test with other exporter configurations:

/etc/sap_host_exporter/sap_host_exporter-<sap_monitoring_nr>-ASCS.yaml

/etc/sap_host_exporter/sap_host_exporter-<sap_monitoring_nr>-DI-01.yaml

If all of the exporters are working correctly, continue with the systemd configuration.

Configure the systemd server for the sap_host_exporter.

Only one extra systemd configuration file is required for all sap_host_exporters. The systemd service can start any process with the configuration name (without the JSON extension as file name).

Create a file sap_host_exporter@.service as user root in the system directory.
```
/etc/systemd/system/sap_host_exporter@.service
```

Add the following content without making any changes.

[Unit]
Description=Prometheus sap_host_exporter for Netweaver clusters metrics
After=network.target
Documentation=https://github.com/SUSE/sap_host_exporter
[Service]
Type=simple
Restart=always
ExecStart=/usr/bin/sap_host_exporter --config /etc/sap_host_exporter/%i.yaml
ExecReload=/bin/kill -HUP $MAINPID
[Install]
WantedBy=multi-user.target
DefaultInstance=default

Run the following command as user root to reload the systemd configuration.
```
systemctl daemon-reload
```

As user root, start and enable the service for all configured sap_host_exporters.

systemctl enable --now sap_host_exporter@sap_host_exporter-${sap_monitoring_nr}-HANA

systemctl enable --now sap_host_exporter@sap_host_exporter-${sap_monitoring_nr}-ASCS

systemctl enable --now sap_host_exporter@sap_host_exporter-${sap_monitoring_nr}-DI-01

systemctl enable --now sap_host_exporter@sap_host_exporter-${sap_monitoring_nr}-DI-02

As user root, check the status of systemd processes.

systemctl status sap_host_exporter@sap_host_exporter-${sap_monitoring_nr}-ASCS

systemctl status sap_host_exporter@sap_host_exporter-${sap_monitoring_nr}-HANA

systemctl status sap_host_exporter@sap_host_exporter-${sap_monitoring_nr}-DI-01

systemctl status sap_host_exporter@sap_host_exporter-${sap_monitoring_nr}-DI-02

Enable systemd autostart.

sudo systemctl enable sap_host_exporter@sap_host_exporter-${sap_monitoring_nr}-ASCS

sudo systemctl enable sap_host_exporter@sap_host_exporter-${sap_monitoring_nr}-HANA

sudo systemctl enable sap_host_exporter@sap_host_exporter-${sap_monitoring_nr}-DI-01

Next steps

If the setup and configuration of the monitoring host was successful, continue with Configuration of Prometheus server metric forwarding.