Customizing Linux monitoring agents

By default, the monitoring agent collects data from a range of platforms and applications. You can edit the agent config file to change its default behavior, and include or exclude data. You can customize the monitoring agent configuration file to include custom metrics such as JMX, StatsD, and Prometheus. You also can filter out metrics and containers.

The monitoring agent uses two configuration files that specify what data to collect automatically:

Table 1. monitoring agent configuration files
File	Description	Location
`dragent.default.yaml`	Main configuration file IMPORTANT: Do not edit this file.	`/opt/draios/etc/dragent.default.yaml`
`dragent.yaml`	Configuration file that you edit to customize the monitoring agent.	`/opt/draios/etc/dragent.yaml`

You can edit the dragent.yaml file to customize the data that is collected. Changes take effect immediately after you save the file. The agent detects the change and automatically restarts. You can find the frequency and the files that a monitoring agent checks in the dragent.default.yaml file.

For example, a dragent.default.yaml includes the following information:

check_frequency_s: 5
files:
- /opt/draios/etc/dragent.yaml
- /opt/draios/etc/kubernetes/config/dragent.yaml
- /opt/draios/etc/kubernetes/secrets/access-key

Editing the dragent yaml file

The yaml file is loacated in /opt/draios/etc/.

Complete the following steps to edit the file and apply the changes:

Access the dragent.yaml directly. The file is located in /opt/draios/etc/dragent.yaml.
Edit the file. Use valid YAML syntax.
Restart the agent. Run the following command:
```
service dragent restart
```

Blocking ports

To block network traffic and metrics from network ports, you must customize the blacklisted_ports section in the dragent.yaml file. You must list the ports from which you want to filter out any data.

Note: Port 53 (DNS) is always blacklisted.

For example, the following sample shows how to set the blacklisted_ports section of a monitoring agent to exclude data coming from ports 6666 and 6379:

blacklisted_ports:
    - 6666
    - 6379

Including and excluding metrics

To filter custom metrics, you must customize the metrics_filter section in the dragent.yaml file. You can specify which metrics to include and which ones to filter out by configuring the include and exclude filtering parameters.

Note: The filtering rule order is set as follows: the first rule that matches a metric is applied.

For example, if the metrics_filter section of a monitoring agent looks as follows:

metrics_filter:
  - include: metricA.*
  - exclude: metricA.*
  - include: metricB.*
  - include: haproxy.backend.*
  - exclude: haproxy.*
  - exclude: metricC.*

You are configuring the monitoring agent to collect all data from metrics that start with metricA, metricB, and haproxy.backend.
You are filtering out metrics that start with metricC and other metrics that start with haproxy.
The entry exclude: metricA.* is ignored.

Changing the log level

To configure the log level, you must customize the log section in the dragent.yaml file.

The monitoring agent generates log entries in /opt/draios/logs/draios.log.

The log file rotates when it reaches 10MB in size.
The 10 most recent log files are kept. The date-stamp that is appended to the filename is used to determine which files to keep.
Valid log levels are: none, error, warning, info, debug, trace
The default log level is info, where an entry is created for each aggregated metrics transmission to the backend servers, once per second, in addition to entries for any warnings and errors.
You can customize the type of log and the entries that are collected by configuring the monitoring agent configuration file /opt/draios/etc/dragent.yaml. After you edit the file, you must restart the agent at the shell with service dragent restart to activate the changes.

Table 2. Log section entries
Use cases	Log section entry
Troubleshoot agent behavior	`file_priority: debug`
Reduce container console output	`console_priority: warning`
Filtering events by severity	`event_priority: warning`
Verify what metrics are included or excluded	`metrics_excess_log: true`