IBM Cloud Docs
Configuring the log level of a monitoring agent

Configuring the log level of a monitoring agent

The log level determines the type and amount of logging of an agent. You can configure the log level by adding parameters and log level arguments.

The monitoring agent writes log entries into the draios.log file.

  • The log file rotates when it reaches 10MB in size.
  • The 10 most recent log files are kept. The date-stamp that is appended to the filename is used to determine which files to keep.
  • The log files are located in the directory /opt/draios/logs/.

By default, when you deploy a monitoring agent, the log level is set to info.

Valid log levels are: none, error, warning, info, debug, trace.

You can configure the level of detail that is written by a monitoring agent to its log file:

Log levels
Log level Detail
none No errors are reported.
info Reports an entry for each aggregated metrics transmission to the backend servers, once per second.
Reports entries for any warnings and errors.
error Reports error logs only.
trace Reports all logs that are available.
  • To get the least number of log entries, set the log level to error.
  • To get the full set of available logs, set the log level to trace.

You should set the log level to info, debug, or trace to troubleshoot agent-related issues.

For example, consider the following use cases and the log section that you can configure to indicate the data that you want to log for an agent:

Log section entries
Use cases Log section entry
Troubleshoot agent behavior file_priority: debug
Reduce container console output console_priority: warning
Filtering events by severity event_priority: warning
Verify what metrics are included or excluded metrics_excess_log: true

Checking the log level of an agent

Linux monitoring agent

To get the log level of a Linux monitoring agent, run the following command from a terminal:

more /opt/draios/etc/statsite.ini | grep log_level

For example, you can get the following result:

# more /opt/draios/etc/statsite.ini | grep log_level
log_level = INFO

Docker monitoring agent

To get the log level of a docker monitoring agent, run the following command:

docker exec -ti sysdig-agent  more /opt/draios/etc/statsite.ini | grep log_level

For example, you can get the following result:

# docker exec -ti sysdig-agent  more /opt/draios/etc/statsite.ini | grep log_level
log_level = INFO

Kubernetes monitoring agent

To get the log level of a Kubernetes monitoring agent, complete the following steps:

  1. Set up the cluster environment. Run the following command:

    First, get the command to set the environment variable and download the Kubernetes configuration files.

    ibmcloud ks cluster config --cluster <cluster_name_or_ID>
    
  2. Get the agent log level. Run the following command:

    kubectl describe daemonset sysdig-agent -n ibm-observe | grep file_priority
    

    If the command does not return any value, the log level is set to the default value info.

Configuring the log level

The log level determines the type and amount of logging of an agent. You can configure the log level by adding parameters and log level arguments.

Linux monitoring agent

To configure the log level, you must customize the log section in the /opt/draios/etc/dragent.yaml file.

By default, a dragent.yaml file includes the following information:

customerid: xxxxxxxxxxxxxxxxx
collector: ingest.us-south.monitoring.cloud.ibm.com
collector_port: 6443
ssl: true
sysdig_capture_enabled: false

To configure a log level, you must edit the file and add information about the log.

  • To set the log level, you must set file_priority.
  • Valid log levels are: none, error, warning, info, debug, trace.

The following configuration sample shows a log level set to error:

customerid: xxxxxxxxxxxxxxxxx
collector: ingest.us-south.monitoring.cloud.ibm.com
collector_port: 6443
ssl: true
sysdig_capture_enabled: false
log:
  file_priority: error

After you edit the dragent.yaml file, you must restart the agent to activate the changes. Run the following command:

service dragent restart

Docker monitoring agent

To change the log level of a docker monitoring agent, complete the following steps:

  1. Open a shell in the running container:

    docker exec –it sysdig-agent /bin/bash
    
  2. Copy to your local machine the dragent.yaml file:

    docker cp sysdig-agent:/opt/draios/etc/dragent.yaml <local directory>/dragent.yaml
    
  3. Modify the configuration file. Include information about the log level and the console log level that you want to set. Add the following section if it is not included:

    log:
      file_priority: error
      console_priority: warning
    

    The file_priority controls the log level. Valid log levels are: none, error, warning, info, debug, trace.

    The console_priority controls the console output. Valid console log levels are: none, error, warning, info, debug, trace.

    The following configuration sample shows a log level set to error, and a console log level set to warning:

    customerid: xxxxxxxxxxxxxxxx
    tags: location:us-east
    collector: ingest.us-east.monitoring.cloud.ibm.com
    collector_port: 6443
    ssl: true
    sysdig_capture_enabled: false
    use_promscrape: true
    log:
      file_priority: error
      console_priority: warning
    
  4. Update the configuration file of the agent. Run the following command:

    docker cp <local directory>/dragent.yaml sysdig-agent:/opt/draios/etc/dragent.yaml
    
  5. Restart the agent. Run the following command:

    docker restart sysdig-agent
    
  6. Verify the level have been set. Run the following commands:

    docker exec -ti sysdig-agent  more /opt/draios/etc/statsite.ini | grep log_level
    

    You should see the log level that you specified in the dragent.yaml file.

Kubernetes monitoring agent

Complete the following steps to change the log level of a Kubernetes monitoring agent:

  1. Set up the cluster environment. Run the following commands:

    First, get the command to set the environment variable and download the Kubernetes configuration files.

    ibmcloud ks cluster config --cluster <cluster_name_or_ID>
    
  2. Modify the agent's DaemonSet file. Include information about the log level and the console log level that you want to set.

    Run the following command to edit the config map:

    kubectl edit daemonset sysdig-agent -n ibm-observe
    

    Then, add or modify the log entries:

    log:
      file_priority: error
      console_priority: warning
    

    The file_priority controls the log level. Valid log levels are: none, error, warning, info, debug, trace.

    The console_priority controls the console output. Valid console log levels are: none, error, warning, info, debug, trace.

    The following configuration sample shows a log level set to debug, and a console log level set to error:

    ...
    spec:
      revisionHistoryLimit: 10
      selector:
        matchLabels:
          app: sysdig-agent
      template:
        metadata:
          creationTimestamp: null
          labels:
             app: sysdig-agent
        spec:
          containers:
          - image: icr.io/ext/sysdig/agent:latest
            imagePullPolicy: Always
            name: sysdig-agent
            env:
            - name: ADDITIONAL_CONF
              value: "log:\n  file_priority: debug\n  console_priority: error"
            readinessProbe:
    ...
    

The log level changes are applied when you save the changes.

If you have changed previously the log level, the DaemonSet section looks like:

...
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: sysdig-agent
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: sysdig-agent
    spec:
      containers:
      - env:
        - name: ADDITIONAL_CONF
          value: |-
            log:
              file_priority: debug
              console_priority: error
        image: icr.io/ext/sysdig/agent:latest
        imagePullPolicy: Always
...

Note, you can also download the configmap by running: kubectl get daemonset sysdig-agent -o yaml -n ibm-observe > sysdig-agent-ds.yaml. To apply the changes, you can run kubectl apply -f sysdig-agent-ds.yaml -n ibm-observe.