IBM Cloud Docs
Monitoring operational logs

Monitoring operational logs

You can configure the Logging agent to collect and send infrastructure and application logs to an IBM Cloud Logs instance directly.

The Logging agent is based on the Fluent Bit open-source agent which is used to collect and process log data. You can deploy the Logging agent in supported environments and manage data from various sources and formats.

The following diagram shows the high level view when the destination is an IBM Cloud Logs instance:

Flow of logs from agent
Flow of logs from agent

When you can configure the Logging agent to send logs to the IBM Cloud Logs service, logs that you send must include a value for the applicationName and subsystemName metadata fields. By default, when you configure the Logging agent, the agent sets default values for these fields. You can configure your own custom values to replace the default values.

Logging agent for orchestrated environments

You can deploy the Logging agent on a Red Hat OpenShift on IBM Cloud or IBM Cloud Kubernetes Service cluster.

You can deploy the agent on clusters that you run on-prem, in IBM Cloud, or in a different cloud.

The Logging agent is a daemon set that is designed to have one pod running on each node of a cluster. Each pod will collect relevant logs for the node its running on. The Logging agent will then forward those logs to the IBM Cloud Logs service.

By default, the Logging agent monitors and collects log data from files matching the specified path pattern in /var/log/containers/, excluding logs from files matching the exclusion pattern. The refresh interval is set to 10 seconds. You can change these values and more in the config map logger-agent-config.

You can deploy the agent in the following platforms:

The following diagram shows the high level view when the source of logs is a Kubernetes or OpenShift cluster:

Flow of logs from cluster
Flow of logs from cluster

Logging agent for non-orchestarted environments

You can deploy the Logging agent in Linux environments.

The following platforms are supported:

  • RHEL 8
  • RHEL 9
  • Ubuntu 20
  • Ubuntu 22
  • Debian 11
  • Debian 12

For more information, see Managing the agent Linux environments.

Supported formats

The agent supports the following input formats:

  • JSON
  • apache
  • apache2
  • apache_error
  • nginx
  • docker (JSON with the docker-specific timestamp format)
  • cri
  • syslog

Supported agent versions

The following table lists the agent versions that are supported and the version of Fluent Bit the agent is based on:

Supported agent versions
Logging agent Based on Fluent Bit Version Helm chart version
v1.4.0 v3.1.9 v1.4.0
v1.3.2 v3.1.9 v1.3.2
v1.3.1 v3.1.4 v1.3.1

For information on recommended and supported Fluent Bit plug-ins see Fluent Bit support

Agent support policy

Release agent updates are planned on a quarterly basis. Support will continue to provide assistance for two releases prior to the latest release.

For example, if agent version 1.3.x is the most currently release, then questions related to agents 1.2.x and 1.1.x will be answered.

However, new functions and security fixes will only be made available as the most current release and modification level.

For example, if a security vulnerability is found in the agent, and the current agent version is 1.3.3, the security vulnerability will be fixed and released as 1.3.4. Even though 1.2.x and 1.1.x are still supported for technical questions, patches to these releases will not be made available.