Deploying the Logging agent for Linux

You can deploy the Logging agent to collect and route infrastructure and application logs from Linux environments such as RHEL8, RHEL9, Debian, and Ubuntu to an IBM Cloud Logs instance. For more information on supported Linux environments, see Logging agent for non-orchestarted environments.

These instructions are for Red Hat Linux systems but can be used for other Linux RPM-based servers.

Complete the following steps to deploy an agent to a supported Linux environment.

Define the authentication method for the agent

Choose the type of identity and the authentication method for the agent. Then, create a trusted profile or an API key. The role that is required for sending logs to IBM Cloud Logs is Sender.

You can use a service ID or a trusted profile as the identity that is used by the agent to authenticate with the IBM® Cloud Logs service. For more information, see Granting IAM permissions for ingestion.

Choose one of the following options:

Option 1: Authentication using a trusted profile

Create a Trusted Profile. For more information, see Generating a Trusted Profile for ingestion.

Option 2: Authentication using a service ID API key

Generate an API Key for service ID authentication. For more information, see Generating an API Key for ingestion.

Download the required RPM or DEB packages

Complete the following steps:

Download the required RPM or DEB packages.

For information about the current Logging agent version, see the agent release notes.
Validate the checksum by running the following command:
```
sha256sum -c <sha256_filename>
```
Where <sha256_filename> is the filename of the download *.sha256 file.

Set up and deploy the Logging agent configuration

Complete the following steps:

Log in to your Linux environment.
Install the agent.
- For RHEL run:
```
rpm -ivh <rpm_filename>
```
- For Debian and Ubuntu run:
```
dpkg -i <deb_filename>
```
Where <rpm_filename> or <deb_filename> is the name of the downloaded *.rpm or *.deb file.
Run the configuration script.
```
/opt/fluent-bit/bin/post-config.sh -h <target_host> -p <target_port> [-t <target_path>] -a <auth_mode> -k <iam_api_key> [-s <vsi_secure_access_enabled>] [-i <IAM_environment>] [--subsystem-name <name>] [--application-name <name>]
```
Where

-t <target_path>

Specify the path to send data to an IBM Cloud Logs instance. If not provided, the value defaults to /logs/v1/singles.

-a <auth_mode>

Specify IAMAPIKey or VSITrustedProfile.

-k <iam_api_key>

Specify the Cloud Identity and Access Management API key (required for IAMAPIKey mode). Make sure you follow the instructions in Generating an API Key.

For more information about Cloud Identity and Access Management API Keys, see Managing API Keys.

-d <trusted_profile_id>

Specify the trusted profile ID (required for VSITrustedProfile mode). When using trusted profiles, set to the ID configured in Setting up Permissions for Ingestion. You must create the instance with the metadata service enabled and link the trusted profile to your instance by specifying the ID when creating it. For more information see Creating virtual server instances.

For more information on Trusted Profiles, see Creating a Trusted Profile.

--send-directly-to-icl Deprecated

Set this parameter to send logs directly to IBM Cloud Logs.

-h <target_host>

The host for IBM Cloud Logs ingestion, found in the Endpoints section of your IBM Cloud Logs instance Overview. Use the ingress endpoint. For more information, see Ingress endpoints

-i <IAM_environment>

Specifies whether a public or private endpoint is used for IAM authentication. Production indicates to use the public endpoint. PrivateProduction specifies to use the private endpoint. Production is the default.

If your system does not have access to the public internet, you must use PrivateProduction to use the private endpoint.

-p <target_port>

Use 443 to send logs directly to IBM Cloud Logs.

-s <vsi_secure_access_enabled>

(Optional) Set this to true if you have secure access enabled in your VSI. It will be set to false by default. For example, -s true.

--application-name <name>

The application name defines the environment that produces and sends logs to IBM Cloud Logs. If not provided, the value defaults to ${HOSTNAME}.

--subsystem-name <name>

The subsystem name is the service or application that produces and sends logs to IBM Cloud Logs. If not provided, the value defaults to not-found.

Run the script to update the configuration with your parameter changes.

Logging agent default configuration

Logging agent comes with a default configuration that includes a predefined input source and filters. This default setup is configured to collect logs from common locations and enrich them with basic metadata.

Default Input Source

Logging agent is pre-configured to monitor log files in the /var/log/ directory and its subdirectories. The following is an example of the default inputs.conf:

[INPUT]
  Name              tail
  Tag               host.*
  Path              /var/log/**/*.log
  Path_Key          file
  Exclude_Path      /var/log/at/**
  DB                /var/lib/fluent-bit/fluent-bit.DB
  Buffer_Chunk_Size 32KB
  Buffer_Max_Size   256KB
  Mem_Buf_Limit     30MB
  Skip_Long_Lines   On
  Refresh_Interval  10
  storage.type      filesystem
  storage.pause_on_chunks_overlimit on

Default Filters

The default configuration includes two filters that enhance log records with metadata and organize it for easier processing:

Modify Filter: Adds metadata fields such as subsystem name, application name, hostname, and platform.
Nest Filter: Groups metadata fields under a single meta key.

The following is an example of the default filters.conf:

[FILTER]
  Name modify
  Match *
  Add subsystemName    ${SUBSYSTEM_NAME}
  Add applicationName  ${APPLICATION_NAME}
  Add meta.hostname    ${HOSTNAME}
  Add meta.environment prod   # Sample values: prod, staging, dev, qa
  Add meta.platform    linux

[FILTER]
  Name nest
  Match *
  Operation nest
  Wildcard meta.*
  Nest_under meta
  Remove_prefix meta.

Logging agent configuration customization

These defaults provide a foundation for log collection and enrichment. To configure the agent to your needs, you can modify the inputs.conf, filters.conf, and outputs.conf files located in the /etc/fluent-bit/ directory.

Add additional metadata fields

You can add additional metadata fields to the routed logs.

Complete the following steps:

Edit the fluent-bit.conf file in the /etc/fluent-bit/ folder.

Add your custom metadata using this structure: Add <meta.key_name> <your_custom_value>

[FILTER]
  Name modify
  Match *
  Add subsystemName    ${SUBSYSTEM_NAME}
  Add applicationName  ${APPLICATION_NAME}
  Add meta.hostname    ${HOSTNAME}
  Add meta.env         prod   # Sample values: prod, staging, dev, qa
  Add meta.platform    linux

Where

<meta.key_name> is the name of the metadata field to be added (for example, meta.env) and <your_custom_value> is the value to be assigned to the field (for example, the name of your environment).

For example, if you want to add the region as metadata, or version as new field, the configuration would be similar to this:

[FILTER]
  Name modify
  Match *
  Add subsystemName    ${SUBSYSTEM_NAME}
  Add applicationName  ${APPLICATION_NAME}
  Add meta.hostname    ${HOSTNAME}
  Add meta.env         prod
  Add meta.platform    linux
  Add meta.region      us-east
  Add version          my-version

Save the configuration file.

Restart the agent to apply the changes.

systemctl daemon-reload && systemctl restart fluent-bit

Include or exclude files

You must configure the log files that the Logging agent monitors.

Complete the following steps:

Edit the fluent-bit.conf file in the /etc/fluent-bit/ folder.

Modify the INPUT section.

Set the Path with the directories and files that you want to monitor.

Set the Exclude_Path with the directories and files that you want to exclude from monitoring.

[INPUT]
  Name              tail
  Tag               *
  Path              /var/log/*.log
  Path_Key          file
  Exclude_Path      /var/log/audit.log
  DB                /var/lib/fluent-bit/fluent-bit.DB
  Buffer_Chunk_Size 32KB
  Buffer_Max_Size   256KB
  Skip_Long_Lines   On
  Refresh_Interval  10
  storage.type      filesystem
  storage.pause_on_chunks_overlimit on

Modify the SERVICE section.

Set the storage.path to a location in the file system to store streams and data.
Save the configuration file.

Restart the agent to apply the changes.

systemctl daemon-reload && systemctl restart fluent-bit

Verify logs are being delivered to your target destination

Complete the following steps:

Go to the web UI for your IBM Cloud Logs instance..
When your agent is correctly configured, you can see logs through the default dashboard view.

For example, if you set the applicationName to the hostname in your agent, you can set the applicationname filter in a view to the name of your host.

Updating the agent

You can update the agent by downloading the desired agent packages and running the appropriate command for your environment. For information about the current Logging agent version, see the agent release notes.

For RHEL run:
```
rpm -Uvh <rpm_filename>
```
For Debian and Ubuntu run:
```
dpkg -i <deb_filename>
```

Where <rpm_filename> or <deb_filename> is the name of the downloaded *.rpm or *.deb file.

When reinstalling or upgrading the agent, your existing configuration files are unchanged.

Restart the service after updating:

systemctl daemon-reload && systemctl restart fluent-bit

Uninstalling the agent

You can remove an installed agent by running the appropriate command from a terminal as the sudo user.

For RHEL, CentOS, and Fedora Linux run:
```
sudo yum erase draios-agent
```
For Debian and Ubuntu run:
```
sudo apt-get remove draios-agent
```