Considerations when you configure the Logging agent
When you configure the IBM® Cloud Logs Logging agent, you need to understand how logs are processed and how changing the default configuration can potentially result in dropped logs.
To take advantage of the best logs processing, with the least potential of dropped data, you want to run the latest Logging agent version. This information assumes that you are running the Logging agent 1.6.1 or later.
Retry limit
By default the Logging agent is configured to keep retrying to send logs until successful if an error occurs. The default behavior is configured by the retryLimit: false parameter. In Kubernetes this is managed in the Logging agent configmap, and in Linux environments it is in the Logging agent config file.
If you change the default retryLimit value, you have the potential for dropped data. For example, retryLimit: "no_retries" indicates that the Logging agent does not retry sending logs at all when an error
occurs. Specifying an integer value, for example retryLimit: 8, indicates that the Logging agent retries sending logs for the specified number (8) of times before the logs are dropped.
For information about how Fluent Bit schedules the retry intervals, see Scheduling and retries.
Fluent Bit buffering
The Logging agent is configured by default to use Fluent Bit filesystem buffering. This configuration means that logs processed by the Logging agent are kept in file storage in case of a system failure. When the Logging agent restarts,
logs in the filesystem buffer are processed.
If you change the configuration to use memory buffering, and an issue occurs where the Logging agent is terminated or terminates unexpectedly, logs in the memory buffer can be dropped.
For more information, see Buffering and storage.
Log rotation
If the environment rotates the log files before the Logging agent can process them, a potential exists that logs might be dropped. Dropped data due to log rotation can occur if the Logging agent stops running for a significant amount of time.
Kubernetes
In IBM managed clusters all stdout container log files in /var/log/containers are managed by kublet. Each container
log file will be rotated when it reaches 100MiB in size. After rotation, up to 3 log files will be retained per container. The size and number of these files is not configurable.
If the log files fill and rotate before the Logging agent processes the data, the unprocessed logs can be dropped.
Linux
Log rotation processing can be configured in Linux. For example, by using the logrotate tool.
If the log files fill and rotate before the Logging agent processes the data, the unprocessed logs can be dropped.
Windows
Log rotation processing can be configured in Windows. However, no built-in Windows tool exists for log rotation. Third-party applications, or built-in application settings, can be used to configure log rotation.
If the log files fill and rotate before the Logging agent processes the data, the unprocessed logs can be dropped.