About cookies on this site Our websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising. For more information, please review your options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.
I think some of my data is not reaching my IBM Cloud Logs instance. What can I look for to determine possible ingestion failures?
You are monitoring your environment by using IBM® Cloud Logs and want to check for issues that affect the ingestion of data to the system.
You believe that data might not be reaching your IBM Cloud Logs instance.
Multiple causes can exist where data is not ingested by IBM Cloud Logs.
The following issues can cause data ingestion problems in your IBM Cloud Logs instance.
Retry limit is exceeded.
Check for logs in the IBM Cloud Logs UI with the string could not send data - this batch will not be retried.
The problem can be due to exceeding the configured retry limit or due to permission issues.
When the Logging agent receive a 403 error from IBM Cloud Logs, a retry error is returned. Check that your IAM permissions for ingestion are properly configured.
If the ingestion permissions are properly configured, you might need to change the Retry_limit in the output plug-in parameters.
Logs are dropped.
Check for logs in the IBM Cloud Logs UI starting with the string ICL dropped logs.
The Logging agent retries sending data to IBM Cloud Logs if the Logging agent does not receive a 200 OK response.
However, sometimes IBM Cloud Logs sends a 200 OK and writes a message in the response body that not all logs were accepted.
For example, logs can be dropped if they are older than a specific threshold.
The message from the Logging agent contains the reason why the logs were dropped.
IAM cannot be reached.
Check the logs for messages that Cloud Identity and Access Management (IAM) cannot be reached. Messages can include received message in token library or Failed to obtain token.
When IAM cannot be reached, the Logging agent cannot get a token to send logs.
This problem could be due to incorrect permissions or network issues.
For example, you might be using an API key that does not exist. Or, a firewall is in place that is blocking the Logging agent from reaching the IAM endpoint.
Check your IAM permissions and API key and correct if necessary. If due to a network issue, resolve the network issue.
Pods are restarting or not starting at all.
If Logging agent pods are frequently restarting, or not starting at all, check for memory consumption of the Logging agent and review the logs for other errors.