Controlling cost

After you start using IBM® Cloud Logs, you might find that you need to adjust how data is processed by IBM Cloud Logs to control costs.

You can configure IBM Cloud Logs or adjust the data that is ingested by IBM Cloud Logs to control your usage cost.

Selecting the best service plan

Review the available service plans and pricing. The cost for the data pipeline where you plan to store data affects your total cost.

Also, consider how long you need to keep your data for fast search in Priority insights. You can configure your IBM Cloud Logs instance to save your data in Priority insights for a shorter time period.

Data is metered by the service per gigabytes ingested. You can configure data usage metrics to monitor your data usage. Data usage metrics requires that you have a metrics bucket configured.

Configuring the TCO optimizer

By default, when an IBM Cloud Logs instance is created, all data flows to the Priority insights data pipeline. While this pipeline gives you the fastest search, it is the highest cost pipeline.

To control costs, you want to optimize the pipeline where data is sent.

By defining the data pipeline based on the importance of the data to your business, the TCO Optimizer can help you improve real-time analysis and alerting and helps you manage costs.

For information on configuring the TCO optimizer, see Configuring the TCO Optimizer.

Using TCO policies

How logs are associated to pipelines is determined by policies. Policies are applied on combinations of applications, subsystems, and log severity as logs are ingested. Logs are assigned to the appropriate TCO pipeline based on the policy content.

You must have a IBM Cloud Object Storage data bucket configured before creating a policy.

Understanding data ingestion

To understand how to contol your cost when using IBM Cloud Logs, you need to understand how data is ingested by the service.

IBM Cloud Logs ingests data from multiple sources and that data is processed in a specific order:

Logs are sent by the source.
- Operational logs are sent using an agent or API REST call.
- Platform logs and activity tracking events are sent by IBM Cloud.
IBM Cloud Logs ingests (receives) the data.
IBM Cloud Logs parsing rules are applied to the ingested data.
TCO policies are applied and the data is assigned to the appropriate data pipeline, or dropped, depending on the policy.

A log is considered to be ingested for the purposes of billing once the parsing rules and TCO policies have been applied. Logs that are blocked before being sent to the data pipelines do not incur charges and these logs are not retained.

Controlling ingested data

In addition to controlling the data pipelines where your data is kept within IBM Cloud Logs, you can control the data that is ingested by IBM Cloud Logs.

Using parsing rules

You can use the block parsing rule to filter out incoming logs based on a RegEx expression. For more information on configuring a block parsing rule, see Blocking log data.

You aren't limited to blocking full log lines. Fields in log lines that you don't need can be removed using parsing rules. For information on removing fields, see Removing fields from logs by using the Remove fields rule.

Filtering and restructuring data sent through the Logging agent

If you are sending operational logs to IBM Cloud Logs by using the Logging agent, you can parse and restructure the log data for consistency and to remove data that you might not require in IBM Cloud Logs.

You can use the Fluent Bit grep filter with the exclude option.
You can use the Fluent Bit modify filter to drop fields that you don't need.
When installing the Logging agent, make use of the excludeLogSourcePaths option when installing with the helm chart or the Exclude_Path option on the tail plug-in (Linux and Windows installations) to not send logs from specific locations.
You can use Lua scripts to filter and modify data that is being sent to IBM Cloud Logs prior to ingestion.

Making sure logging data is well structured

When you modify log data, you need to make sure that the data sent to IBM Cloud Logs is well-formatted so IBM Cloud Logs can process the data correctly.

Logs should be sent in either JSON format or a filter is used with the Logging agent to convert the content into JSON format.
Since TCO policies use severity information within the log data to assign logs to the appropriate data pipeline, for consistent assignment it is best to make sure that the log data contain an appropriate level, loglevel, or severity value rather than having the Logging agent or IBM Cloud Logs imply the severity.
For logs originating from non-Kubernetes sources, log lines need to have application and subsystemName fields included. These fields are also used by TCO processing.
Logs should also include timestamps in UTC format. Alternately the Logging agent can be configure to offset the timestamp before sending the logs. See the information on Time_Offset and Time_System_Timezone in the Fluent Bit documentation.
Fields that will be frequently used in searches (for example, http_error_code) should be extracted into separate keys, rather than being included in the log line.