TCO Optimizer

With IBM® Cloud Logs data pipelines and the TCO Optimizer, you can define the way logs are distributed across 3 distinct use cases. You can balance costs in the environment by using these pipelines.

By defining the data pipeline based on the importance of the data to your business, the TCO Optimizer can help you improve real-time analysis and alerting and helps you manage costs.

How logs are associated to pipelines is determined by policies. Policies are applied on combinations of applications, subsystems, and log severity as logs are ingested. Logs are assigned to the appropriate TCO pipeline based on the policy content. The default policy for all logs is high priority.

Policies simplify assigning TCO pipelines and capture applicable logs on ingestion. Each policy creates new default values for the logs for the applicable policy. If policies conflict, the first policy that is listed on the TCO Optimizer page takes precedence.

The three TCO pipelines are:

Priority insights: Logs that require immediate access and full IBM Cloud Logs analysis capabilities. These logs are typically high-severity or business-critical logs that need to be analyzed or queried individually.
Analyze and alert: Logs that require processing and can be queried later if needed from an archive. These logs are typically logs used for monitoring and statistical analysis.
Store and search: Logs that need to be kept for compliance or post-processing reasons but can be maintained and queried from an archive.

When you configure TCO policies, the selected priority determines the TCO pipeline for the logs that match the criteria.

Mapping of policy priority to TCO pipeline
Priority value	TCO pipeline
`High`	Priority insights
`Medium`	Analyze and alert
`Low`	Store and search
`Blocked`	`[*]`

[*] Logs matching policies with the Blocked priority are dropped and are not sent to any TCO pipeline.

All data is stored in the IBM Cloud Object Storage data bucket, including logs that are Blocked by a policy.

Accessing the TCO Optimizer

Compete the following steps to access the TCO Optimizer:

Launch the IBM Cloud Logs UI.
Click the Data pipeline icon > TCO optimiser.

The TCO Optimizer page shows the percentage of ingested data that is flowing to each pipeline after the configured policies are applied.

Creating a policy

You must have a IBM Cloud Object Storage data bucket configured before creating a policy.

On the TCO Optimizer page, complete the following steps to create a new policy:

Click ADD NEW POLICY.
Enter a policy name.
Enter the policy details with the relevant applications, subsystems, and severity. Add more criteria as needed.

Logs received by IBM Cloud Logs without a severity are treated as if their severity is debug.

Create a parsing rule to set the priority of logs at ingestion when a priority value is not included.

For applications and subsystems, criteria can be specified when the value matches one of: All, Is, Is Not, Includes, or Starts With.
Set the priority for the policy. The priority determines the pipeline for logs that are matched by the policy.
Click APPLY.

Modifying a policy

On the TCO Optimizer page, complete the following steps to modify an existing policy:

Click the policy that you want to change.
Modify the criteria.
Click APPLY.

If you want to change the policy priority value, you can also change the priority in the PRIORITY drop-down list for the policy in the policy list. The priority determines the pipeline for logs that are matched by the policy.

Deleting a policy

On the TCO Optimizer page, complete the following steps to delete an existing policy:

Click the policy that you want to delete.
Click DELETE.
Confirm that you want to delete the policy.