Data pipelines

In IBM® Cloud Logs, you can define the way logs are distributed across different data pipelines and balance cost in your environment.

About data pipelines

There are 3 data pipelines. Each pipeline has a different storage price.

Priority insights pipeline
Analyze and alert pipeline
Store and search pipeline

Each pipeline offers different features that you can use to work with the data.

Table 1. Features available in each data pipeline
Feature	Priority insights	Analyze and alert	Store and search
Fast query	Yes	No	No
High-speed search using hot storage	Yes	No	No
Dashboards and analytics using hot storage	Yes	No	No
Dashboards and analytics using IBM Cloud Object Storage	Yes	Yes	No
Intelligent log analytics	Yes	Yes	No
Alert on logs	Yes	Yes	No
Metrics maintained on log data for up to 1 year	Yes	Yes	No
Re-index logs for further analysis	Yes	Yes	Yes
Search logs in IBM Cloud Object Storage	Yes	Yes	Yes
Store logs in IBM Cloud Object Storage	Yes	Yes	Yes

Using the TCO Optimizer

You use the TCO Optimizer to configure policies that define which data pipeline handles data after ingestion.

You can apply policies to data based on the application name, the subsystem name, and log severity. These 3 fields are metadata fields that all log data must have. For more information, see Metadata fields.

The TCO Optimizer can help you improve real-time analysis and alerting while managing costs.

You can exclude data by defining a TCO policy that is applied immediately after ingestion or by configuring a Block parsing rule. Data excluded by using the TCO policy is dropped. Data excluded by using a Block parsing rule offers 2 options: drop the data or allow the data to be available through live tail. When you configure the option to view data in live tail, data is also archived to the service instance's COS bucket.

When you configure parsing rules that add metadata, drop data, or replace data in a log record, or when you enrich log data, the log line that is archived is the log line modified.

The image shows the different available data pipelines. — Figure 1. Available data pipelines.

The following table outlines core features by data pipeline for data ingested after TCO policies are applied:

Table 2. Core features by data pipeline for data ingested after TCO policies are applied
Feature	Priority insights	Analyze and alert	Store and search
Parsing rules	Yes	Yes	Yes
Custom data enrichment	Yes	Yes	Yes
Schema store	Yes	Yes	Yes
Events to Metrics	Yes	Yes	No
Dynamic alerting	Yes	Yes	No
Templating	Yes	Yes	No
Anomaly detection	Yes	Yes	No
Indexing	Yes	No	No

IBM Cloud Object Storage requirement

You must have a IBM Cloud Object Storage bucket configured for your instance to archive data for long term storage from the Priority insights data pipeline, to archive data for the Analyze and alert and Store and search data pipelines, and if you configure parsing rules that block data with the option to see data through live tail.

Priority insights data pipeline

Use the Priority insights data pipeline for high priority logs that require the most immediate attention and intervention such as logs for troubleshooting problems or analyzing unexpected behaviour.

Features available for high prioirty logs are:

Serverless monitoring
Rapid query
Custom dashboards
Service Catalog
Service Map
Alerting
Events to Metrics
Query archive
Viewing traces in your explore screen

Analyze and alert data pipeline

Use the Analyze and alert data pipeline for medium priority logs that may require attention at some point, but do not require immediate attention.

Features available for medium priority logs are:

Service Catalog
Service Map
Alerting
Events to Metrics
Query archive
Viewing traces in your explore screen

Store and search data pipeline

Use the Store and search data pipeline for logs that you must keep for compliance purposes but do not require action to be taken.

Features available for low priority logs include:

Query archive
Viewing traces in your explore screen