Managing volume spike protection and costs

Configure index rates alerts in your Activity Tracker instance to monitor and be alerted of unexpected spikes in your data volumes. Use the index rate to analyze and predict costs associated with storage of searchable data.

As of 28 March 2024 the IBM Log Analysis and IBM Cloud Activity Tracker services are deprecated and will no longer be supported as of 30 March 2025. Customers will need to migrate to IBM Cloud Logs, which replaces these two services, prior to 30 March 2025. For information about IBM Cloud Logs, see the IBM Cloud Logs documentation.

Index rates are calculated based on data that is ingested over the last 30 days and collected in 5 minute intervals.

Data collection commences on January 31, 2022 for existing instances. For new instances, data collection starts from the day that the instance is provisioned.

The standard hourly index rate is used to identify spikes or deviations in data volumes. The standard deviation is calculated by comparing the average hourly index rate with the average of the same hour over a rolling 30 day period. The severity of a spike is measured in standard deviations.

The z-score represents the count of standard deviations. You can use it to predict costs associated with storage of searchable data.

How are index rates calculated

The number of log lines that are indexed, that is, the number of log lines that are available for search, are collected every 5 minutes.

A day is based on a 24 hour period that is based on UTC time zone.

To calculate the daily average index rate, the following formula is applied:

The total number of indexed lines for the past hour, that is, the number of log lines over 1 hour rolling period, is collected.
The total number of indexed lines per hour is then divided by 60 to get the average of log lines per minute.
The average log lines per minute is then divided by 60 to get the average of log lines per second.
The index rate is calculated as the average value of all index rates that are calculated over 24 hours.

To calculate the hourly average index rate, also known as the standard hourly index rate, the following formula is applied:

The total number of indexed lines for the past hour, that is, the number of log lines over 1 hour rolling period, is collected.
The total number of indexed lines per hour is then divided by 60 to get the average of log lines per minute.
The average log lines per minute is then divided by 60 to get the average of log lines per second.
The index rate is calculated as the average value of all index rates that are calculated over the last hour by using 5 minutes increments.

The daily min index rate is the lowest index rate calculated over a 24 hour period.

The daily max index rate is the highest index rate calculated over a 24 hour period.

The max lines threshold indicates the maximum number of lines per second that you configure for your instance. This number represents the maximum number of lines that you expect to ingest per second for that instance.

If no log lines are ingested for over a 60-minute period, you get a Check Later message instead of the index rate value. When the ingestion of log lines is resumed, you get the index rate value. Measurements are taken every five minutes, therefore, the first measurement of ingested log lines for that five minutes will be averaged out for the full hour. Wait at least 1 hour before using the index rate value for analysis.

Only one alert for each alert channel will be sent until the hour or day has elapsed. This alert will show all thresholds that have been exceeded during the period. If index rate alerts are configured on each separate threshold, an alert will be sent indicating the first threshold crossed (either Max lines/s, Max z-score, or both)

Types of index rate alerts

There are two types of index rate alerts:

Max lines/s alerts: Use this alert to monitor and control the number of log lines per second that are indexed and stored for search.
- To monitor high flow rates, set an index rate threshold by configuring the max lines alert field so that you can get an alert when the value of lines per second is exceeded.
- When the average index rate changes over time, you must review and configure the threshold accordingly.
Max z-score alerts: Use this alert to monitor and control anomalous flow rates.
- To monitor for spikes on logs and anomalous flow rates, set a threshold value by configuring the max z-score field so that you can get an alert when the hourly index rate is significantly higher than the monthly mean.
- The baseline value is calculated using data from the last 30 days.
  
  After you provision an instance, you must wait a minimum of 30 days of data that has been indexed to use this feature.
- Each z-score represents 1 standard deviation that is based on the distribution of index rates as seen over the past 30 days.
  
  0, 1 indicates a small spike of data.
  
  3 indicates a spike of data that requires troubleshooting and control.
  
  5 indicates a spike of data that requires quick action such as defining exclusion rules while you find out what source and app are causing a spike of data.
Setting a maximum of 3 standard deviations is a good starting point for alert triggering. You can then refine your alerting based on your needs.

Launch the index rate alert page

The index rate alert page offers an overview of your account's current rate of ingestion and indexing of searchable data. Through this page, you can see the ratio between the log lines that are ingested versus the number of log lines that are being stored.

To use the index rate feature on an instance, an administrator of the service must turn on the feature.

Complete the following steps to view the index rate dashboard and enable the feature:

Navigating to the web UI.
Select the settings icon.
In the section Usage, select Index Rate Alerts.
Enable the feature.

When the feature is enabled, the index rate alert page is displayed.

You can use the index rate alerts page to monitor the daily average index rates or the standard deviation from the past 30 days.
You can toggle between viewing index rates or standard deviations for the past 30 days, the past 7 days, or the past 1 day.
If no log lines are ingested in a 60 minute period, Check Later will be displayed instead of the index rate value. Since measurements are taken every five minutes, the first measurement of ingested log lines for that five minutes will be averaged for the first hour. Wait at least an hour before using the index rate value for analysis any time there has been an ingestion interruption.

Activate the index rate alert feature

You must set at least 1 threshold to activate the index rate alert feature.

By default the index rate alert feature is disabled.

Configure index rate alerts

When you configure index rate alerts, you can gain insight into which applications, sources, or tags produce data spikes, as well as any recently added sources. The alert message that is sent includes the following information:

The top 20 sources and apps that have had the largest index rate growth in the past 2 hours.
The list of the 20 newest sources added.
A link to download generated lists of all applications and sources. You can view the “Manage Usage” Dashboard to download these lists at any time.

Complete the following steps:

Step 1. Configure 1 or more rules

Choose 1 or more of the following options to activate the index rate alert feature:

Max lines/s: Set an index rate threshold by configuring the max lines alert field so that you can get an alert when the value of lines per second is exceeded.
Max z-score: Set a threshold value by configuring the max z-score field so that you can get an alert when the hourly index rate is significantly higher than the monthly mean.

Step 2. Configure the notification channels

Choose 1 or more notification channels to receive an alert when the threshold of a rule is exceeded:

Configure the email notification channel by selecting Add email and entering 1 or more emails.
Configure the Slack notification channel by selecting Add slack and configuring the channel details.
Configure the PagerDuty notification channel by selecting Add PagerDuty and entering the credentials.
Configure the Webhook notification channel by selecting Add Webhook and entering the credentials.

Step 3. Configure the notification frequency

Choose the notification frequency. Valid options are:

Hourly notifications
Daily notifications at midnight UTC

Notifications are sent until the index rate reports a value that is below the thresholds that you have configured.

Individual recipients may mute alert emails.

You can also configure a custom schedule to set the days and hours that you want alerts to be send if the thresholds are exceeded.

Configuring index rate alerts using the API

You can configure index rates and what alerts are triggered using the API.

Creating or updating an index rate alert

Use the PUT REST API command to create or update your index rate alerts. If no index rate alert is configured, the first PUT will create the index rate alert.

Run the following command:

curl -X PUT https://<ENDPOINT>/v1/config/index-rate -H 'content-type: application/json'   -H 'servicekey: <SERVICE_KEY>' -d '{"max_lines":<MAX_LINES>,"max_z_score":<MAX_Z_SCORE>,"enabled":<ENABLED>,"channels":<CHANNELS>,"threshold_alert":"<THRESHOLD_ALERT>","frequency":"<FREQUENCY>"}'

Where:

ENDPOINT: Is the [endpoint]/docs/activity-tracker?topic=activity-tracker-endpoints) for your IBM Cloud Activity Tracker instance.
SERVICE_KEY: Is the service key for your IBM Cloud Activity Tracker instance.
MAX_LINES: Is the number of lines required in order to trigger the alert.
MAX_Z_SCORE: Is the number of standard deviations above the 30 day average lines in order to trigger the alert.
ENABLED: Specify if the index rate configuration is enabled (true) or disabled (false).
CHANNELS: Can be email or pagerduty. If email then specify a comma-separated list of email addresses. For example, {"email":["emailaddress@company.com"]}. If pagerduty specify the PagerDuty key.
THRESHOLD_ALERT: Is tetermines if you want alerts to be triggered if both the max lines and standard deviation have been triggered, or if one of the thresholds has been reached. Valid values are separate and both.
FREQUENCY: Determines if recipients are notified once per hour, or once per day, (starting when the threshold is passed the first time) until the index rate drops below the thresholds. When the index rate drops below the thresholds, alerting stops. Valid values are hourly and daily

Getting the index rate alert configuration

You can retrieve your index rate alert configuration by running the following:

curl -X GET https://<ENDPOINT>/v1/config/index-rate -H 'content-type: application/json'   -H 'servicekey: <SERVICE_KEY>'

Where:

ENDPOINT: Is the [endpoint]/docs/activity-tracker?topic=activity-tracker-endpoints) for your IBM Cloud Activity Tracker instance.
SERVICE_KEY: Is the service key for your IBM Cloud Activity Tracker instance.

Output similar to the following is returned:

{"max_lines":1,"max_z_score":3,"enabled":true,"channels":{"email":["emailaddress@company.com"]},"threshold_alert":"both","frequency":"hourly"}

If no index rate alert is configured, the following is returned:

{"error":"No Index Rate Alert found","code":"NotFound","status":"error"}

Disabling an index rate alert

Use the PUT REST API command to disable your index rate alerts. To disable your index rate alert configuration, run the following:

curl -X PUT https://<ENDPOINT>/v1/config/index-rate -H 'content-type: application/json'   -H 'servicekey: <SERVICE_KEY>' -d '{"max_lines":0,"max_z_score":0,"enabled":true,"channels":{"email":[""]},"threshold_alert":"","frequency":""}'

Where:

ENDPOINT: Is the [endpoint]/docs/activity-tracker?topic=activity-tracker-endpoints) for your IBM Cloud Activity Tracker instance.
SERVICE_KEY: Is the service key for your IBM Cloud Activity Tracker instance.