Configuring buckets for long term storage and search

Configure a data bucket and a metrics bucket in IBM Cloud® Object Storage to store your IBM Cloud Logs data for long term storage and search.

You can configure the same bucket for data and metrics. However, notice that the IBM Cloud Logs service needs deletion permission on metrics files. Therefore, if you need to configure your bucket with lifecycle policies to manage log data files in the bucket, you must use different buckets to store your log data and your metrics data.

IBM Cloud Logs does not support IBM Cloud® Object Storage buckets configured with retention policies.

For production environments, consider using different buckets for data and for metrics.

You should create a bucket with Cross Region resiliency to store and access data across multiple geographical regions to ensure high availability, durability, and disaster recovery capabilities. See Creating and modifying IBM Cloud Object Storage buckets.

About buckets

IBM Cloud Object Storage is a highly available, durable, and secure platform for storing unstructured data. The files that are uploaded into IBM Cloud Object Storage are called objects. Objects can be anywhere from a few bytes up to 10TB. They are organized into buckets that serve as containers for objects, and which can be configured independently from one another in terms of locations, resiliency, billing rates, security, and object lifecycle. For more information, see What is IBM Cloud Object Storage?.

To manage buckets, your user must be granted permissions to work with buckets on the IBM Cloud Object Storage instance. For more information about roles, see Identity and Access Management roles.

To create a bucket, you can choose 1 of the following options:

Create bucket requests
Action	More info
Create a bucket through the IBM Cloud UI	Learn more
Create a bucket through the IBM Cloud CLI	Learn more
Create a bucket by using cURL	Learn more
Create a bucket by using the REST API	Learn more
Create a bucket with a different storage class by using the REST API	Learn more
Create a bucket with Key Protect or Hyper Protect Crypto Services managed encryption keys (SSE-KP) by using the REST API	Learn more
Create a bucket by using Terraform	Learn more

For more information, see Getting started with IBM Cloud Object Storage.

IAM Service to service authorization

You must define a service to service (S2S) authorization between IBM Cloud Logs and IBM Cloud Object Storage to allow IBM Cloud Logs to read and write data into the buckets. For more information, see Creating a S2S authorization to grant access to a bucket.

Data bucket

You can configure a data bucket for a IBM Cloud Logs instance.

The data bucket stores and retains logs for as long as you need them.
You can configure a data bucket in the same region where the IBM Cloud Logs instance is provisioned.

If you have regulatory and compliance requirements, check the location where you can create the bucket. Then, if performance is critical, consider creating the bucket in the same region where the IBM Cloud Logs instance is provisioned.
You are responsible for the maintenance of the data bucket.

For more information, see Configuring the data bucket.

Data bucket restrictions

IBM Cloud Object Storage buckets used by IBM Cloud Logs as data buckets can be configured only with the following storage classes:

Smart Tier
Standard

The following storage classes are not supported by IBM Cloud Logs as data buckets:

Vault
Cold Vault

IBM Cloud Object Storage allows you to define archive rules on buckets that archive objects automatically after the defined time period. Archived objects have a lower cost than regular objects, but need to be restored before they can be read again.

IBM Cloud Logs cannot read archived objects. IBM Cloud Logs searching of archived objects in the All Logs view, or querying in Archive queries, returns an error message.

IBM Cloud Object Storage buckets used as IBM Cloud Logs data buckets must not define archive rules that immediately archive objects, or archive objects within a few hours.

If you do not need to search logs older than a certain time period, for example, a month, you can define an IBM Cloud Object Storage archive rule to archive objects older that the time period required for searching. Do not configure archiving for a period of less than 7 days.

By archiving data that you do not need to search, you can retain the log data at a reduced cost. If required, you can restore archived objects if you need to search the data by using IBM Cloud Logs in the future.

Metrics bucket

You can configure a metrics bucket for a IBM Cloud Logs instance.

The metrics bucket stores and retains metrics from your events in a long-term index for as long as you need them.

When you enable metrics, you can generate metrics from logs. These metrics are stored in the metrics bucket as Prometheus index blocks.
You can configure a metrics bucket in the same region where the IBM Cloud Logs instance is provisioned.
You are responsible for the maintenance of the metrics bucket.

If you have regulatory and compliance requirements, check the location where you can create the bucket. Then, if performance is critical, consider creating the bucket in the same region where the IBM Cloud Logs instance is provisioned.

For more information, see Configuring the metrics bucket.