Configuring buckets for long term storage and search
Configure a data bucket and a metrics bucket in IBM Cloud® Object Storage to store your IBM Cloud Logs data for long term storage and search.
You can configure the same bucket for data and metrics. However, notice that the IBM Cloud Logs service needs deletion permission on metrics files. Therefore, if you need to configure your bucket with lifecycle policies to manage log data files in the bucket, you must use different buckets to store your log data and your metrics data.
For production environments, consider using different buckets for data and for metrics.
You should create a bucket with Cross Region resiliency to store and access data across multiple geographical regions to ensure high availability, durability, and disaster recovery capabilities. See Creating and modifying IBM Cloud Object Storage buckets.
About buckets
IBM Cloud Object Storage is a highly available, durable, and secure platform for storing unstructured data. The files that are uploaded into IBM Cloud Object Storage are called objects. Objects can be anywhere from a few bytes up to 10TB. They are organized into buckets that serve as containers for objects, and which can be configured independently from one another in terms of locations, resiliency, billing rates, security, and object lifecycle. For more information, see What is IBM Cloud Object Storage?.
To manage buckets, your user must be granted permissions to work with buckets on the IBM Cloud Object Storage instance. For more information about roles, see Identity and Access Management roles.
To create a bucket, you can choose 1 of the following options:
| Action | More info |
|---|---|
| Create a bucket through the IBM Cloud UI | Learn more |
| Create a bucket through the IBM Cloud CLI | Learn more |
| Create a bucket by using cURL | Learn more |
| Create a bucket by using the REST API | Learn more |
| Create a bucket with a different storage class by using the REST API | Learn more |
| Create a bucket with Key Protect or Hyper Protect Crypto Services managed encryption keys (SSE-KP) by using the REST API | Learn more |
| Create a bucket by using Terraform | Learn more |
For more information, see Getting started with IBM Cloud Object Storage.
IAM Service to service authorization
You must define a service to service (S2S) authorization between IBM Cloud Logs and IBM Cloud Object Storage to allow IBM Cloud Logs to read and write data into the buckets. For more information, see Creating a S2S authorization to grant access to a bucket.
Data bucket
You can configure a data bucket for a IBM Cloud Logs instance.
-
The data bucket stores and retains logs for as long as you need them.
-
You can configure a data bucket in the same region where the IBM Cloud Logs instance is provisioned.
If you have regulatory and compliance requirements, check the location where you can create the bucket. Then, if performance is critical, consider creating the bucket in the same region where the IBM Cloud Logs instance is provisioned.
-
You are responsible for the maintenance of the data bucket.
For more information, see Configuring the data bucket.
Metrics bucket
You can configure a metrics bucket for a IBM Cloud Logs instance.
-
The metrics bucket stores and retains metrics from your events in a long-term index for as long as you need them.
When you enable metrics, you can generate metrics from logs. These metrics are stored in the metrics bucket as Prometheus index blocks.
-
You can configure a metrics bucket in the same region where the IBM Cloud Logs instance is provisioned.
-
You are responsible for the maintenance of the metrics bucket.
If you have regulatory and compliance requirements, check the location where you can create the bucket. Then, if performance is critical, consider creating the bucket in the same region where the IBM Cloud Logs instance is provisioned.
For more information, see Configuring the metrics bucket.