Step 1: Configure custom archive retention tags in IBM Cloud Logs through the UI

Complete the following steps:

1. Navigate to your IBM Cloud Logs instance: Observability > Logging > Cloud Logs > Your Service Instance > Open Dashboard.

2. In the dashboard, click Data Pipeline > Retention Tags.

   The Archive retention tags page opens.

   

3. Click Activate to enable this feature.

4. Click Edit and enter custom values for 1 or more retention tags. You can also use the predefined values Short, Intermediate and Long.

   Define per tag how data is classified. Each tag is used to define a different retention period of the data. You can define your own values and criteria.

   If you plan to have different retention policies per TCO data pipeline, enter the following values:

   • high for Retention Tag 1, where high represents data processed through the Priority insights TCO data pipeline.
   • medium for Retention Tag 2, where medium represents data processed through the Analyze and alert TCO data pipeline.
   • low for Retention Tag 3, where low represents data processed through the Store and search TCO data pipeline.

   If you plan to have different retention policies per log priority, enter the following values:

   • debug for for Retention Tag 1, where critical represents data that have a log priority set to debug or verbose.
   • info for for Retention Tag 2, where info represents data that have a log priority set to info or warning.
   • critical for for Retention Tag 3, where debug represents data that have a log priority set to error or critical.

Step 1: Configure custom archive retention tags in IBM Cloud Logs by using the API

Complete the following steps:

1. Retrieve your access token programmatically by first creating a service ID API key for your application, and then exchanging your API key for an IBM Cloud IAM token. For more information, see Retrieving an access token.

2. Run the following cURL command to activate tags in your IBM Cloud Logs instance and to configure custom tags:

   curl -X PUT --location --header "Authorization: Bearer ${IAM_TOKEN}"   --header "Content-Type: application/json"   --data   '{
       "tags": [
       "tag1",
       "tag2",
       "tag3"
       ]
   }'   "https://<INSTANCE_API_ENDPOINT>/v1/log_data_retention_tags"
   

   Where INSTANCE_API_ENDPOINT is the API endpoint of your IBM Cloud Logs instance. For more information, see Service API endpoints.

Step 2: Create expiration rules in IBM Cloud Object Storage through the UI

Create an expiration rule for each custom archive retention tag, and an expiration rule for the Default value.

For more information, see see About deleting stale data with expiration rules and Deleting stale data with expiration rules.

To create expiration rules, complete the following steps:

1. Launch the IBM Cloud Object Storage dashboard: Dashboard

2. In Instances, select the instance where the bucket that is associated to the IBM Cloud Logs instance is available.

3. Select the bucket that is attached as a data bucket to your IBM Cloud Logs instance. Then, click Object Lifecycle.

4. In the Expiration Rules section, click Add a rule. The Add expiration rule wizard opens.

5. Enable the toggle Status and enter a Rule ID* that is meaningful for one of the tags that you have activated in your IBM Cloud Logs instance.

6. In the Filter Type section, choose Limit the scope of the rule using one or more filters

   Click Object tags.

   Click Add.

   Enter the key ICL_ARCHIVE_RETENTION.

   Enter the value of the tag. For example, Long.

   Make sure the value matches the archive retention tag in your IBM Cloud Logs instance. Tag values are case-sensitive.

   

7. In the Current version expiration section, choose an option to expire the current version of objects in your bucket after a specified number of days or after a specific date.

   Valid options are: Do not expire, Days to expire after and Expiration date.

   

8. In the Incomplete multi-part upload clean up section, choose the option Do not clean up.

   Valid options are: Do not clean up and Days to clean up after. However, tags are not supported with incomplete multi-part upload objects.

   If you choose the option Days to clean up after and specify a number of days, when you try to save the policy, you get the following error:BMCOSUI060000: AbortIncompleteMultipartUpload cannot be specified with Tags.

   You can run the following command to see if a bucket includes incomplete multi-part objects: ibmcloud cos multipart-uploads --bucket BUCKET_NAME --region REGION

   

9. Click Save.

Step 2: Create expiration rules in IBM Cloud Object Storage by using the API

Create an expiration rule for each custom archive retention tag, and an expiration rule for the Default value.

To create expiration rules, see About deleting stale data with expiration rules and Deleting stale data with expiration rules using APIs.

Step 3: Configure TCO policies

Create policies and configure the archive retention section. For more information, see Creating a TCO policy.

You can use the default tag to define a default expiration period that you can apply to data that is not explicitly managed through a custom object tag. You can use any of the custom retention tags that you have defined.

Complete the following steps:

1. Launch the IBM Cloud Logs UI.

2. Click the Data pipeline icon > TCO Optimizer.

3. Click Create policy.

4. In the Details section, complete the following tasks:

   Enter a policy name.

   Enter a description. The description is optional.

   Define the policy order. This order determines which rule is applied when multiple policies match. By default, the first policy has the highest priority.

5. In the Filters section, add 1 or more filters to configure the applications, subsystems, and severity values that are relevant for this policy.

   For applications and subsystems, criteria can be specified when the value matches one of: All, Is, Is Not, Includes, or Starts With.

6. In the Priority section, set the priority for the policy. The priority determines the pipeline for logs that are matched by the policy.

   Valid values are: High for data managed through Priority insights, Medium for data managed through Analyze and alert, Low for data managed through Store and search and Block for data that you drop and is not available for search.

   The default value is High.

7. In the Archive retention section, choose a retention tag.

   By default, the Default tag is selected.

   Notice that retention tags are available if they are defined and activated.

8. Click Apply.

Step 3: Configure TCO policies by using the API

Complete the following steps:

1. Retrieve your access token programmatically by first creating a service ID API key for your application, and then exchanging your API key for an IBM Cloud IAM token. For more information, see Retrieving an access token.

2. Run the following cURL command to configure a TCO policy:

   curl -X POST --location --header "Authorization: Bearer ${IAM_TOKEN}"   --header "Accept: application/json"   --header "Content-Type: application/json"   --data   '{
   "application_rule": {
     "name": "APP_NAME",
     "rule_type_id": "is"
   },
   "subsystem_rule": {
     "name": "SUBSYSTEM_NAME",
     "rule_type_id": "is"
   },
   "log_rules": {
     "severities": [
       "debug",
       "verbose",
       "info",
       "warning",
       "error"
     ]
   },
   "deleted": false,
   "description": "Policy for sending data to Priority Insights",
   "enabled": true,
   "archive_retention_tag": "TAG_VALUE",
   "name": "NAME",
   "order": 2,
   "priority": "PRIORITY"
   }'   "https://<INSTANCE_API_ENDPOINT>/v1/policies"
   

   Where

   INSTANCE_API_ENDPOINT is the API endpoint of your IBM Cloud Logs instance. For more information, see Service API endpoints.

   IAM_TOKEN is the IAM bearer token used to request authorization to create a TCO policy. For more information, see Retrieving an access token.

   TAG_VALUE is the value of one of the tags that you configured in a previous step.

   NAME is the name of the TCO Policy

   PRIORITY defines the data pipeline that is associated to the data that meets the conditions of this policy. Valid values are: type_block, type_low, type_medium, and type_high.

For more information, see Create a TCO policy.