Replicating objects

What is replication?

Replication copies newly created objects and object updates from a source bucket to a target bucket.

• Only new objects or new versions of the existing objects (created after the replication rule was added to the bucket) are copied to the target bucket. Existing objects can be replicated by copying them onto themselves, creating a new version that is replicated.
• The metadata of the source object is applied to the replicated object.
• Bi-directional replication between two buckets requires rules to be active on both buckets.
• Filters (composed of prefixes and/or tags) can be used to scope the replication rule to only apply to a subset of objects. Multiple rules can be defined in a single policy and these rules can specify different destinations. In this manner, different objects in the same bucket can be replicated to different destinations.

Why use replication?

• Keep a copy of data in a bucket in a different geographic location.
• Meet compliance regulations for data sovereignty by defining replication rules that store replicas only within the allowable locations.
• Keep production and test data in sync, as replication retains object metadata such as last modified time, version ID, and so on.
• Manage the storage class and lifecycle policies for the replicated objects independent of the source, by defining a different storage class and/or lifecycle rules for the target bucket. Similarly, you can store replicas in a bucket in a separate service instance or even IBM Cloud account, and also independently control access to the replicas.

Getting started with replication

To get started, here are some prerequisites that must be met:

• Set the the Writer or Manager platform role on the source bucket, or a custom role with the appropriate replication actions (such as cloud-object-storage.bucket.put_replication) assigned.
• You do not need to have access to the target bucket, but do need to have sufficient platform roles to create new IAM policies that allow the source bucket to write to the target bucket.
• Both the source and target buckets must have versioning enabled.
• The target bucket must not have a legacy bucket firewall enabled, but can use context-based restrictions.
• Objects encrypted using SSE-C cannot be replicated, although managed encryption (SSE-KMS) like Key Protect is fully compatible with replication.
• Objects in an archived state cannot be replicated.
• If the source and target buckets are in different IBM accounts, be sure to create the buckets in each account.
• Enable Versioning on each bucket.

As versioning is a requirement for replication, it is impossible to replicate objects in buckets configured with an Immutable Object Storage policy.

Using one IBM account

To replicate objects between buckets in the same IBM account, do the following:

1. After navigating to your chosen source bucket, click the Configuration tab.
2. Look for Bucket replication and click the Setup replication button.
3. Select Replication source and click Next.
4. Select the instance and bucket from the drop-down menus. Alternatively, toggle the radio button to No and paste in the CRN of the target bucket.
5. Click on the Check permissions button.

Now, you'll need to grant the source bucket Writer permissions on the target bucket. There are several ways to do this, but the easiest is to use the IBM Cloud Shell and the IBM Cloud CLI.

1. Open an IBM Cloud Shell in a new window or tab.
2. Copy the IBM Cloud CLI command shown in the Object storage console, and paste it into the new shell.
3. Return to the bucket configuration window or tab, and click on the Check permissions button again.

Now you'll create a replication rule.

1. Ensure the rule status radio button is set to Enabled.
2. Give the rule a name and a priority, as well as any prefix or tag filters that will limit the objects subject to the replication rule.
3. Click Done.

Using different IBM accounts

To replicate objects between buckets in different IBM accounts, do the following:

1. Set up an IAM policy on the destination IBM account. For information about creating an IAM policy, see What are IAM policies and who can assign them.

2. Find the account ID and the Service instance ID in CRN format on the Bucket Configuration page.

3. Using the IBM Cloud UI of the destination account, click Manage>Access(IAM).

4. Click Authentication in the left panel.

5. Click Create to create a new IAM policy.

6. Grant a service authorization page configuration. This is the page where you will land after creating a new IAM policy.

7. Select Another account and provide the Account ID of the source account.

8. Provide service access as Cloud Object Storage.

9. In the Scope of Access, select Specific Resources.

10. Select Source Service Instance and enter the service instance ID for the source bucket.

11. Under Target, select Cloud Object Storage for the source bucket access.

12. For Target Scope, select Specific resources>**Service Instance.

13. Select the destination account's service instance ID from the drop down menu.

14. Select the role Object Writer or Writer as required.

    The Object writer role is sufficient to enable replication.

Terminology

Source bucket: The bucket for which a replication policy is configured. It is the source of replicated objects.

Target bucket: The bucket that is defined as the destination in the source bucket replication policy. It is the target of replicated objects. Also referred to as a 'destination' bucket.

Replica: The new object created in a target bucket because of a request made to a source bucket.

What is replicated?

New objects created via CopyObject, PutObject, or CompleteMultipartUpload will be replicated from the source bucket to the target bucket. The replicated objects will inherit the following metadata fields from the source object: Etag, Last Modified Time, Version ID, user-attributes, and Tags.

Delete markers will be replicated if configured by the replication policy.

Updates to a version's tags will be replicated from the source bucket to the target bucket.

The following are not replicated:

• Actions initiated by lifecycle events
• Objects written directly to archive
• Objects restored from an archive tier
• Objects encrypted via SSE-C
• Object ACLs
• Object-Lock state

Using replication for business continuity and disaster recovery

Replication can be used to provide continuity of service in the event of an outage:

• Ensure that the source and target buckets are in different locations.
• Verify that the latest versions of objects are in sync between both buckets. A tool such as Rclone (the rclone check command) can be useful for checking synchronicity from the command line.
• In the event of an outage, an application's traffic can be redirected to the target bucket.

Consistency and data integrity

While IBM Cloud Object Storage provides strong consistency for all data IO operations, bucket configuration is eventually consistent. After enabling replication rules for the first time on a bucket, it may take a few moments for the configuration to propagate across the system and new objects to start being replicated.

IAM actions

There are new IAM actions associated with replication.

Activity Tracker events

Replication generates additional events.

• cloud-object-storage.bucket-replication.create
• cloud-object-storage.bucket-replication.read
• cloud-object-storage.bucket-replication.delete
• cloud-object-storage.object-replication.sync (generated at the source)
• cloud-object-storage.object-replication.create (generated at the target)

For cloud-object-storage.bucket-replication.create events, the following fields provide extra information:

When replication is active, operations on objects may generate the following extra information:

You can trace an object from when it is written to the source until it is written on the target. Search for the request ID associated with the object write and three events should appear:

• The original PUT.
• The sync request from the source.
• The PUT request on the target.

Any of these three missing indicates a failure.

Usage and accounting

All replicas are objects themselves, and contribute usage just like any other data. Successful replication results in billable PUT, GET, and HEAD requests, although any bandwidth consumed in the replication process is not billed.

Replication generates additional metrics for use with IBM Cloud Monitoring:

• ibm_cos_bucket_replication_sync_requests_issued
• ibm_cos_bucket_replication_sync_requests_received

Interactions

Replicating existing objects

A replication rule can only act on objects that are written after the rule is configured and applied to a bucket. If there are existing objects in a bucket that should be replicated, the replication processes needs to be made aware of the existence of the objects. This can be easily accomplished by using the PUT copy operation to copy objects onto themselves.

This process will reset some object metadata, including creation timestamps. This will impact lifecycle policies and any other services that use creation or modification timestamps (such as content delivery networks). Ensure that any disruptions that may arise from resetting object metadata are dealt with appropriately.

The process involves:

1. Creating a list of all the objects in a bucket that should be subject to replication rules,
2. Iterating over that list, performing a PUT copy operation on each object with the source being identical to the target of the request.

This example will only replicate the new version of the object created by the PUT copy request. In order to replicate all versions of the object, it would be necessary to copy each individual version as well.

The following example is written in Python, but the algorithm could be applied in any programming language or context.

import os
import sys
import ibm_boto3
from ibm_botocore.config import Config

# Create client connection
cos = ibm_boto3.client("s3",
                       ibm_api_key_id=os.environ.get('IBMCLOUD_API_KEY'),
                       ibm_service_instance_id=os.environ['SERVICE_INSTANCE_ID'],
                       config=Config(signature_version="oauth"),
                       endpoint_url=os.environ['US_GEO']
                       )

# Define the bucket with existing objects for replication
bucket = os.environ['BUCKET']

def copy_in_place(BUCKET_NAME):
    print("Priming existing objects in " + bucket + " for replication...")

    paginator = cos.get_paginator('list_objects_v2')
    pages = paginator.paginate(Bucket=bucket)

    for page in pages:
        for obj in page['Contents']:
            key = obj['Key']
            print("  * Copying " + key + " in place...")
            try:
                headers = cos.head_object(
                    Bucket=bucket,
                    Key=key
                    )
                md = headers["Metadata"]
                cos.copy_object(
                    CopySource={
                        'Bucket': bucket,
                        'Key': key
                        },
                    Bucket=bucket,
                    Key=key,
                    TaggingDirective='COPY',
                    MetadataDirective='REPLACE',
                    Metadata=md
                    )
                print("    Success!")
            except Exception as e:
                print("    Unable to copy object: {0}".format(e))
    print("Existing objects in " + bucket + " are now subject to replication rules.")

copy_in_place(bucket)

REST API examples

The following examples are shown using cURL for ease of use. Environment variables are used to represent user specific elements such as $BUCKET, $TOKEN, and $REGION. Note that $REGION would also include any network type specifications, so sending a request to a bucket in us-south using the private network would require setting the variable to private.us-south.

curl -X "PUT" "https://$BUCKET.s3.$REGION.cloud-object-storage.appdomain.cloud/?replication" \
     -H 'Authorization: bearer $TOKEN' \
     -H 'Content-MD5: exuBoz2kFBykNwqu64JZuA==' \
     -H 'Content-Type: text/plain; charset=utf-8' \
     -d $'<ReplicationConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
            <Rule>
              <ID>SimpleReplication</ID>
              <Priority>1</Priority>
              <Status>Enabled</Status>
              <DeleteMarkerReplication>
                <Status>Disabled</Status>
              </DeleteMarkerReplication>
              <Filter/>
              <Destination>
                <Bucket>$DESTINATION_CRN</Bucket>
              </Destination>
          	</Rule>
          </ReplicationConfiguration>'

curl -X "PUT" "https://$BUCKET.s3.$REGION.cloud-object-storage.appdomain.cloud/?replication" \
     -H 'Authorization: bearer $TOKEN' \
     -H 'Content-MD5: exuBoz2kFBykNwqu64JZuA==' \
     -H 'Content-Type: text/plain; charset=utf-8' \
     -d $'<ReplicationConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
            <Rule>
              <ID>ProjectA</ID>
              <Priority>10</Priority>
              <Status>Enabled</Status>
              <DeleteMarkerReplication>
                <Status>Enabled</Status>
              </DeleteMarkerReplication>
              <Filter>
                <Prefix>project_a/</prefix>
              </Filter>
              <Destination>
                <Bucket>$DESTINATION_CRN_A</Bucket>
              </Destination>
          	</Rule>
            <Rule>
              <ID>ProjectB</ID>
              <Priority>5</Priority>
              <Status>Enabled</Status>
              <DeleteMarkerReplication>
                <Status>Enabled</Status>
              </DeleteMarkerReplication>
              <Filter>
                <Prefix>project_b/</prefix>
              </Filter>
              <Destination>
                <Bucket>$DESTINATION_CRN_B</Bucket>
              </Destination>
          	</Rule>
            <Rule>
              <ID>AcmeCorp</ID>
              <Priority>20</Priority>
              <Status>Enabled</Status>
              <DeleteMarkerReplication>
                <Status>Enabled</Status>
              </DeleteMarkerReplication>
              <Filter>
                <Tag>
                  <Key>Client</Key>
                  <Value>ACME</Value>
                </Tag>
              </Filter>
              <Destination>
                <Bucket>$DESTINATION_CRN_C</Bucket>
              </Destination>
          	</Rule>
          </ReplicationConfiguration>'

curl -X "GET" "https://$BUCKET.s3.$REGION.cloud-object-storage.appdomain.cloud/?replication" \
     -H 'Authorization: bearer $TOKEN'

<ReplicationConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Rule>
    <ID>SimpleReplication</ID>
    <Status>ENABLED</Status>
    <DeleteMarkerReplication>
      <Status>DISABLED</Status>
    </DeleteMarkerReplication>
    <Destination>
      <Bucket>crn:v1:bluemix:public:cloud-object-storage:global:a/9978e07eXXXXXXXX66c89c428028654:ef1c725e-XXXX-4967-bcc1-734c03a2b846:bucket:replication-destination</Bucket>
    </Destination>
    <Priority>1</Priority>
    <Filter/>
  </Rule>
</ReplicationConfiguration>

curl -X "DELETE" "https://$BUCKET.s3.$REGION.cloud-object-storage.appdomain.cloud/?replication" \
     -H 'Authorization: bearer $TOKEN'

SDK examples

The following examples make use of the IBM COS SDKs for Python and Node.js, although the implementation of object versioning should be fully compatible with any S3-compatible library or tool that allows for the setting of custom endpoints. Using third-party tools requires HMAC credentials to calculate AWS V4 signatures. For more information on HMAC credentials, see the documentation.

#!/usr/bin/env python3

import ibm_boto3
from ibm_botocore.config import Config
from ibm_botocore.exceptions import ClientError

# Define constants
API_KEY = os.environ.get('IBMCLOUD_API_KEY')
SERVICE_INSTANCE = os.environ.get('SERVICE_INSTANCE_ID')
ENDPOINT = os.environ.get('ENDPOINT')

BUCKET = "my-replication-bucket" # The bucket that will enable replication.

# Create resource client with configuration info pulled from environment variables.
cosClient = ibm_boto3.client("s3",
                         ibm_api_key_id=API_KEY,
                         ibm_service_instance_id=SERVICE_INSTANCE,
                         config=Config(signature_version="oauth"),
                         endpoint_url=ENDPOINT
                         )

response = cosClient.put_bucket_versioning(
    Bucket=BUCKET,
    ReplicationConfiguration={
        'Rules': [
            {
                'ID': 'string',
                'Priority': 123,
                'Filter': {
                    'Prefix': 'string',
                    'Tag': {
                        'Key': 'string',
                        'Value': 'string'
                    },
                    'And': {
                        'Prefix': 'string',
                        'Tags': [
                            {
                                'Key': 'string',
                                'Value': 'string'
                            },
                        ]
                    }
                },
                'Status': 'Enabled'|'Disabled',
                'Destination': {
                    'Bucket': 'string',
                },
                'DeleteMarkerReplication': {
                    'Status': 'Enabled'|'Disabled'
                }
            },
        ]
    }
)

resp = cosClient.list_object_versions(Prefix='some-prefix', Bucket=BUCKET)

const IBM = require('ibm-cos-sdk');

var config = {
    endpoint: '<endpoint>',
    apiKeyId: '<api-key>',
    serviceInstanceId: '<resource-instance-id>',
};

var cos = new IBM.S3(config);

var params = {
  Bucket: 'STRING_VALUE', /* required */
  ReplicationConfiguration: { /* required */
    Role: 'STRING_VALUE', /* required */
    Rules: [ /* required */
      {
        Destination: { /* required */
          Bucket: 'STRING_VALUE', /* required */
        },
        Status: Enabled | Disabled, /* required */
        Filter: {
          And: {
            Prefix: 'STRING_VALUE',
            Tags: [
              {
                Key: 'STRING_VALUE', /* required */
                Value: 'STRING_VALUE' /* required */
              },
              /* more items */
            ]
          },
          Prefix: 'STRING_VALUE',
          Tag: {
            Key: 'STRING_VALUE', /* required */
            Value: 'STRING_VALUE' /* required */
          }
        },
        ID: 'STRING_VALUE',
        Prefix: 'STRING_VALUE',
        Priority: 'NUMBER_VALUE',
        }
      }
    ]
  },
  ContentMD5: 'STRING_VALUE',
};
cos.putBucketReplication(params, function(err, data) {
  if (err) console.log(err, err.stack); // an error occurred
  else     console.log(data);           // successful response
});