Enabling mirroring

Setup

Mirroring is not supported for a Satellite cluster, therefore neither can be a source nor a target cluster.

Ensure that you provision two Enterprise plan clusters. Both clusters must have the same throughput and storage capacity and have service-to-service bindings (see Step 2 for more information).

Because mirroring is unidirectional, decide which direction of mirroring you want. One cluster is the source and the other cluster is the target.

Decide which topics from your source cluster that you want to mirror. By default no topics are mirrored and you can enable mirroring by using the user controls after mirroring is enabled as shown in step 4. You must specify the selection as one or more patterns.

Consider your bandwidth requirements; is there enough bandwidth available in the source cluster? Your source cluster needs to have some headroom to run mirroring. See Choosing your plan for cluster bandwidth limits and use Event Streams metrics to determine how busy your source cluster is and whether it has the headroom for mirroring.

Although mirroring from a Enterprise multi-zone-region cluster to a Enterprise single-zone-region cluster and vice versa is allowed. We do not recommend this configuration unless you have specific residency requirements and are aware of the implications. The Service Level Agreement (SLA) policy of a Enterprise multi-zone-region cluster to a Enterprise single-zone-region cluster might be lower or vice versa.

Enable service-to-service bindings

You must configure a service-to-service binding between both instances to allow both instances to communicate. To configure, complete the following steps:

When creating a service-to-service binding, IAM uses the terminology 'source' and 'target' in the opposite way to that of Event Streams. In that the IAM Source account contains the Event Streams mirroring target instance, and vice-versa.

1. Select the IBM Cloud account containing the Event Streams mirroring source service instance.
2. Navigate to the Authorizations panel in IAM and click Create.
3. For the Source section:
   • If you are mirroring to a target instance in a different account, select "another account" under the Source heading, then pick the account containing the mirroring target instance. If you a mirroring between service instances in the same account, you can leave the default of "this account" selected.
   • Select the mirroring target Event Streams instance as the IAM source service instance.
4. For the Target selection, select the mirroring source Event Streams instance as the IAM target service instance.
5. Assign the Reader role and click Authorize.

If your requirement is to fail back, you also need the service-to-service binding in the opposite direction.

The following example shows how to use the command line to configure service-to-service binding.

1. Login to the IBM Cloud account containing the Event Streams instance that you want to act as the mirroring source instance:

   ibmcloud login -c <account containing mirroring source instance>
   

2. Setup an authorization policy, as follows:

   ibmcloud iam authorization-policy-create messagehub messagehub Reader --source-service-instance-id <instance id of the mirroring target cluster> [--source-service-account <account containing mirroring target instance>] --target-service-instance-id <instance id of the mirroring source cluster>
   ```    Note that the `--source-service-account` option can be omitted if you are setting up mirroring between two Event Streams instances in the same IBM Cloud account.
   
   

For more information about service-to-service bindings, see Manage authorizations panel and Using authorizations to grant access between services.

Enable mirroring

To enable mirroring, you need to run a service-instance-update command against your target cluster by using the CLI with the following required parameters:

• The source_crn is in this format: "crn:v1:bluemix:public:messagehub:us-south:a/aaa:aaaa::"
• The source_alias and the target_alias are the aliases that you want to configure for each of the two service instances when you enable mirroring. The aliases appear in topic names. Choose short and descriptive names. For example, "us-south" and "us-east".

ibmcloud resource service-instance-update "Event Streams resource instance name" -p '{"mirroring":{"source_crn":"<source_crn>", "source_alias":"<source_alias>", "target_alias":"<target_alias>"}}'

Validation

You can get the current service instance information by running the following command:

ibmcloud resource service-instance "Event Streams resource instance name" --output=json

Review the last operation section of the output. The information is continuously updated as the update proceeds. When the mirroring enablement process has completed, the last operation information indicates whether the update succeeded or the sync succeeded.

"last_operation": {
  "type": "update",
  "state": "in progress",
  "description": "Update in progress.",
  "updated_at": null,
  "cancelable": false
}

Run the command again until success is indicated as follows:

"last_operation": {
  "type": "update",
  "state": "succeeded",
  "description": "Update succeeded.",
  "updated_at": null,
  "cancelable": false
}

The IBM Cloud Monitoring dashboard Event Streams Mirroring shows the state of mirroring.

Select topics and consumer groups

When the service instance update has completed, we want to select some topics from the source cluster to mirror. This is done with the CLI by using the ibmcloud es mirroring-topic-selection-set command.

Topic selection is in the form of a regex pattern, or comma-separated list of such patterns.

The following command selects all topics to be mirrored:

ibmcloud es mirroring-topic-selection-set --select '.*'

You can select topics by listing the topics you want to mirror as follows:

ibmcloud es mirroring-topic-selection-set --select topic1,topic2,topic3

For more information about making the selection, see Mirroring user controls.

After the topic selection is completed, the target cluster shows the topics that are selected for mirroring using the Mirroring user controls suffixed with the source cluster's alias.

{
  "mirroring": {
    "source_crn": "crn:v1:...",
    "source_alias": "source",
    "target_alias": "target",
    "options": {
      "topic_name_transform": {
        "type": "rename",
        "rename": {
          "add_prefix": "newprefix-",
          "remove_prefix": "oldprefix-",
          "add_suffix": "-newsuffix",
          "remove_suffix": "-oldsuffix"
        }
      }
    }
  }
}

{
  "mirroring": {
    "source_crn": "crn:v1:...",
    "source_alias": "source",
    "target_alias": "target",
    "options": {
        "topic_name_transform": {
            "type": "use_alias"
      }
    }
  }
}

{
  "mirroring": {
    "source_crn": "crn:v1:...",
    "source_alias": "source",
    "target_alias": "target",
    "options": {
        "topic_name_transform": {
            "type": "none"
      }
    }
  }
}

{
  "group_id_transform": {
    "type": "rename",
    "rename": {
       "add_prefix": "newprefix-",
       "remove_prefix": "oldprefix-",
       "add_suffix": "-newsuffix",
       "remove_suffix": "-oldsuffix"
    }
  }
}

"group_id_transform": {
  "type": "none"
}