Enabling mirroring

Example CLI command

ibmcloud resource service-instance-update "Event Streams resource instance name" -p '{"mirroring":{"source_crn":"<source_crn>", "source_alias":"<source_alias>", "target_alias":"<target_alias>"}}'

Select the topics to mirror

When the service instance update is complete, you must select which topics will be mirrored from the source to the target cluster. This is done with the CLI by using the 'ibmcloud es mirroring-topic-selection-set' command. Any consumer groups used to consume from these selected topics will be mirrored from the source to the target cluster. Topic selection is in the form of a regex pattern, or comma-separated list of such patterns.

The following command selects all topics to be mirrored:

ibmcloud es mirroring-topic-selection-set --select '.*'

You can select topics by listing the topics you want to mirror as follows:

ibmcloud es mirroring-topic-selection-set --select topic1,topic2,topic3

For more information about making the selection, see Mirroring user controls.

After topic selection is complete, the target cluster shows the topics that are selected for mirroring using the Mirroring user controls suffixed with the source cluster's alias.

Step 3.1: Specify how topic and group names are transformed

You can specify transformation rules that allow you to mirror data into topics with different names in the target cluster. The following three scenarios describe possible transformations, and explain the use-cases for each.

You can specify which topics or consumer groups are mirrored at anytime once mirroring has been enabled, however topic or group transformation is only possible at the point mirroring is enabled. If mirroring is already enabled, it will need to be disabled first before a subsequent enable request is made to specify topic or group transformation.

Scenario 1: Transforming topics by removing the old prefix or suffix and adding a new prefix or suffix

Configure the following four additional parameters.

The ibmcloud resource service-instance-update command needs to be specified via the -p command line argument. When these options are specified, only topics with the matching prefixes or suffixes will be eligible for mirroring. For example, if you have a remove_prefix of app1-, and specify a topic selection of abc.*, only topics that start with app1-abc will be mirrored.

If you specify the "rename" type of transformation and don't specify either the parameters for add_prefix or add_suffix, the mirrored topic in the target cluster will have these parameters removed. Topic patterns are applied to the topic name after any source prefix or suffix is removed and before any prefix or suffix has been added.

See the following CLI command example:

{
  "mirroring": {
    "source_crn": "crn:v1:...",
    "source_alias": "source",
    "target_alias": "target",
    "options": {
      "topic_name_transform": {
        "type": "rename",
        "rename": {
          "add_prefix": "newprefix-",
          "remove_prefix": "oldprefix-",
          "add_suffix": "-newsuffix",
          "remove_suffix": "-oldsuffix"
        }
      }
    }
  }
}

Scenario 2: Adding the source alias as a suffix to mirrored topics

Apply a topic name transformation with the topic_name_transform type set to use_alias. With this configuration, a topic called app1-topic in the source cluster will be mirrored to a topic called app1-topic.source in the target cluster, because the source alias specified in the configuration is source.

See the following CLI command example:

{
  "mirroring": {
    "source_crn": "crn:v1:...",
    "source_alias": "source",
    "target_alias": "target",
    "options": {
        "topic_name_transform": {
            "type": "use_alias"
      }
    }
  }
}

Scenario 3: Topics are mirrored with their names unchanged

In this scenario, you also apply the topic_name_transform with the type set to none. With this configuration, a topic called app1-topic in the source cluster will be mirrored to a topic called app1-topic in the target cluster.

See the following CLI command example:

{
  "mirroring": {
    "source_crn": "crn:v1:...",
    "source_alias": "source",
    "target_alias": "target",
    "options": {
        "topic_name_transform": {
            "type": "none"
      }
    }
  }
}

Step 3.2: Transforming corresponding consumer group IDs

By default, Mirror Maker will not modify consumer group IDs when mirroring to the target cluster. However, Event Streams allows you to modify group IDs' data, as outlined in the following two scenarios. Similar to topics, group ID patterns are applied after removing any source prefix or suffix and before adding any prefix or suffix. If you specify the "rename" type of transformation and don't specify either the parameters for add_prefix or add_suffix, the mirrored group ID in the target cluster will have these parameters removed.

The ibmcloud resource service-instance-update command needs to be specified via the -p command line argument.

Scenario 1: Transform group ID by removing the old prefix or suffix and adding a new prefix or suffix

Configure the following four additional parameters.

When these options are specified, only group IDs with the matching prefixes or suffixes will be eligible for mirroring. For example, if you have a remove_prefix of aaa and add_prefix of bbb, consumer groups that start with aaa-group-id in the source cluster will be mirrored to bbb-group-id in the target cluster.

See the following CLI command example:

{
  "group_id_transform": {
    "type": "rename",
    "rename": {
       "add_prefix": "newprefix-",
       "remove_prefix": "oldprefix-",
       "add_suffix": "-newsuffix",
       "remove_suffix": "-oldsuffix"
    }
  }
}

Scenario 2: Consumer group IDs are mirrored with their names unchanged

In this scenario, you also apply the topic_name_transform with the type set to none. With this configuration, a topic called aaa-group-id in the source cluster will be mirrored to a topic called aaa-group-id in the target cluster.

See the following CLI command example:

"group_id_transform": {
  "type": "none"
}