Enabling mirroring

Example CLI command

ibmcloud resource service-instance-update "Event Streams resource instance name" -p '{"mirroring":{"source_crn":"<source_crn>", "source_alias":"<source_alias>", "target_alias":"<target_alias>"}}'

Select the topics to mirror

When the service instance update is complete, you must select which topics will be mirrored from the source to the target cluster. This is done with the CLI by using the 'ibmcloud es mirroring-topic-selection-set' command. Any consumer groups used to consume from these selected topics will be mirrored from the source to the target cluster. Topic selection is in the form of a regex pattern, or comma-separated list of such patterns.

The following command selects all topics to be mirrored:

ibmcloud es mirroring-topic-selection-set --select '.*'

You can select topics by listing the topics you want to mirror as follows:

ibmcloud es mirroring-topic-selection-set --select topic1,topic2,topic3

For more information about making the selection, see Mirroring user controls.

After topic selection is complete, the target cluster shows the topics that are selected for mirroring using the Mirroring user controls suffixed with the source cluster's alias.

Step 3.1: Specify how topic and group names are transformed

You can specify transformation rules that allow you to mirror data into topics with different names in the target cluster. The following three scenarios describe possible transformations, and explain the use-cases for each.

You can specify which topics or consumer groups are mirrored at anytime once mirroring has been enabled, however topic or group transformation is only possible at the point mirroring is enabled. If mirroring is already enabled, it will need to be disabled first before a subsequent enable request is made to specify topic or group transformation.

Scenario 1: Transforming topics by removing the old prefix or suffix and adding a new prefix or suffix

Configure the following four additional parameters.

The ibmcloud resource service-instance-update command needs to be specified via the -p command line argument. When these options are specified, only topics with the matching prefixes or suffixes will be eligible for mirroring. For example, if you have a remove_prefix of app1-, and specify a topic selection of abc.*, only topics that start with app1-abc will be mirrored.

If you specify the "rename" type of transformation and don't specify either the parameters for add_prefix or add_suffix, the mirrored topic in the target cluster will have these parameters removed. Topic patterns are applied to the topic name after any source prefix or suffix is removed and before any prefix or suffix has been added.

See the following CLI command example:

{
  "mirroring": {
    "source_crn": "crn:v1:...",
    "source_alias": "source",
    "target_alias": "target",
    "options": {
      "topic_name_transform": {
        "type": "rename",
        "rename": {
          "add_prefix": "newprefix-",
          "remove_prefix": "oldprefix-",
          "add_suffix": "-newsuffix",
          "remove_suffix": "-oldsuffix"
        }
      }
    }
  }
}

Scenario 2: Adding the source alias as a suffix to mirrored topics

Apply a topic name transformation with the topic_name_transform type set to use_alias. With this configuration, a topic called app1-topic in the source cluster will be mirrored to a topic called app1-topic.source in the target cluster, because the source alias specified in the configuration is source.

See the following CLI command example:

{
  "mirroring": {
    "source_crn": "crn:v1:...",
    "source_alias": "source",
    "target_alias": "target",
    "options": {
        "topic_name_transform": {
            "type": "use_alias"
      }
    }
  }
}

Scenario 3: Topics are mirrored with their names unchanged

In this scenario, you also apply the topic_name_transform with the type set to none. With this configuration, a topic called app1-topic in the source cluster will be mirrored to a topic called app1-topic in the target cluster.

See the following CLI command example:

{
  "mirroring": {
    "source_crn": "crn:v1:...",
    "source_alias": "source",
    "target_alias": "target",
    "options": {
        "topic_name_transform": {
            "type": "none"
      }
    }
  }
}

Step 3.2: Transforming corresponding consumer group IDs

By default, Mirror Maker will not modify consumer group IDs when mirroring to the target cluster. However, Event Streams allows you to modify group IDs' data, as outlined in the following two scenarios. Similar to topics, group ID patterns are applied after removing any source prefix or suffix and before adding any prefix or suffix. If you specify the "rename" type of transformation and don't specify either the parameters for add_prefix or add_suffix, the mirrored group ID in the target cluster will have these parameters removed.

The ibmcloud resource service-instance-update command needs to be specified via the -p command line argument.

Scenario 1: Transform group ID by removing the old prefix or suffix and adding a new prefix or suffix

Configure the following four additional parameters.

When these options are specified, only group IDs with the matching prefixes or suffixes will be eligible for mirroring. For example, if you have a remove_prefix of aaa and add_prefix of bbb, consumer groups that start with aaa-group-id in the source cluster will be mirrored to bbb-group-id in the target cluster.

See the following CLI command example:

{
  "group_id_transform": {
    "type": "rename",
    "rename": {
       "add_prefix": "newprefix-",
       "remove_prefix": "oldprefix-",
       "add_suffix": "-newsuffix",
       "remove_suffix": "-oldsuffix"
    }
  }
}

Scenario 2: Consumer group IDs are mirrored with their names unchanged

In this scenario, you also apply the topic_name_transform with the type set to none. With this configuration, a topic called aaa-group-id in the source cluster will be mirrored to a topic called aaa-group-id in the target cluster.

See the following CLI command example:

"group_id_transform": {
  "type": "none"
}

Cautions

Before synchronizing schemas using mirroring, review the following precautions:

1. A maintenance window is required while schemas are bulk exported/imported between the two registries – this would be in the order of a few hours or less.
2. For topic renaming to work, schemas must use Confluent Avro Serdes, so the subject can be derived from the topic name. Confluent Avro Serdes because the subject associated with a schema can be derived from the topic name (for example, for topic and topic/record subject naming strategies).
3. Mirroring S2S authorization must be uninterrupted; disabling s2s authorization or mirroring will prevent schema registry requests from being forwarded.
4. If transformation is required, the target instance schema registry must be configured with topic renaming rules before any migration takes place.
5. Renaming rules cannot be changed until migration is complete. Making changes during migration will result in inconsistencies between registries.

Instructions

The following instructions outline how you can use schema registry mirroring to move schemas between two instances.

An export utility has not yet been added to the CLI.

ibmcloud resource service-instance-update \
"trgt-instance-name" \
-p '{
"mirroring": {
"source_crn": "<src instance crn>",
"source_alias": "source",
"target_alias": "target",
"schemas": "proxied"
}
}'

Topic name transformation

Topic names can be transformed during forwarding. For example, with the right rules,old-my-topic could become new-my-topic. When enabled, the source instance only recognizes the original name, whilst the target instance only recognizes the new (transformed) topic name. All results returned are transformed accordingly.

If no transformation rules are supplied, use_alias is used, in line with existing mirroring behaviour in Event Streams. To forward without changing topic names, use topic_name_transform type none. Transformation is configured using the existing CLI transformation fields.

Migration flow

When migrating between two Event Streams instances, the following flow is suggested.

1. Enable mirroring between two instances, specifying schemas: proxied.
2. Update your applications to use the target schema registry.
3. Block any writes to the target registry by switching to schemas: read-only. Before making this change, mirroring must be disabled momentarily.
4. Export all schemas from the source instance.
5. Import all schemas exported from the source instance into the target. This can be done using the IBM Cloud® CLI: ibmcloud [...].
6. Disable mirroring.

Exporting schemas

For more detailed information, see the Confluent documentation.

The Event Streams CLI requires that schema imports have a v1 exportVersion value.

1. Download latest v2.x.x source code, for example, https://github.com/Apicurio/apicurio-registry/archive/refs/tags/2.6.13.Final.zip.

2. Build the export client: mvn -pl utils/exportConfluent -am -DskipTests -Pprod package.

3. Run the exporter (saves output to confluent-schema-registry-export.zip):

java -jar utils/exportConfluent/target/apicurio-registry-utils-exportConfluent-2.6.13.Final.jar
" https://token: @<my-event-streams-instance.com>/confluent"
--client-props basic.auth.credentials.source=URL

### Importing schemas {: #schemamigration_import}

Schemas can be imported using the Event Streams CLI.

1. Ensure you have the `event-streams[es]` plugin installed: `ibmcloud plugin list`.
2. Log in to IBM Cloud®: `ibmcloud login [...]`.
3. Initialize the Event Streams instance you'd like to import in to: `ibmcloud es init`.
4. Import the schema:
 
  ``` sh  {: pre}
ibmcloud es schema-import \
-f confluent-schema-registry-export.zip