Migrate Continuous Delivery resources to another region

Limitations for Toolchains and Delivery Pipeline

Migration of resources from one region to another is subject to the following limitations.

1. Classic pipelines are not supported.
2. DevOps Insights is not supported.
3. Secrets stored directly in Toolchains or Delivery Pipeline (environment properties or trigger properties) will not be copied. An export-secrets command is provided to export secrets into a Secrets Manager instance, replacing the stored secrets with secret references. Secret references are supported.
4. Tekton pipeline webhook trigger secrets will not be copied, as references are not supported for webhook trigger secrets. You will need to add the secret after copying the toolchain.
5. Tekton pipeline run history, logs, and assets will not be copied. You can keep the original pipelines for some time to retain history.
6. GitHub and Git Repos and Issue Tracking tool integrations configured with OAuth type authentication will automatically be converted to use the OAuth identity of the user performing the copy (the owner of the API key) rather than the original user. This is to simplify the copy operation. You can re-configure the tool integrations after copying to use a different user.
7. Git Repos and Issue Tracking tool integrations that use Personal Access Tokens (PATs) for authentication will automatically be converted to use OAuth. You can re-configure the tool integrations after copying to use a PAT again.

Limitations for Git Repos and Issue Tracking

The following limitations apply only if you are migrating Git Repos and Issue Tracking projects.

1. Personal projects are not supported. If you created a project under a personal namespace, you can either move your personal project to a group or convert your personal namespace into a group, then update references in the toolchain with the new URL. It is recommended that you store projects in groups, as they allow multiple administrators and allow better continuity of a project over time.
2. Projects are copied using the GitLab direct transfer feature, which is subject to certain limitations.
3. Copying large projects, or projects with large files or many resources, can take time.
4. As each region of Git Repos and Issue Tracking is independent, your projects' users may not yet exist in the destination region. The copy-project-group command will ensure that the users exist in the new region, however there may be user name conflicts with other users in the destination region. In the event of a user name conflict, the user name in the destination region may be changed slightly by adding a suffix.

Important notes

You must review the following important notes before you begin migration.

Billing considerations

During the migration, you will need to create a new Continuous Delivery instance in the destination region and resource group to enable your toolchains, pipelines, and projects in the destination region. You may also wish to keep the original resources in the source region available for use until you have transitioned to the new region. If you use the Continuous Delivery service with the Professional plan, be aware that you will be charged for both regions according to the number of authorized users configured in each instance. If costs are a concern, you can change the plan in the source region's Continuous Delivery instance to Lite once you have transitioned to the new region. The resources will be read-only if you have exceeded the Lite plan limits. However you can change back to the Professional plan at any time if you wish to use it again. Learn more about Continuous Delivery billing and plans.

Duplicate Pipeline runs

During the migration, you may create new pipelines in the destination region. If those pipelines have timed triggers set to run automatically on a schedule, or Git triggers configured to run automatically on Git events like PRs or commits, those events could trigger duplicate pipeline runs (one in the original pipeline and one in the new pipeline). To avoid potential disruption, copied pipelines will have these types of triggers disabled by default. It is recommended to manage these triggers such that there is only one set enabled at a time. Once you are comfortable with the transition to the new pipeline, you can enable the triggers on it and disable the triggers on the original pipeline.

Hardcoded assumptions

After the migration, your Git Repos and Issue Tracking repos (if applicable) will have a different URL, and your toolchains and pipelines will have different IDs and URLs. There may be some assumptions about the URL/ID or location of your resources in your Tekton definitions, scripts, environment properties, or other automation. It is your responsibility to update these after migration.

macOS

Run the following commands to install dependencies on macOS.

brew install node
brew tap hashicorp/tap
brew install hashicorp/tap/terraform

Other platforms

• Node.js install instructions
• Terraform install instructions

CRN

IBM Cloud resources are uniquely identified by a Cloud Resource Name (CRN). You will need the CRN of the toolchain you want to copy. You can get the CRN of a toolchain a few ways:

1. Locate the toolchain in the Platform Automation > Toolchains page, open the toolchain, and click Details to see the toolchain details, which shows the CRN.
2. Locate the toolchain in the Resource list page, click on the toolchain row to expand the details panel, which shows the CRN.
3. Using the ibmcloud cli, you can list toolchains and their CRNs via

   ibmcloud resource service-instances --service-name toolchain --long
   

4. Using the Toolchain API.

Check for stored toolchain/pipeline secrets

Toolchains and Tekton pipelines can contain secrets, which are sensitive values such as API keys or passwords, in the following places:

• Tool integration properties, e.g. the Service ID API Key property of the Delivery Pipeline Private Worker tool integration
• Tekton pipeline environment properties
• Tekton pipeline trigger properties

There are two ways to configure secrets:

1. Stored directly in the toolchain or pipeline
2. Referencing secrets stored in a secret storage service such as IBM Cloud Secrets Manager or IBM Cloud Key Protect.

Copying a toolchain will automatically include secret references and those references will remain intact in the new toolchain. However, to minimize the risk of leaking sensitive data, secrets stored directly in toolchains or pipeline will not be included in the toolchain copy. You can use the export-secrets command described in the next section or manually input the secrets again in the copied toolchain or pipeline after copying. Note, however, that if you do not export the secrets, some tool integrations may not provision successfully when copying the toolchain if they are missing required secret values, and may need to be recreated manually after running the command.

First, check whether your toolchain or its Tekton pipelines contain any stored secrets that are not references by running:

npx @ibm-cloud/cd-tools export-secrets -c ${CRN} --check

Export stored toolchain/pipeline secrets to Secrets Manager

If your toolchain or pipelines do not contain any stored secrets, you can skip this step and continue to copying the toolchain. Exporting secrets to Secrets Manager will create secrets in the Secrets Manager instance and also modify your original toolchain to convert the existing secrets to reference the newly created secrets in Secrets Manager. This will allow the toolchain to be copied with secret references intact, and is a recommended practice for added security.

To prevent accidental exposure of secrets, you should review the IAM permissions of your Secrets Manager instance to ensure that only intended access to read secrets is granted.

To export secrets stored in your toolchain or pipeline to Secrets Manager, follow these steps:

1. If you do not yet have a Secrets Manager instance, create one. Note that the instance must be created in the account associated with the API key you'll be using.
2. Ensure that the owner of the API key you'll be using has IAM permission to create secrets in the Secrets Manager instance.
3. Open the toolchain and Secrets Manager tool integration, create and authorization policy when prompted, and then create the tool integration.
4. Run the export-secrets command to export the secrets:

   npx @ibm-cloud/cd-tools export-secrets -c ${CRN}
   

5. When prompted, select the Secrets Manager instance from your toolchain to store the secrets. If you do not see your instance listed, it may be in a different account. Ensure that you use an API key that is in the same account as the instance.
6. When prompted, for each secret found, specify whether or not to copy the secret, and the name and group to store the secret in, or press enter to accept the defaults.

You can run the command as many times as needed to export all your secrets.

Copy toolchains

To copy a toolchain, run the @ibm-cloud/cd-tools copy-toolchain command. To see the available options, run:

npx @ibm-cloud/cd-tools copy-toolchain -h
Usage: @ibm-cloud/cd-tools copy-toolchain [options]

Copies a toolchain, including tool integrations and Tekton pipelines, to another region or resource group.

Examples:
  export IBMCLOUD_API_KEY='...'
  npx @ibm-cloud/cd-tools copy-toolchain -c ${TOOLCHAIN_CRN} -r us-south
      Copy a toolchain to the Dallas region with the same name, in the same resource group.
  npx @ibm-cloud/cd-tools copy-toolchain -c ${TOOLCHAIN_CRN} -r eu-de -n new-toolchain-name -g new-resource-group --apikey ${APIKEY}
      Copy a toolchain to the Frankfurt region with the specified name and target resource group, using the given API key

Environment Variables:
  IBMCLOUD_API_KEY                       API key used to authenticate. Must be a user API key, with IAM permission to read and create toolchains and service-to-service authorizations in source and target
region / resource group

Basic options:
  -c, --toolchain-crn <crn>              The CRN of the source toolchain to copy
  -r, --region <region>                  The destination region of the copied toolchain (choices: "br-sao", "eu-de", "eu-gb", "jp-tok", "us-south")
  -a, --apikey <api_key>                 API key used to authenticate. Must be a user API key, with IAM permission to read and create toolchains and service-to-service authorizations in source and target
                                         region / resource group
  -n, --name <name>                      (Optional) The name of the copied toolchain (default: same name as original)
  -g, --resource-group <resource_group>  (Optional) The name or ID of destination resource group of the copied toolchain (default: same resource group as original)
  -t, --tag <tag>                        (Optional) The tag to add to the copied toolchain
  -h, --help                             Display help for command

Advanced options:
  -d, --terraform-dir <path>             (Optional) The target local directory to store the generated Terraform (.tf) files
  -D, --dry-run                          (Optional) Skip running terraform apply; only generate the Terraform (.tf) files
  -f, --force                            (Optional) Force the copy toolchain command to run without user confirmation
  -S, --skip-s2s                         (Optional) Skip creating toolchain-generated service-to-service authorizations
  -T, --skip-disable-triggers            (Optional) Skip disabling Tekton pipeline Git or timed triggers. Note: This may result in duplicate pipeline runs
  -C, --compact                          (Optional) Generate all resources in a single resources.tf file
  -v, --verbose                          (Optional) Increase log output
  -q, --quiet                            (Optional) Suppress non-essential output, only errors and critical warnings are displayed

The copy-toolchain works by first translating the toolchain into Terraform (.tf) files, and then applying the Terraform to create a new toolchain in the destination region. The command will display the Terraform output and prompt for confirmation before creating the toolchain. You can review the Terraform output before creating the new toolchain copy.

export IBMCLOUD_API_KEY='<your_api_key>'
npx @ibm-cloud/cd-tools copy-toolchain -c 'crn:v1:bluemix:public:toolchain:au-syd:a/9d5d528aa786af01ce99593a827a05f0:69e8d78b-0d1a-49ed-9a46-3b4c1bb4f24a::' -r jp-tok

npx @ibm-cloud/cd-tools copy-toolchain -c "${CRN}" -r eu-de --apikey '<your_api_key>'

export IBMCLOUD_API_KEY='<your_api_key>'
npx @ibm-cloud/cd-tools copy-toolchain -c "${CRN}" -r us-south -n 'toolchain-dallas'

for i in $(ibmcloud resource service-instances --service-name toolchain --location ca-tor --all-resource-groups -o json | jq -r '.[].crn'); do
    npx @ibm-cloud/cd-tools copy-toolchain -c ${i} -r us-south --dry-run -f
done

for i in $(ibmcloud resource service-instances --service-name toolchain -g my-resource-group -o json | jq -r '.[].crn'); do
    npx @ibm-cloud/cd-tools copy-toolchain -c ${i} -r eu-de -q
done

for i in $(ibmcloud resource service-instances --service-name toolchain --all-resource-groups -o json | jq -r '.[] | select(.name | startswith("test-")) | .crn'); do
    npx @ibm-cloud/cd-tools copy-toolchain -c ${i} -r jp-tok
done

$ cd output-1764102115772
$ terraform apply
var.ibmcloud_api_key
  Enter a value: {api_key}
...

Verify resources

After copying your toolchains, Tekton pipelines, and Git Repos and Issue Tracking projects (if applicable) to a new region, you should verify that they were correctly copied and are functioning correctly before disabling or deleting the original resources. Please note:

• Tekton pipeline timed and Git triggers were not enabled by default in copied pipelines to prevent duplicate pipeline runs between the new and original pipeline. Once you are comfortable, you can enable the triggers in the new pipeline, and disable the triggers in the original pipeline.
• If you have any webhook type Tekton pipeline triggers, you will need to reconfigure the trigger and re-enter the secret. This secret does not support secret references and is not copied with the pipeline.
• Users with Personal Access Tokens in the copied Git Repos and Issue Tracking projects will need to recreate new tokens, as they were not copied.
• Tool integrations for Git Repos and Issue Tracking repos will have been converted to use the OAuth identity of the user who performed the copy. If you wish to use a different identity, log in with that user and re-save the tool integrations, or switch to use Personal Access Tokens.
• There may be assumptions in your Tekton definitions, scripts, environment properties, or other automations about the ID, URL, or location of your resources. It is recommended that you review these to ensure that the new IDs, URLs, and locations are used.

Disable original resources

After you have verified that the copied resources are functioning correctly, you can disable your original resources to avoid conflicts or confusion.

• For Tekton pipelines, you can disable your triggers to avoid unwanted pipeline runs and to signal to other users that these triggers should no longer be used.
• For Continuous Delivery service instances, if you were using the Professional plan, you can switch to the Lite plan to avoid further charges for the original resources. This may cause your resources to become read-only if you have exceeded the limits of the Lite plan, but you can switch back to Professional at any time if you need to use them again.
• For Git Repos and Issue tracking projects (if applicable), you can archive your original projects to make them read-only and prevent users from making further changes, and optionally update the project description or readme to indicate where the new project is.

Even if you do not intend to keep the original resources, you may want to keep the original resources for some time as a backup in case you discover issues later. Once you are comfortable, you can then delete the original resources.