Migrating from HPCS to Key Protect Dedicated with the Key Migration Tool

The Key Migration Tool (CRKM) helps HPCS users migrate Customer Root Keys (CRKs) to Key Protect Dedicated. This document describes how to use CRKM to migrate your keys and associated IBM Cloud resources. CRKM applies only to CRKs that are used by IBM Cloud services and IBM software that participate in the automated migration workflow. See Migrating from Hyper Protect Crypto Services (HPCS) to Key Protect Dedicated for detailed information.

In this topic, "key" refers to a customer root key (CRK) unless otherwise specified.

Downloading the tool

  1. Create an IBM Support ticket for Key Protect to request access to the HPCS to Key Protect migration tooling.

  2. Download the tool binary provided in the support ticket.

  3. Verify the SHA-256 checksum of the downloaded binary matches the value that is provided in the support ticket. Compare the values directly; they must match exactly.

Run the appropriate command for your operating system to get the SHA-256 checksum and compare with the value that is provided in the support ticket:

macOS

shasum -a 256 <crkm-binary>

Example:

shasum -a 256 crkm-darwin-arm64-1.0.0

Linux

sha256sum <crkm-binary>

Example:

sha256sum crkm-linux-amd64-1.0.0

Windows (Command Prompt)

certutil -hashfile <crkm-binary> SHA256

Example:

certutil -hashfile crkm-windows-amd64-1.0.0.exe SHA256

Windows (PowerShell)

Get-FileHash <crkm-binary> -Algorithm SHA256

Example:

Get-FileHash crkm-windows-amd64-1.0.0.exe -Algorithm SHA256

Make the binary executable (macOS/Linux):

chmod +x <crkm-binary>

Understanding the migration tool output

The status, create, and sync operations from the CLI tool produce the following two files:

.csv output file
Created after the operation completes, with execution in the file name, the operation name, and the date-time of command run. The file contains a copy of the input data, and for each row:
  • The number of associations that are found on the HPCS key
  • The number of associations that are found on the Key Protect Dedicated key
  • The status of the operation on the HPCS key
.log output file
Created after the operation completes, with summary in the file name, the operation name, and the date-time of command run. The file provides an overview of the status of keys with migration intents and their number of associations.

The status operation is useful for verifying that your input file is correctly formatted, because this command takes no action. You can use the same input file repeatedly without removing keys that are fully migrated, as the status indicates why the operation was not performed.

Before you begin

You must complete all migration prerequisites before using the migration tool.

Setting up IAM authorization policies

Configure IAM authorization policies from services to Key Protect Dedicated as documented. When you create a policy to Hybrid HPCS, you must select the "Enable authorizations to be delegated by source and dependent services" checkbox if the documentation requires it.

For examples, see:

Creating target keys

Create keys in Key Protect Dedicated that are the target of the migration.

Setting up the input file

The CLI takes as input a .csv file without a header row. Each row in the file contains:

  • First column: The CRN of the source HPCS key (sourceCRK)
  • Second column: The CRN of the target Key Protect Dedicated key (targetCRK)

Each row represents the intent to migrate any cloud resource encrypted by the sourceCRK to be encrypted by the targetCRK after migration is complete.

Example input file

(The following table shows the structure. Do not include headers in your CSV file.)

Table 1. Example input file format
Source HPCS Key CRN Target Key Protect Dedicated Key CRN
crn:v1:bluemix:public:hs-crypto:eu-fr2:a/00000000000000000000000000000000:deadbeef-0000-0000-0000-1234567890ab:key:deadbeef-0000-0000-0000-1234567890ab crn:v1:bluemix:public:kms:eu-fr2:a/00000000000000000000000000000000:deadbeef-0000-0000-0000-1234567890ab:key:deadbeef-0000-0000-0000-1234567890ab
crn:v1:bluemix:public:hs-crypto:eu-fr2:a/00000000000000000000000000000000:deadbeef-0000-0000-0000-1234567890ab:key:deadbeef-0000-0000-0000-1234567890ab crn:v1:bluemix:public:kms:eu-fr2:a/00000000000000000000000000000000:deadbeef-0000-0000-0000-1234567890ab:key:deadbeef-0000-0000-0000-1234567890ab
crn:v1:bluemix:public:hs-crypto:eu-fr2:a/00000000000000000000000000000000:deadbeef-0000-0000-0000-1234567890ab:key:deadbeef-0000-0000-0000-1234567890ab crn:v1:bluemix:public:kms:eu-fr2:a/00000000000000000000000000000000:deadbeef-0000-0000-0000-1234567890ab:key:deadbeef-0000-0000-0000-1234567890ab

Do not include a header row in the actual CSV file.

Configuring the migration tool

Run the following commands in your terminal:

export HPCS_API_ENDPOINT=<HPCS endpoint>
export KP_ST_API_ENDPOINT=<Key Protect Dedicated endpoint>
export IBMCLOUD_API_KEY=<IBM Cloud API key>
export IBMCLOUD_API_KEY_KP_ST=<IBM Cloud API key that can access Key Protect Dedicated instance>

Example for HPCS_API_ENDPOINT and KP_ST_API_ENDPOINT:

export HPCS_API_ENDPOINT=https://fadedbee-0000-0000-0000-1234567890ab.api.us-south.hs-crypto.appdomain.cloud
export KP_ST_API_ENDPOINT=https://fadedbee-0000-0000-0000-1234567890ab.api.us-south.kms.appdomain.cloud

The IBMCLOUD_API_KEY must be associated with an IAM identity (for example, an IBM Cloud user or service ID) that has the Manager role for both HPCS and Key Protect Dedicated.

The IBMCLOUD_API_KEY_KP_ST variable is required only if IBMCLOUD_API_KEY does not have access to the target Key Protect Dedicated instance. This occurs when:

  • The HPCS instance and Key Protect Dedicated instance are in different IBM Cloud accounts
  • Separate IAM identities are used for source and target environments

Optional environment variables

The following optional environment variables can be configured:

IAM_API_ENDPOINT
Takes a custom IAM endpoint if the default https://iam.cloud.ibm.com/identity/token is not used.
DEBUG_MODE
Takes true or false. When set to true, the tool produces detailed logs.

Validating the input file

Use the status operation to check the status of the migration and verify that your input file is correctly formatted. No migration action is taken with this command.

./<crkm-binary> status <.csv input file>

You might see "FAILED - No Migration Intent that is found for the key" in the status columns for all key inputs in the output .csv file and in the console log. This is expected when no migration intent is created yet.

Checking IAM authorization policies

Use the authz-check command to verify that IAM authorization policies are in place so that cloud services that currently use HPCS source keys can use the corresponding Key Protect Dedicated target keys after migration.

./<crkm-binary> authz-check <.csv input file>

The command works in two phases:

Phase 1: Fetch policies

The tool extracts the distinct account IDs from the TargetCRK column in the CSV, fetches all IAM authorization policies for each account, and writes them to JSON files (policies-{serviceName}-{accountID}.json) for reference.

Phase 2: Match associations

For each row in the CSV, the tool fetches all associations on the source HPCS key (the cloud resources encrypted by that key). The tool checks whether an IAM authorization policy exists that would allow each associated service to use the target Key Protect Dedicated key. The tool also looks up the resource group of each target Key Protect Dedicated instance through the IBM Cloud resource controller API so that policies that are scoped to a resource group can be matched.

For each association, the tool reports either:

MATCH
A valid authorization policy was found. The policy JSON is printed.
NO MATCH
No authorization policy was found. The tool prints a message that explains what is missing and a colored JSON template of a policy that would satisfy the match. In the template, required fields are shown in green and optional fields in blue.

A summary is printed at the end with the total number of checked, matched, and unmatched associations.

Authorization check flags

--match-all-authz-policies
By default, the tool stops at the first matching policy per association. Use this flag to log all matching policies.

Matching criteria

An authorization policy matches an association when all of the following are true:

  1. The policy's subjects (who is authorized) contain a serviceName and accountId matching the associated cloud service.
  2. The policy's resources (what is being accessed) contain a serviceName and accountId matching the target Key Protect Dedicated key's service.
  3. The policy's roles grant appropriate access (Reader, ReaderPlus, Writer, or Manager).
  4. If the associated service requires it (for example, databases-for-*, messages-for-rabbitmq, containers-kubernetes), the policy also includes the AuthorizationDelegator role.

If the policy specifies optional attributes such as resourceGroupId, serviceInstance, keyRing, or resource, those must also match the target key or its Key Protect Dedicated instance.

Creating migration intents

After you create a .csv file with the pairing of HPCS keys to migrate to their respective Key Protect Dedicated keys, use the create operation to create migration intents on the HPCS keys.

./<crkm-binary> create <.csv input file>

This operation performs the following actions:

  • Triggers sync events to services that have associations (cloud resources that use the keys being migrated) so they can start migrating the HPCS key
  • You can monitor event acknowledgment and migration in the activity tracking logs (see Migration that is not completed in time for troubleshooting)

If a migration intent exists on a key and you want to update it with an updated .csv input file, run the CLI with the --replace-mi flag.

See troubleshooting if there are issues that are observed in the creation of migration intents.

Syncing keys

After five minutes from the completion of the previous step, use the sync operation:

./<crkm-binary> sync <.csv input file>

This operation triggers sync events to services that have associations (cloud resources that use the keys being migrated). This step ensures that migration can occur even for services that use HPCS keys indirectly. For example, ICD uses COS to store backups, and COS uses Key Protect.

You must wait at least five minutes before checking the migration status. The migration is performed by HPCS-adopting services, which have up to four hours to respond to HPCS. For Event Streams, the migration might take up to one business day after the creation of a migration intent.

Checking migration status

Use the status operation to check on the status of the migration:

./<crkm-binary> status <.csv input file>

If the number of associations of an HPCS key was not zero at the start of the migration and drops to zero, the migration is complete.

If you observe issues in the status column or the number of HPCS associations (KpStAssociationsCount) is still non-zero, see Troubleshooting.

Troubleshooting

Use the following information to troubleshoot common issues with the migration tool.

Migration not completed in time

Sometimes services might encounter problems migrating keys.

Use the sync operation with the .csv file as input to notify services with associations to the key to try the migration again:

./<crkm-binary> sync <.csv input file>

If the number of associations of an HPCS key was not zero at the start of the migration and does not drop to zero, services might be encountering problems migrating keys. Alternatively, there might be stale associations. Create an IBM Support ticket for Key Protect and mention HPCS to Key Protect migration.

Invalid resource CRN format

Error message:

invalid key crn for key : crn must be specified to 10 segments

The CSV status column output also reads: "FAILED - crn must be specified to 10 segments"

Review the input .csv file and ensure the CRN for the HPCS keys and Key Protect Dedicated keys are formatted correctly.

Key not found

Error message:

2023/03/29 09:23:39 Sleeping for 100ms before processing row 5
2023/03/29 09:23:39 Retrieving Key...
2023/03/29 09:23:39 Get Key failed for keyID d4c4cbee-20ed-4deb-b59d-4674f70713b5 with the following error: kp.Error: correlation_id='643a49af-a3cc-4eba-b23b-368bbc33115d', msg='Not Found: Key could not be retrieved: Please see `reasons` for more details (KEY_NOT_FOUND_ERR)', reasons='[KEY_NOT_FOUND_ERR: key does not exist - FOR_MORE_INFO_REFER: https://cloud.ibm.com/apidocs/key-protect]'
2023/03/29 09:23:39 -----------------------------------------------

The CSV status column output also reads:

FAILED - kp.Error: correlation_id='643a49af-a3cc-4eba-b23b-368bbc33115d', msg='Not Found: Key could not be retrieved: Please see `reasons` for more details (KEY_NOT_FOUND_ERR)', reasons='[KEY_NOT_FOUND_ERR: key does not exist - FOR_MORE_INFO_REFER: https://cloud.ibm.com/apidocs/key-protect]'

The CRN for the key does not exist. Review the input .csv file and ensure that both the HPCS keys to migrate and the Key Protect Dedicated target keys exist.

Instance does not exist or the user is unauthorized

Error message:

2023/03/29 09:26:44 Retrieving Key...
2023/03/29 09:26:44 Get Key failed for keyID d4c4cbee-20ed-4deb-b59d-4674f70713b5 with the following error: kp.Error: correlation_id='4629f1af-1a9b-44ea-9139-3c5c39b9790e', msg='Unauthorized: Either the user does not have access to the specified resource, the resource does not exist, or the region is incorrectly set'
2023/03/29 09:26:44 -----------------------------------------------

The CSV status column output also reads:

FAILED - kp.Error: correlation_id='4629f1af-1a9b-44ea-9139-3c5c39b9790e', msg='Unauthorized: Either the user does not have access to the specified resource, the resource does not exist, or the region is incorrectly set'

The user does not have access to the key, or the instance in the CRN of the key does not exist.

Create command: No associations found

Error message:

2023/03/22 10:39:36 Retrieving Key...
2023/03/22 10:39:36 Key retrieved successfully
2023/03/22 10:39:36 Retrieving associations...
2023/03/22 10:39:36 Associations retrieved successfully
2023/03/22 10:39:36 Retrieving Migration Intent...
2023/03/22 10:39:36 Migration Intent retrieved successfully
2023/03/22 10:39:36 No associations found for the key
2023/03/22 10:39:36 -----------------------------------------------

The CSV status column output also reads: "FAILED - No associations were found for the key"

The HPCS key has no cloud resources that are associated with it. Therefore, you do not need to create a migration intent on the HPCS key or perform migration.

Create command: Migration intent exists

Error message:

2023/03/22 10:39:35 Retrieving Key...
2023/03/22 10:39:35 Key retrieved successfully
2023/03/22 10:39:35 Retrieving associations...
2023/03/22 10:39:35 Associations retrieved successfully
2023/03/22 10:39:35 Retrieving Migration Intent...
2023/03/22 10:39:35 Migration Intent retrieved successfully
2023/03/22 10:39:35 Migration Intent already exists
2023/03/22 10:39:35 -----------------------------------------------

The CSV status column output also reads: "FAILED - Migration Intent exists for the key"

The HPCS key already has an existing migration intent. To update the migration intent:

  1. Update the input .csv file with the new CRN of the Key Protect Dedicated key to migrate to
  2. Add the --replace-mi flag to the create command
  3. Run the command again

Status command: No migration intent found

Error message:

2023/03/22 13:15:56 Retrieving Key...
2023/03/22 13:15:56 Key retrieved successfully
2023/03/22 13:15:56 Retrieving associations...
2023/03/22 13:15:56 Associations retrieved successfully
2023/03/22 13:15:56 Retrieving Migration Intent...
2023/03/22 13:15:56 No Migration Intent found for keyID a2cd689e-9a49-4cd7-ba0d-07d25a0b3a25
2023/03/22 13:15:56 -----------------------------------------------

The CSV status column output also reads: "FAILED - No Migration Intent found for the key"

A migration intent does not exist for that HPCS key. Retry Creating migration intents, ensuring the key resource CRN is included in the input file, and check the status of the operation.

Sync command: No associations found

Error message:

2023/03/22 13:13:45 Retrieving Key...
2023/03/22 13:13:45 Key retrieved successfully
2023/03/22 13:13:45 Retrieving associations...
2023/03/22 13:13:45 Associations retrieved successfully
2023/03/22 13:13:45 Retrieving Migration Intent...
2023/03/22 13:13:45 Migration Intent retrieved successfully
2023/03/22 13:13:45 No associations found for keyID d4c3cbee-20ed-4deb-b59d-4674f70713b5
2023/03/22 13:13:45 -----------------------------------------------

The CSV status column output also reads: "FAILED - No associations were found for the key"

The HPCS key has no cloud resources that are associated with it. Therefore, you do not need to perform a syncing of resources. This also indicates that the migration is complete for the key.

Sync command: No migration intent found

Error message:

2023/03/22 13:15:13 Retrieving Key...
2023/03/22 13:15:13 Key retrieved successfully
2023/03/22 13:15:13 Retrieving associations...
2023/03/22 13:15:13 Associations retrieved successfully
2023/03/22 13:15:13 Retrieving Migration Intent...
2023/03/22 13:15:13 Migration Intent retrieved successfully
2023/03/22 13:15:13 No Migration Intent found for keyID a2cd689e-9a49-4cd7-ba0d-07d25a0b3a25
2023/03/22 13:15:13 -----------------------------------------------

The CSV status column output also reads: "FAILED - No Migration Intent found for the key"

A migration intent does not exist for that key. Retry Creating migration intents, ensuring the key resource CRN is included in the input file, and check the status of the operation.

Sync command: Request too early

Error message:

2023/03/29 09:17:06 Sleeping for 100ms before processing row 2
2023/03/29 09:17:06 Retrieving Key...
2023/03/29 09:17:06 Key retrieved successfully
2023/03/29 09:17:06 Retrieving associations...
2023/03/29 09:17:06 Associations retrieved successfully
2023/03/29 09:17:06 Retrieving Migration Intent...
2023/03/29 09:17:06 Migration Intent retrieved successfully
2023/03/29 09:17:06 Post Sync Resources for key...
2023/03/29 09:17:06 Post Sync Call Failed: kp.Error: correlation_id='8f088c29-2f61-4dc4-b00f-c99f7edd00fe', msg='Conflict: Could not initiate sync requests to associated resources: Please see `reasons` for more details (REQ_TOO_EARLY_ERR)', reasons='[REQ_TOO_EARLY_ERR: The key was updated recently: Please wait and try again: Sync cannot be performed until at least 2023-03-29T14:17:19Z - FOR_MORE_INFO_REFER: https://cloud.ibm.com/apidocs/key-protect]'
2023/03/29 09:17:06 -----------------------------------------------

The CSV status column output also reads:

FAILED - kp.Error: correlation_id='0cefcea8-7e29-4c4a-9721-00244eb9752c', msg='Conflict: Could not initiate sync requests to associated resources: Please see `reasons` for more details (REQ_TOO_EARLY_ERR)', reasons='[REQ_TOO_EARLY_ERR: The key was updated recently: Please wait and try again: Sync cannot be performed until at least 2023-03-29T14:17:19Z - FOR_MORE_INFO_REFER: https://cloud.ibm.com/apidocs/key-protect]'

The sync operation is being performed too quickly. You must wait at least five minutes before trying again.