Migrating from HPCS to Key Protect Dedicated with the Key Migration Tool

The Key Migration Tool (CRKM) helps HPCS users migrate Customer Root Keys (CRKs) to Key Protect Dedicated. This document describes how to use CRKM to migrate your keys and associated IBM Cloud resources. CRKM applies only to CRKs that are used by IBM Cloud services and IBM software that participate in the automated migration workflow. See Migrating from Hyper Protect Crypto Services (HPCS) to Key Protect Dedicated for detailed information.

In this topic, "key" refers to a customer root key (CRK) unless otherwise specified.

Downloading the tool

  1. Create an IBM Support ticket for Key Protect to request access to the HPCS to Key Protect migration tools.

  2. Download the tool binary provided in the support ticket.

  3. Verify the SHA-256 checksum of the downloaded binary matches the value that is provided in the support ticket. Compare the values directly; they must match exactly.

Run the appropriate command for your operating system to get the SHA-256 checksum and compare with the value that is provided in the support ticket:

macOS

shasum -a 256 <crkm-binary>

Example:

shasum -a 256 crkm-darwin-arm64-1.0.0

Linux

sha256sum <crkm-binary>

Example:

sha256sum crkm-linux-amd64-1.0.0

Windows (Command Prompt)

certutil -hashfile <crkm-binary> SHA256

Example:

certutil -hashfile crkm-windows-amd64-1.0.0.exe SHA256

Windows (PowerShell)

Get-FileHash <crkm-binary> -Algorithm SHA256

Example:

Get-FileHash crkm-windows-amd64-1.0.0.exe -Algorithm SHA256

Make the binary executable (macOS/Linux):

chmod +x <crkm-binary>

Understanding the migration tool output

  • The status, create, and sync operations from the CLI tool produce the following two files:

    .csv output file
    Created after the operation completes, with execution in the file name, the operation name, and the date-time of command run. The file contains a copy of the input data, and for each row:
    • The number of associations that are found on the HPCS key
    • The number of associations that are found on the Key Protect Dedicated key
    • The status of the operation on the HPCS key
    • The state of the HPCS key and the Key Protect Dedicated key (for example, Active (1))
    .log output file
    Created after the operation completes, with summary in the file name, the operation name, and the date-time of command run. The file provides an overview of the status of keys with migration intents and their number of associations.

  • The delete operation produces only the execution .csv file.

  • The status operation is useful for verifying that your input file is correctly formatted, because this command takes no action. You can use the same input file repeatedly without removing keys that are fully migrated, as the status indicates why the operation was not performed.

For a migration to succeed, the source HPCS key must be in the Active or Deactivated state (so that it can unwrap data) and the target Key Protect Dedicated key must be in the Active state (so that it can wrap and unwrap data). If a key is in any other state, the tool prints a warning to the console because the migration is likely to fail.

Before you begin

Complete all migration prerequisites to use the migration tool.

Setting up IAM authorization policies

Configure IAM authorization policies from services to Key Protect Dedicated as documented. When you create a policy to Hybrid HPCS, select the Enable authorizations to be delegated by source and dependent services checkbox, if the documentation requires it.

For examples, see:

Creating target keys

Create keys in Key Protect Dedicated that are the target of the migration.

Setting up the input file

The CLI takes as input a .csv file without a header row. Each row in the file contains:

  • First column: The CRN of the source HPCS key (sourceCRK)
  • Second column: The CRN of the target Key Protect Dedicated key (targetCRK)

Each row specifies which cloud resources you want to migrate. The migration tool re-encrypts resources that use the sourceCRK with the targetCRK.

Example input file

(The following table shows the structure. Do not include headers in your CSV file.)

Table 1. Example input file format
Source HPCS Key CRN Target Key Protect Dedicated Key CRN
crn:v1:bluemix:public:hs-crypto:eu-fr2:a/00000000000000000000000000000000:deadbeef-0000-0000-0000-1234567890ab:key:deadbeef-0000-0000-0000-1234567890ab crn:v1:bluemix:public:kms:eu-fr2:a/00000000000000000000000000000000:deadbeef-0000-0000-0000-1234567890ab:key:deadbeef-0000-0000-0000-1234567890ab
crn:v1:bluemix:public:hs-crypto:eu-fr2:a/00000000000000000000000000000000:deadbeef-0000-0000-0000-1234567890ab:key:deadbeef-0000-0000-0000-1234567890ab crn:v1:bluemix:public:kms:eu-fr2:a/00000000000000000000000000000000:deadbeef-0000-0000-0000-1234567890ab:key:deadbeef-0000-0000-0000-1234567890ab
crn:v1:bluemix:public:hs-crypto:eu-fr2:a/00000000000000000000000000000000:deadbeef-0000-0000-0000-1234567890ab:key:deadbeef-0000-0000-0000-1234567890ab crn:v1:bluemix:public:kms:eu-fr2:a/00000000000000000000000000000000:deadbeef-0000-0000-0000-1234567890ab:key:deadbeef-0000-0000-0000-1234567890ab

Do not include a header row in the actual CSV file.

Configuring the migration tool

Run the following commands in your terminal:

export HPCS_API_ENDPOINT=<HPCS endpoint>
export KP_ST_API_ENDPOINT=<Key Protect Dedicated endpoint>
export IBMCLOUD_API_KEY=<IBM Cloud API key>
export IBMCLOUD_API_KEY_KP_ST=<IBM Cloud API key that can access Key Protect Dedicated instance>

Example for HPCS_API_ENDPOINT and KP_ST_API_ENDPOINT:

export HPCS_API_ENDPOINT=https://fadedbee-0000-0000-0000-1234567890ab.api.us-south.hs-crypto.appdomain.cloud
export KP_ST_API_ENDPOINT=https://fadedbee-0000-0000-0000-1234567890ab.api.us-south.kms.appdomain.cloud

For the status, create, sync, and delete commands, associate the API keys with IAM identities that have the following service access roles:

IBMCLOUD_API_KEY (HPCS)
Requires the Manager service access role. Service access roles are additive, so Manager includes Reader and Writer, covering all four commands. The Manager role is required because the delete command and the create command with the --replace-mi option remove a migration intent from the source key.
IBMCLOUD_API_KEY_KP_ST (Key Protect Dedicated)
Requires the Reader service access role. These commands only read the target key, so Reader is sufficient. The delete command does not access Key Protect Dedicated.

The authz-check command requires different service access roles than the other commands. For more information, see Checking IAM authorization policies.

The IBMCLOUD_API_KEY_KP_ST variable is required only if IBMCLOUD_API_KEY does not have access to the target Key Protect Dedicated instance. It occurs when:

  • The HPCS instance and Key Protect Dedicated instance are in different IBM Cloud accounts
  • Separate IAM identities are used for source and target environments

Optional environment variables

The following optional environment variables can be configured:

IAM_API_ENDPOINT
Takes a custom IAM endpoint if the default https://iam.cloud.ibm.com/identity/token is not used.
DEBUG_MODE
Takes true or false. When set to true, the tool produces detailed logs: the resolved configuration at startup, and per-request HTTP detail for every API call (method, path with query string, response status, and correlation ID). For requests that fail, the response body is also logged. Authorization headers and API keys are never logged.

Validating the input file

Use the status operation to check the status of the migration and verify that your input file is correctly formatted. No migration action is taken with this command.

./<crkm-binary> status <.csv input file>

You might see FAILED - No Migration Intent that is found for the key in the status columns for all key inputs in the output .csv file and in the console log. This is expected when no migration intent is created yet.

Checking IAM authorization policies

Use the authz-check command to verify that IAM authorization policies are in place so that cloud services that currently use HPCS source keys can use the corresponding Key Protect Dedicated target keys after migration.

./<crkm-binary> authz-check <.csv input file>

Required access

The authz-check command only reads information. It does not create, update, or delete any keys or policies, so Writer or Manager access is not required.

The command requires the following access:

HPCS instance
Reader service access role to read the source key's associations.
Key Protect Dedicated instance
Reader service access role to read the target key's metadata.
IAM authorization policies
Account-level permission to view the IAM authorization policies of every account that is referenced in the TargetCRK column. This is not a Key Protect or HPCS service access role. It is account-level access to IBM Cloud Identity and Access Management.

The identity used for the IAM policy lookup is the one associated with IBMCLOUD_API_KEY_KP_ST. If IBMCLOUD_API_KEY_KP_ST is not set, the identity associated with IBMCLOUD_API_KEY is used.

If the identity cannot view the policies in a target account, the tool logs the failure and continues. It reports every association in that account as NO MATCH. Insufficient IAM access appears as false NO MATCH results rather than an authorization error. Verify this access before you rely on the output.

The command works in two phases:

Phase 1: Fetch policies

The tool extracts the distinct account IDs from the TargetCRK column in the CSV, fetches all IAM authorization policies for each account, and writes them to JSON files (policies-{serviceName}-{accountID}.json) for reference.

Phase 2: Match associations

For each row in the CSV, the tool fetches all associations on the source HPCS key (the cloud resources encrypted by that key). The tool checks whether an IAM authorization policy exists that allows each associated service to use the target Key Protect Dedicated key.

The tool also looks up the resource group of each target Key Protect Dedicated instance through the IBM Cloud resource controller API. This lookup enables the tool to match policies that are scoped to a resource group.

For each association, the tool reports either:

MATCH
A valid authorization policy was found. The policy JSON is printed.
NO MATCH
No authorization policy was found. The tool prints a message that explains what is missing and a colored JSON template of a policy that satisfies the match. In the template, required fields are shown in green and optional fields in blue.

A summary is printed at the end with the total number of checked, matched, and unmatched associations.

Authorization check flags

--match-all-authz-policies
By default, the tool stops at the first matching policy per association. Use this flag to log all matching policies.

Matching criteria

An authorization policy matches an association when the following conditions are true:

  1. The policy's subjects (who is authorized) contain a serviceName and accountId that match the associated cloud service.
  2. The policy's resources (what is being accessed) contain a serviceName and accountId that match the target Key Protect Dedicated key's service.
  3. The policy's roles grant appropriate access (Reader, ReaderPlus, Writer, or Manager).
  4. If the associated service requires it (for example, databases-for-*, messages-for-rabbitmq, containers-kubernetes), the policy also includes the AuthorizationDelegator role.

If the policy specifies optional attributes such as resourceGroupId, serviceInstance, keyRing, or resource, those attributes must also match the target key or its Key Protect Dedicated instance.

Creating migration intents

After you create a .csv file with the pairing of HPCS keys to migrate to their respective Key Protect Dedicated keys, use the create operation to create migration intents on the HPCS keys.

./<crkm-binary> create <.csv input file>

This operation performs the following actions:

  • Triggers sync events to services that have associations (cloud resources that use the keys being migrated) so they can start migrating the HPCS key
  • You must monitor event acknowledgment and migration in the activity tracking logs (see Migration that is not completed in time for troubleshooting)

If a migration intent exists on a key and you want to update it with an updated .csv input file, run the CLI tool with the --replace-mi flag.

For any issues that are observed in the creation of migration intents, see troubleshooting section.

Syncing keys

After five minutes from the completion of the previous step, use the sync operation:

./<crkm-binary> sync <.csv input file>

This operation triggers sync events to services that have associations (cloud resources that use the keys being migrated). This step ensures that the migration can occur even for services that use HPCS keys indirectly. For example, ICD uses IBM Cloud Object Storage to store backups, and IBM Cloud Object Storage uses Key Protect.

Wait at least five minutes and check the migration status. The HPCS-adopting services do the migration, which has up to four hours to respond to HPCS. For Event Streams, the migration might take up to one business day after you create a migration intent.

Checking migration status

Use the status operation to check on the status of the migration:

./<crkm-binary> status <.csv input file>

If the number of associations of an HPCS key was not zero at the start of the migration and drops to zero, the migration is complete.

If you observe issues in the status column or the number of HPCS associations (KpStAssociationsCount) is still nonzero, see troubleshooting.

Deleting migration intents

Use the delete operation to remove the migration intent from one or more HPCS keys. For example, use this operation to undo a migration intent that was created with the wrong target key before migration begins. For each key, the tool checks whether a migration intent exists. If one exists, the tool prints the migration intent's properties (ID, source CRK, target CRK, creator, and creation date) and then deletes it. If no migration intent exists, the tool reports that none was found.

Unlike other operations, the delete operation accepts either a .csv input file or a single HPCS key CRN.

To delete the migration intents for every HPCS key in a .csv file, use the following command:

./<crkm-binary> delete <.csv input file>

To delete the migration intent for a single HPCS key by passing its full CRN, use the following command:

./<crkm-binary> delete <HPCS key CRN>

Example:

./<crkm-binary> delete crn:v1:bluemix:public:hs-crypto:eu-fr2:a/00000000000000000000000000000000:deadbeef-0000-0000-0000-1234567890ab:key:deadbeef-0000-0000-0000-1234567890ab

The delete operation acts only on the source HPCS key. Therefore, only the HPCS_API_ENDPOINT and IBMCLOUD_API_KEY environment variables are required. The Key Protect Dedicated variables are not required.

The delete operation removes the migration intent immediately without prompting for confirmation. When you pass a .csv file, the migration intent of every key in the file is deleted.

Troubleshooting

Use the following information to troubleshoot common issues with the migration tool.

Migration not completed in time

Sometimes services might encounter problems, while migrating keys.

Use the sync operation with the .csv file as input to notify services with associations to the key to try the migration again:

./<crkm-binary> sync <.csv input file>

If the number of associations of an HPCS key was not zero at the start of the migration and does not drop to zero, services might be encountering problems migrating keys. Alternatively, there might be stale associations. Create an IBM Support ticket for Key Protect and mention HPCS to Key Protect migration.

Invalid resource CRN format

Error message:

invalid key crn for key : crn must be specified to 10 segments

The CSV status column output also reads: FAILED - crn must be specified to 10 segments

Review the input .csv file and ensure the CRN for the HPCS keys and Key Protect Dedicated keys are formatted correctly.

Key not found

Error message:

2023/03/29 09:23:39 Sleeping for 100ms before processing row 5
2023/03/29 09:23:39 Retrieving Key...
2023/03/29 09:23:39 Get Key failed for keyID d4c4cbee-20ed-4deb-b59d-4674f70713b5 with the following error: kp.Error: correlation_id='643a49af-a3cc-4eba-b23b-368bbc33115d', msg='Not Found: Key could not be retrieved: Please see `reasons` for more details (KEY_NOT_FOUND_ERR)', reasons='[KEY_NOT_FOUND_ERR: key does not exist - FOR_MORE_INFO_REFER: https://cloud.ibm.com/apidocs/key-protect]'
2023/03/29 09:23:39 -----------------------------------------------

The CSV status column output also reads:

FAILED - kp.Error: correlation_id='643a49af-a3cc-4eba-b23b-368bbc33115d', msg='Not Found: Key could not be retrieved: Please see `reasons` for more details (KEY_NOT_FOUND_ERR)', reasons='[KEY_NOT_FOUND_ERR: key does not exist - FOR_MORE_INFO_REFER: https://cloud.ibm.com/apidocs/key-protect]'

The CRN for the key does not exist. Review the input .csv file and ensure that both the HPCS keys to migrate and the Key Protect Dedicated target keys exist.

Instance does not exist or the user is unauthorized

Error message:

2023/03/29 09:26:44 Retrieving Key...
2023/03/29 09:26:44 Get Key failed for keyID d4c4cbee-20ed-4deb-b59d-4674f70713b5 with the following error: kp.Error: correlation_id='4629f1af-1a9b-44ea-9139-3c5c39b9790e', msg='Unauthorized: Either the user does not have access to the specified resource, the resource does not exist, or the region is incorrectly set'
2023/03/29 09:26:44 -----------------------------------------------

The CSV status column output also reads:

FAILED - kp.Error: correlation_id='4629f1af-1a9b-44ea-9139-3c5c39b9790e', msg='Unauthorized: Either the user does not have access to the specified resource, the resource does not exist, or the region is incorrectly set'

The user does not have access to the key, or the instance in the CRN of the key does not exist.

Create command: No associations found

Error message:

2023/03/22 10:39:36 Retrieving Key...
2023/03/22 10:39:36 Key retrieved successfully
2023/03/22 10:39:36 Retrieving associations...
2023/03/22 10:39:36 Associations retrieved successfully
2023/03/22 10:39:36 Retrieving Migration Intent...
2023/03/22 10:39:36 Migration Intent retrieved successfully
2023/03/22 10:39:36 No associations found for the key
2023/03/22 10:39:36 -----------------------------------------------

The CSV status column output also reads: FAILED - No associations were found for the key

The HPCS key has no cloud resources that are associated with it. Therefore, you do not need to create a migration intent on the HPCS key or perform migration.

Create command: Migration intent exists

Error message:

2023/03/22 10:39:35 Retrieving Key...
2023/03/22 10:39:35 Key retrieved successfully
2023/03/22 10:39:35 Retrieving associations...
2023/03/22 10:39:35 Associations retrieved successfully
2023/03/22 10:39:35 Retrieving Migration Intent...
2023/03/22 10:39:35 Migration Intent retrieved successfully
2023/03/22 10:39:35 Migration Intent already exists
2023/03/22 10:39:35 -----------------------------------------------

The CSV status column output also reads: FAILED - Migration Intent exists for the key

The HPCS key already has an existing migration intent. To update the migration intent, complete the following steps:

  1. Update the input .csv file with the new CRN of the Key Protect Dedicated key to migrate to.
  2. Add the --replace-mi flag to the create command.
  3. Run the command again.

Status command: No migration intent found

Error message:

2023/03/22 13:15:56 Retrieving Key...
2023/03/22 13:15:56 Key retrieved successfully
2023/03/22 13:15:56 Retrieving associations...
2023/03/22 13:15:56 Associations retrieved successfully
2023/03/22 13:15:56 Retrieving Migration Intent...
2023/03/22 13:15:56 No Migration Intent found for keyID a2cd689e-9a49-4cd7-ba0d-07d25a0b3a25
2023/03/22 13:15:56 -----------------------------------------------

The CSV status column output also reads: FAILED - No Migration Intent found for the key

A migration intent does not exist for that HPCS key. Retry Creating migration intents, to ensure that the key resource CRN is included in the input file, and check the status of the operation.

Sync command: No associations found

Error message:

2023/03/22 13:13:45 Retrieving Key...
2023/03/22 13:13:45 Key retrieved successfully
2023/03/22 13:13:45 Retrieving associations...
2023/03/22 13:13:45 Associations retrieved successfully
2023/03/22 13:13:45 Retrieving Migration Intent...
2023/03/22 13:13:45 Migration Intent retrieved successfully
2023/03/22 13:13:45 No associations found for keyID d4c3cbee-20ed-4deb-b59d-4674f70713b5
2023/03/22 13:13:45 -----------------------------------------------

The CSV status column output also reads: FAILED - No associations were found for the key

The HPCS key has no cloud resources that are associated with it. Therefore, you do not need to perform a syncing of resources. This also indicates that the migration is complete for the key.

Sync command: No migration intent found

Error message:

2023/03/22 13:15:13 Retrieving Key...
2023/03/22 13:15:13 Key retrieved successfully
2023/03/22 13:15:13 Retrieving associations...
2023/03/22 13:15:13 Associations retrieved successfully
2023/03/22 13:15:13 Retrieving Migration Intent...
2023/03/22 13:15:13 Migration Intent retrieved successfully
2023/03/22 13:15:13 No Migration Intent found for keyID a2cd689e-9a49-4cd7-ba0d-07d25a0b3a25
2023/03/22 13:15:13 -----------------------------------------------

The CSV status column output also reads: FAILED - No Migration Intent found for the key

A migration intent does not exist for that key. Retry Creating migration intents, ensuring the key resource CRN is included in the input file, and check the status of the operation.

Sync command: Request too early

Error message:

2023/03/29 09:17:06 Sleeping for 100ms before processing row 2
2023/03/29 09:17:06 Retrieving Key...
2023/03/29 09:17:06 Key retrieved successfully
2023/03/29 09:17:06 Retrieving associations...
2023/03/29 09:17:06 Associations retrieved successfully
2023/03/29 09:17:06 Retrieving Migration Intent...
2023/03/29 09:17:06 Migration Intent retrieved successfully
2023/03/29 09:17:06 Post Sync Resources for key...
2023/03/29 09:17:06 Post Sync Call Failed: kp.Error: correlation_id='8f088c29-2f61-4dc4-b00f-c99f7edd00fe', msg='Conflict: Could not initiate sync requests to associated resources: Please see `reasons` for more details (REQ_TOO_EARLY_ERR)', reasons='[REQ_TOO_EARLY_ERR: The key was updated recently: Please wait and try again: Sync cannot be performed until at least 2023-03-29T14:17:19Z - FOR_MORE_INFO_REFER: https://cloud.ibm.com/apidocs/key-protect]'
2023/03/29 09:17:06 -----------------------------------------------

The CSV status column output also reads:

FAILED - kp.Error: correlation_id='0cefcea8-7e29-4c4a-9721-00244eb9752c', msg='Conflict: Could not initiate sync requests to associated resources: Please see `reasons` for more details (REQ_TOO_EARLY_ERR)', reasons='[REQ_TOO_EARLY_ERR: The key was updated recently: Please wait and try again: Sync cannot be performed until at least 2023-03-29T14:17:19Z - FOR_MORE_INFO_REFER: https://cloud.ibm.com/apidocs/key-protect]'

The sync operation is being performed too quickly. Wait at least five minutes before trying again.