Understanding high availability and disaster recovery for Secrets Manager

IBM Cloud® Secrets Manager is a highly available, regional service that runs in the Dallas (us-south), Frankfurt (eu-de), London (eu-gb), Madrid (eu-es), Osaka (jp-osa), Sydney (au-syd), Sao Paulo (br-sao), Tokyo (jp-tok), Toronto (ca-tor), and Washington DC (us-east) regions.

In each supported region, the service exists in multiple availability zones with no single point of failure. All the data that is associated with your instance of the service, including your secrets, is backed up daily and stored in cross-region Cloud Object Storage buckets managed by Secrets Manager.

However, because Secrets Manager is a regional service, cross-regional failover, and cross-regional disaster recovery are not automatic. If all the availability zones in a region fail, Secrets Manager becomes unavailable in that location. When the region is available again, data and traffic is restored without any need for action from you.

See How do I ensure zero downtime? to learn more about the high availability and disaster recovery standards in IBM Cloud. You can also find information about Service Level Agreements.

Application-level high availability

Applications that communicate over networks and cloud services are subject to transient connection failures. You want to design your applications to retry connections when errors are caused by a temporary loss in connectivity to your deployment or to IBM Cloud.

Because Secrets Manager is a managed service, regular updates and database maintenance may occur as part of normal operations. Such maintenance can occasionally cause short interruption intervals.

Your applications must be designed to handle temporary interruptions, implement error handling for failed requests, and implement retry logic to recover from a temporary interruption.

Learn more around topics such as: implementing code best practices, organizing secrets, and rotating secrets.

Manually backing up secrets

To manually back up your secrets across regions, you must first have an instance of Secrets Manager in another region. Then, use the following steps to ensure cross-region availability.

List and download secrets from your instance by using the Secrets Manager API or CLI.

If you have existing configurations on secrets engines in your instance, you can also retrieve the information programmatically so that it can be re-created in a new instance. For more information, see the Get the configuration of a secret type API.
Add your downloaded secrets to the newly created instance.

Automatically backing up secrets

Creating an automatic backup of your secrets is possible by automating the manual flow, which can be done in various ways. Check out some of the following examples to see whether one of them might work for you.

Create a script that periodically downloads all of your secrets and then imports them into your backup instance.
Create a destination and subscription in Event Notifications that points to an IBM Cloud Code Engine action. Configure the action to listen for lifecycle events such as secret_created and secret_rotated. Then, when the action receives the event, the action downloads the secret from one instance and adds it to the backup instance.

Currently, Secrets Manager supports notifications for certificates only. To learn about the various available lifecycle event types, see Enabling event notifications.

Recovering data in BYOK-protected instances

If your Secrets Manager instance was provisioned by using the root key from either Key Protect or IBM Cloud® Hyper Protect Crypto Services (HPCS) and you accidentally deleted the root key, open a case and include the following information:

Your Secrets Manager instance's CRN
Your backup Key Protect or HPCS instance's CRN
The new root key ID
The original Key Protect or HPCS instance's CRN and key ID, if available