Configuring IBM Cloudant for cross-region disaster recovery

Before you begin an active-active deployment

For an active-active deployment, a strategy for managing conflicts must be in place, so be sure to understand how replication and conflicts work before you consider this architecture.

Go to the IBM Cloud Support portal if you need help with how to model data to handle conflicts effectively.

Overview

In the following material, a bidirectional replication is created. This configuration allows two databases to work in an active-active topology.

The configuration assumes that you have two accounts in different regions:

• myaccount-dc1.cloudant.com
• myaccount-dc2.cloudant.com

After these accounts are created, follow these steps:

1. Create a pair of peer databases within the accounts.
2. Set up API keys to use for the replications between these databases.
3. Grant appropriate permissions.
4. Set up replications.
5. Test replications are working as expected.
6. Configure application and infrastructure for either active-active or active-passive use of the databases.

Step 1. Create your databases

Create the databases that you want to replicate between within each account.

In this example, a database that is called mydb is created.

The names that are used for the databases in this example aren't important, but using the same name is clearer.

curl "https://myaccount-dc1.cloudant.com/mydb" -XPUT -u myaccount-dc1
curl "https://myaccount-dc2.cloudant.com/mydb" -XPUT -u myaccount-dc2

Step 2. Create an API key for your replications

It's a good idea to use an API key for continuous replications. The advantage is that if your primary account details change, for example after a password reset, your replications can continue unchanged.

API keys aren't tied to a single account. This means that a single API key can be created, then granted suitable database permissions for both accounts.

For example, the following command requests an API key for the account myaccount-dc1:

curl -XPOST "https://myaccount-dc1.cloudant.com/_api/v2/api_keys" -u myaccount-dc1

A successful response is similar to the following abbreviated example:

{
  "password": "YPN...Tfi",
  "ok": true,
  "key": "ble...igl"
}

Take careful note of the password. It isn't possible to retrieve the password later.

Step 3. Grant access permission

Give the API Key permission to read and to write on both databases.

If you also need to replicate indexes, assign admin permissions.

Use the IBM Cloudant Dashboard, or see the authorization information for details of how to grant permissions programmatically.

Step 4. Set up replications

Replications in IBM Cloudant are always uni-directional: from one database to another database. To replicate both ways between two databases, two replications are required, one for each direction.

A replication is created in each account that uses the API Key that is created earlier.

First, create a replication from database myaccount-dc1.cloudant.com/mydb to database myaccount-dc2.cloudant.com/mydb.

curl -XPOST "https://myaccount-dc1.cloudant.com/_replicator"
	-u myaccount-dc1
	-H "Content-Type: application/json"
	-d '{ "_id": "mydb-myaccount-dc1-to-myaccount-dc2",
	"source": "https://ble...igl:YPN...Tfi@myaccount-dc1.cloudant.com/mydb",
	"target": "https://ble...igl:YPN...Tfi@myaccount-dc2.cloudant.com/mydb",
	"continuous": true
}'

Next, create a replication from database myaccount-dc2.cloudant.com/mydb to database myaccount-dc1.cloudant.com/mydb.

curl -XPOST "https://myaccount-dc2.cloudant.com/_replicator"
	-u myaccount-dc2
	-H "Content-Type: application/json"
	-d '{ "_id": "mydb-myaccount-dc2-to-myaccount-dc1",
	"source": "https://ble...igl:YPN...Tfi@myaccount-dc2.cloudant.com/mydb",
	"target": "https://ble...igl:YPN...Tfi@myaccount-dc1.cloudant.com/mydb",
	"continuous": true
}'

If this step fails because the _replicator database doesn't exist, create it.

Step 5. Test your replication

Test the replication processes by creating, modifying, and deleting documents in either database.

After each change in the database, check that you can also see that change in the other database.

Step 6. Configure your application

The databases are set up to remain synchronized with each other.

The next decision is whether to use the databases in an active-active or active-passive manner.

Step 7. Next steps

• Consider monitoring the replications between the databases. Use the data to determine whether your configuration might be optimized further.
• Consider how your design documents and indexes are deployed and updated. You might find it more efficient to automate these tasks.

Failing over between IBM Cloudant regions

Typically, the process of managing a failover between regions or datacenters is handled higher up within your application stack, for example by configuring application server failover changes, or by balancing the load.

IBM Cloudant doesn't provide a facility for you to manage explicitly any failover or reroute requests between regions. This constraint is partly for technical reasons, and partly because the conditions under when it might happen tend to be application-specific. For example, you might want to force a failover in response to a custom performance metric.

However, if you decide that you need the ability to manage failover, consider the following possible options:

• Put your own HTTP proxy in front of IBM Cloudant. Configure your application to talk to the proxy rather than the IBM Cloudant instance. This configuration means that the task of changing the IBM Cloudant instances that are used by applications can be handled through a modification to the proxy configuration rather than a modification to the application settings. Many proxies can balance the load, based on user-defined health checks.
• Use a global load balancer such as IBM Cloud® Internet Services to route to IBM Cloudant. This option requires a CNAME definition that routes to different IBM Cloudant accounts, based on a health check or latency rule.

Recovering from failover

If a single IBM Cloudant instance is unreachable, avoid redirecting traffic back to it as soon as it becomes reachable again. The reason is that some time is required for intensive tasks such as synchronizing the database state from any peers, and ensuring that indexes are up to date.

It's helpful to have a mechanism for monitoring these tasks to help decide when a database is in a suitable state to service your production traffic.

As a guide, a typical list of checks to apply include:

• Replications
• Indexes

If you implement rerouting for requests or failover based on a health test, you might want to incorporate corresponding checks to avoid premature rerouting back to a service instance that is still recovering.