IBM Cloud Docs
Elasticsearch data migration using snapshot and restore

Elasticsearch data migration using snapshot and restore

Migrate data between two instances of Elasticsearch with the help of Object Storage. Using the snapshot and restore method, take snapshots from your source instance, store them in Object Storage, and then restore those snapshots from Object Storage into your target instance.

This tutorial uses snapshot and restore, IBM Cloud Object Storage and two instances of Databases for Elasticsearch. However, this process is applicable to any S3-compatible object storage solution and any deployment of Elasticsearch.

We've simplified the process by using Terraform and shell scripts. Simply follow the procedure outlined on this page, plugging in the necessary variables as you go.

If you are using the snapshot and restore process to upgrade from Elasticsearch 7.9/7.10 to version 7.17, you must change your plan from Standard to Enterprise. Change your plan by clicking Restore on the Provisioning page on the backup you want to restore, then select 7.17 from the dropdown menu.

Getting Productive

Before you migrate your data, install Terraform to codify and deploy necessary infrastructure. You also need an IBM Cloud account.

Step 1: Obtain an IBM Cloud API key and clone the GitHub repository

Follow these steps to create an IBM Cloud API key that enables Terraform to provision infrastructure into your account. You can create up to 20 API keys.

For security reasons, the API key is only available to be copied or downloaded at the time of creation. If the API key is lost, you must create a new API key.

Next, clone the Elasticsearch Snapshot/Restore GitHub Repository to your local machine.

git clone https://github.com/IBM/elasticsearch-cos-snapshot-restore.git

After cloning this folder, navigate to the newly created project folder on your local machine.

Step 2: Install and run the Terraform script

The terraform folder contains files that create the necessary infrastructure to create and restore your snapshots:

cos.tf

cos.tf creates a Cloud Object Storage instance and a bucket. Update your resources for the restoreCOSInstance, restoreBucket, resourceKey. This script then outputs the bucket_credentials and bucket_name.

elastic.tf

elastic.tf creates a source, target, and necessary configurations. Input the variables for the resource "ibm_database" "esSource" and resource "ibm_database" "esTarget". This script then outputs the necessary configuration variables.

main.tf

main.tf contains the main set of configuration for your module. For the ibmcloud_api_key variable, create or retrieve an IBM Cloud® API key. Then, specify the Resource Group and ibm_resource_group variable, which outputs the resource_group_name.

variables.tf

variables.tf contains the variable definitions ibmcloud_api_key, region, and elastic_password. Update these variables with your API key, preferred region, and your Elasticsearch password.

After setting up your resources, configurations, and variables, go ahead and run your Terraform script. Navigate to your terraform folder and install the infrastructure with the following command:

terraform init
terraform apply --auto-approve

The Terraform script outputs configuration data that is needed to run the application, so copy it into the root folder:

terraform output -json >../config.json

Step 3: Run the shell snapshot script

Run the migrate.sh file in the main project folder. This shell script uses the information that is provided by the config.json file to perform the necessary migration steps.

  • Create different snapshot names by adding a timestamp.
  • Get database and S3/COS parameters.
  • Mount S3/COS bucket on source deployment.
  • Mount S3/COS bucket on target deployment.
  • Close all indexes on the target so the restore can be run without touching the icd-auth index, which is protected by Cloud Databases.

Run migrate.sh as many times as necessary to fully back up your data.

Snapshots are incremental, so the first snapshot takes longer than the rest.

Once your COS bucket has all the necessary snapshots, stop any writes to the source. Then, run migrate.sh one more time to take a final snapshot and restore it to the target. All of your data is now in the target. Point your applications to the target database and your upgrade is complete.

Retrieve and update user passwords

If you're restoring to Elasticsearch 7.17 as an update from an earlier version, existing user passwords will be invalidated and must be reset after the restore. For more information, see Retrieve and update user passwords.