Backing up and restoring data

Before you begin

• When you create a backup by using this procedure, the backup includes all of the assistants and skills from all of the service instances. Meaning it can include even skills and assistants to which you do not have access.
• The access permissions information of the original service instances is not stored in the backup. Meaning original access rights, which determine who can see a service instance and who cannot, are not preserved.
• You cannot use this procedure to back up the data that is returned by the search skill. Data that is retrieved by the search skill comes from a data collection in a Discovery instance. See the Discovery documentation to find out how to back up its data.
• If you back up and restore or otherwise change the Discovery service that your search skill connects to, then you cannot restore the search skill, but must recreate it. When you set up a search skill, you map sections of the assistant's response to fields in a data collection that is hosted by an instance of Discovery on the same cluster. If the Discovery instance changes, your mapping to it is broken. If your Discovery service does not change, then the search skill can continue to connect to the data collection.
• The tool that restores the data clears the current database before it restores the backup. Therefore, if you might need to revert to the current database, create a backup of it first.
• The target IBM Cloud Pak for Data cluster where you restore the data must have the same number of provisioned Watson Assistant service instances as the environment from which you back up the database. To verify in the IBM Cloud Pak for Data web client, select Services from the main navigation menu, select Instances, and then open the Provisioned instances tab. If more than one user created instances, then ask the other users who created instances to log in and check the number they created. You can then add up the total sum of instances for your deployment. Note that not even an administrative user can see instances that were created by others from the web client user interface.

Backing up data by using the CronJob

A CronJob named $INSTANCE-store-cronjob is created and enabled for you automatically when you deploy the service. A CronJob is a type of Kubernetes controller. A CronJob creates Jobs on a repeating schedule. For more information, see CronJob in the Kubernetes documentation.

The jobs that are created by the store cron job are called $INSTANCE-backup-job-$TIMESTAMP. Each job deletes old logs and runs a backup of the store Postgres database. Postgres provides a tool that is called pg_dump. The dump tool creates a backup by sending the database contents to stdout where you can write it to a file. The backups are created with the pg_dump command and stored in a persistent volume claim (PVC) named $INSTANCE-store-pvc.

You are responsible for moving the backup to a more secure location after its initial creation, preferrably a location that can be accessed outside of the cluster where the backups cannot be deleted easily. Ensure this happens for all environments, especially for Production clusters.

The following table lists the configuration values that control the backup cron job. You can edit these settings by editing the cron job after the service is deployed by using the oc edit cronjob $INSTANCE-store-cronjob command.

Backing up data by using the script

The backupPG.sh script gathers the pod name and credentials for one of your Postgres pods, which is the pod from which the pg_dump command must be run, and then runs the command for you.

To back up data by using the provided script, complete the following steps:

1. Download the backupPG.sh script.

   Go to GitHub, and find the directory for your version to find the file.

2. Log in to the OpenShift project namespace where you installed the product.

3. Run the script:

   ./backupPG.sh --instance ${INSTANCE} > ${BACKUP_DIR}
   

   Replace the following values in the command:

   • ${BACKUP_DIR}: Specify a file where you want to write the downloaded data. Be sure to specify a backup directory in which to store the file. For example, /bu/backup-file-name.dump creates a backup directory named bu.
   • --instance ${INSTANCE}: Select the specific instance of Watson Assistant to be backed up.

If you prefer to back up data by using the Postgres tool directly, you can complete the procedure to back up data manually.

Backing up data manually

Complete the steps in this procedure to back up your data by using the Postgres tool directly.

To back up your data, complete these steps:

1. Fetch a running Postgres pod:

   oc get pods -l app=${INSTANCE}-postgres -o jsonpath="{.items[0].metadata.name}"
   

   Replace ${INSTANCE} with the instance of the Watson Assistant deployment that you want to back up.

2. Fetch the store VCAP secret name:

   oc get secrets -l component=store,app.kubernetes.io/instance=${INSTANCE} -o=custom-columns=NAME:.metadata.name | grep store-vcap
   

3. Fetch the Postgres connection values. You will pass these values to the command that you run in the next step. You must have jq installed.

   • To get the database:

     oc get secret $VCAP_SECRET_NAME -o jsonpath="{.data.vcap_services}" | base64 --decode | jq --raw-output '.["user-provided"][]|.credentials|.database'
     

   • To get the hostname:

     oc get secret $VCAP_SECRET_NAME -o jsonpath="{.data.vcap_services}" | base64 --decode | jq --raw-output '.["user-provided"][]|.credentials|.host'
     

   • To get the username:

     oc get secret $VCAP_SECRET_NAME -o jsonpath="{.data.vcap_services}" | base64 --decode | jq --raw-output '.["user-provided"][]|.credentials|.username'
     

   • To get the password:

     oc get secret $VCAP_SECRET_NAME -o jsonpath="{.data.vcap_services}" | base64 --decode | jq --raw-output '.["user-provided"][]|.credentials|.password'
     

4. Run the following command:

   oc exec $KEEPER_POD -- bash -c "export PGPASSWORD='$PASSWORD' && pg_dump -Fc -h $HOSTNAME -d $DATABASE -U $USERNAME" > ${BACKUP_DIR}
   

   The following lists describes the arguments. You retrieved the values for some of these parameters in the previous step:

   • $KEEPER_POD: Any Postgres pod in your Watson Assistant instance.
   • ${BACKUP_DIR}: Specify a file where you want to write the downloaded data. Be sure to specify a backup directory in which to store the file. For example, /bu/backup-file-name.dump creates a backup directory named bu.
   • $DATABASE: The store database name that was retrieved from the Store VCAP secret in step 3.
   • $HOSTNAME: The hostname that was retrieved from the Store VCAP secret in step 3.
   • $USERNAME: The username that was retrieved from the Store VCAP secret in step 3.
   • $PASSWORD: The password that was retrieved from the Store VCAP secret in step 3.

   To see more information about the pg_dump command, you can run this command:

   oc exec -it ${KEEPER_POD} -- pg_dump --help
   

Restoring data

IBM created a restore tool called pgmig. The tool restores your database backup by adding it to a database you choose. It also upgrades the schema to the one that is associated with the version of the product where you restore the data. Before the tool adds the backed-up data, it removes the data for all instances in the current service deployment, so any spares are also removed.

1. Install the target IBM Cloud Pak for Data cluster to which you want to restore the data.

   From the web client for the target cluster, create one service instance of Watson Assistant for each service instance that was backed up on the old cluster. The target IBM Cloud Pak for Data cluster must have the same number of instances as there were in the environment where you backed up the database.

2. Back up the current database before you replace it with the backed-up database.

   The tool clears the current database before it restores the backup. So, if you might need to revert to the current database, be sure to create a backup of it first.

3. Go to the backup directory that you specified in the ${BACKUP_DIR} parameter in the previous procedure.

4. Run the following command to download the pgmig tool from the GitHub Watson Developer Cloud Community repository.

   In the first command, update <WA_VERSION> to the version that you want to restore. For example, update <WA_VERSION> to 4.6.0 if you want to restore Watson Assistant 4.6.0.

   wget https://github.com/watson-developer-cloud/community/raw/master/watson-assistant/data/<WA_VERSION>/pgmig
   chmod 755 pgmig
   

5. Create the following two configuration files and store them in the same backup directory:

   • resourceController.yaml: The Resource Controller file keeps a list of all provisioned Watson Assistant instances. See Creating the resourceController.yaml file.

   • postgres.yaml: The Postgres file lists details for the target Postgres pods. See Creating the postgres.yaml file.

6. Get the secret:

   oc get secret ${INSTANCE}-postgres-ca -o jsonpath='{.data.ca\.crt}' | base64 -d | tee ${BACKUP_DIR}/ca.crt | openssl x509 -noout -text
   

   • Replace ${INSTANCE} with the name of the Watson Assistant instance that you want to back up.
   • Replace ${BACKUP_DIR} with the directory where the postgres.yaml and resourceController.yaml files are located.

7. Copy the files that you downloaded and created in the previous steps to any existing directory on a Postgres pod.

   1. Run the following command to find Postgres pods:

      oc get pods | grep ${INSTANCE}-postgres
      

   2. The files that you must copy are pgmig, postgres.yaml, resourceController.yaml, ca.crt (the secret file generated in step 6), and the file that you created for your downloaded data. Run the following commands to copy the files.

      If you are restoring data to a stand-alone IBM Cloud Pak for Data cluster, then replace all references to oc with kubectl in these sample commands.

      oc exec -it ${POSTGRES_POD} -- mkdir /controller/tmp
      oc exec -it ${POSTGRES_POD} -- mkdir /controller/tmp/bu
      oc rsync ${BACKUP_DIR}/ ${POSTGRES_POD}:/controller/tmp/bu/
      

   • Replace ${POSTGRES_POD} with the name of one of the Postgres pods from the previous step.

8. Stop the store deployment by scaling the store deployment down to 0 replicas:

   oc scale deploy ibm-watson-assistant-operator -n ${OPERATOR_NS} --replicas=0
   oc get deployments -l component=store
   

   Make a note of how many replicas there are in the store deployment:

   oc scale deployment ${STORE_DEPLOYMENT} --replicas=0
   

9. Initiate the execution of a remote command in the Postgres pod:

   oc exec -it ${POSTGRES_POD} /bin/bash
   

10. Run the pgmig tool:

    cd /controller/tmp/bu
    export PG_CA_FILE=/controller/tmp/bu/ca.crt
    ./pgmig --resourceController resourceController.yaml --target postgres.yaml --source <backup-file-name.dump>
    

    • Replace <backup-file-name.dump> with the name of the file that you created for your downloaded data.

    For more command options, see Postgres migration tool details.

    As the script runs, you are prompted for information that includes the instance on the target cluster to which to add the backed-up data. The data on the instance you specify will be removed and replaced. If there are multiple instances in the backup, you are prompted multiple times to specify the target instance information.

11. Scale the store deployment back up:

    oc scale deployment ${STORE_DEPLOYMENT} --replicas=${ORIGINAL_NUMBER_OF_REPLICAS}
    oc scale deploy ibm-watson-assistant-operator -n ${OPERATOR_NS} --replicas=1
    

    You might need to wait a few minutes before the skills you restored are visible from the web interface.

12. After restoring the data, you have to train the backend model. Ensure you reopen only one assistant or dialog skill at a time. Each time you open a dialog skill after its training data has been changed, training is initiated automatically. Give the skill time to retrain on the restored data. It usually takes less than 10 minutes to get trained. Remember that the process of training a machine learning model requires at least one node to have 4 CPUs that can be dedicated to training. Therefore, open restored assistants and skills during low traffic periods and open them one at a time. If the assistant or dialog skill does not respond, then modify the workspace (for example, add an intent and then remove it). Check and confirm.

accessTokens: 
  - value
  - value2
host: localhost
port: 5000

host: localhost
port: 5432
database: store
username: user
su_username: admin
su_password: password

instance-mappings:
  00000000-0000-0000-0000-001570184978: 00000000-0000-0000-0000-001570194490