Backing up and restoring data for IBM Cloud Pak for Data

Before you begin

• When you create a backup by using this procedure, the backup includes all of the assistants and skills from all of the service instances. It can include skills and assistants to which you do not have access.
• The access permissions information of the original service instances is not stored in the backup. Meaning original access rights, which determine who can see a service instance and who cannot, are not preserved.
• You cannot use this procedure to back up the data that is returned by the search integration. Data that is retrieved by the search integration comes from a data collection in a Discovery instance. See the Discovery documentation to find out how to back up its data.
• If you back up and restore or otherwise change the Discovery service that your search integration connects to, then you cannot restore the search integration, but must re-create it. When you set up a search integration, you map sections of the assistant's response to fields in a data collection that is hosted by an instance of Discovery on the same cluster. If the Discovery instance changes, your mapping to it is broken. If your Discovery service does not change, then the search integration can continue to connect to the data collection.
• The tool that restores the data clears the current database before it restores the backup. Therefore, if you might need to revert to the current database, create a backup of it first.
• The target IBM Cloud Pak for Data cluster where you restore the data must have the same number of provisioned service instances as the environment from which you back up the database. To verify in the IBM Cloud Pak for Data web client, select Services from the main navigation menu, select Instances, and then open the Provisioned instances tab. If more than one user created instances, then ask the other users who created instances to log in and check the number that they created. You can then add up the total sum of instances for your deployment. Not even an administrative user can see instances that were created by others from the web client user interface.

Backing up data by using the CronJob

A CronJob named $INSTANCE-store-cronjob is created and enabled for you automatically when you deploy the service. A CronJob is a type of Kubernetes controller. A CronJob creates Jobs on a repeating schedule. For more information, see CronJob in the Kubernetes documentation.

The store CronJob creates the $INSTANCE-backup-job-$TIMESTAMP jobs. Each $INSTANCE-backup-job-$TIMESTAMP job deletes old logs and runs a backup of the store PostgreSQL database. PostgreSQL provides a pg_dump tool that creates a backup. To create a backup, the pg_dump tool sends the database contents to stdout, which you can then write to a file. The pg_dump tool creates the backups with the pg_dump command and stores them in a persistent volume claim (PVC) named $INSTANCE-store-db-backup-pvc.

You are responsible for moving the backup to a more secure location after its initial creation, preferably a location that can be accessed outside of the cluster where the backups cannot be deleted easily. Ensure this happens for all environments, especially for Production clusters.

The following table lists the configuration values that control the backup cron job. You can edit these settings by editing the cron job after the service is deployed by using the oc edit cronjob $INSTANCE-store-cronjob command.

Backing up data by using the script

You cannot backup data by using script in watsonx Assistant for IBM Cloud Pak® for Data 4.6.3 or later.{ .note}

The backupPG.sh script gathers the pod name and credentials for one of your PostgreSQL pods. Then, the backupPG.sh script uses the PostgreSQL pod to run the pg_dump command.

To back up data by using the provided script, complete the following steps:

1. Download the backupPG.sh script.

   Go to GitHub, and find the directory for your version to find the file.

   If the backupPG.sh script doesn't exist in the directory of your version, backup your data by using KubernetesCronJob or pg_dump tool.

2. Log in to the Red Hat OpenShift project namespace where you installed the product.

3. Run the script:

   ./backupPG.sh --instance ${INSTANCE} > ${BACKUP_DIR}
   

   Replace the following values in the command:

   • ${BACKUP_DIR}: Specify a file where you want to write the downloaded data. Be sure to specify a backup directory in which to store the file. For example, /bu/backup-file-name.dump creates a backup directory named bu.
   • --instance ${INSTANCE}: Select the specific instance to be backed up.

If you prefer to back up data by using the PostgreSQL tool directly, you can complete the procedure to back up data manually.

Backing up data manually

Complete the steps in this procedure to back up your data by using the PostgreSQL tool directly.

To back up your data, complete these steps:

1. Fetch a running PostgreSQL pod:

   Only for version 5.1.0 or greater, use:

   oc get pods -l app=${INSTANCE}-postgres-16 -o jsonpath="{.items[0].metadata.name}"
   

   For versions below 5.1.0, use:

   oc get pods -l app=${INSTANCE}-postgres -o jsonpath="{.items[0].metadata.name}"
   

   Replace ${INSTANCE} with the instance of the deployment that you want to back up.

2. Perform the following two steps only if you have version 5.0.0 or 4.8.5 and before:

   a. Fetch the store VCAP secret name:

   oc get secrets -l component=store,app.kubernetes.io/instance=${INSTANCE} -o=custom-columns=NAME:.metadata.name | grep store-vcap
   

   b. Fetch the PostgreSQL connection values. You will pass these values to the command that you run in the next step. You must have jq installed.

   • To get the database:

     oc get secret $VCAP_SECRET_NAME -o jsonpath="{.data.vcap_services}" | base64 --decode | jq --raw-output '.["user-provided"][]|.credentials|.database'
     

   • To get the hostname:

     oc get secret $VCAP_SECRET_NAME -o jsonpath="{.data.vcap_services}" | base64 --decode | jq --raw-output '.["user-provided"][]|.credentials|.host'
     

   • To get the username:

     oc get secret $VCAP_SECRET_NAME -o jsonpath="{.data.vcap_services}" | base64 --decode | jq --raw-output '.["user-provided"][]|.credentials|.username'
     

   • To get the password:

     oc get secret $VCAP_SECRET_NAME -o jsonpath="{.data.vcap_services}" | base64 --decode | jq --raw-output '.["user-provided"][]|.credentials|.password'
     

3. Perform the following two steps only if you have version 4.8.6 or 5.0.1 and later:

   a. Fetch the store connection secret name:

   oc get secrets -l component=store-subsystem,app.kubernetes.io/instance=${INSTANCE} -o=custom-columns=NAME:.metadata.name | grep store-datastore-connection
   

   b. Fetch the PostgreSQL connection values. You will pass these values to the command that you run in the next step. You must have jq installed.

   • To get the database:

     oc get secret $VCAP_SECRET_NAME -o jsonpath="{.data.store_vcap_services}" | base64 --decode | jq --raw-output '.["user-provided"][]|.credentials|.database'
     

   • To get the hostname:

     oc get secret $VCAP_SECRET_NAME -o jsonpath="{.data.store_vcap_services}" | base64 --decode | jq --raw-output '.["user-provided"][]|.credentials|.host'
     

   • To get the username:

     oc get secret $VCAP_SECRET_NAME -o jsonpath="{.data.store_vcap_services}" | base64 --decode | jq --raw-output '.["user-provided"][]|.credentials|.username'
     

   • To get the password:

     oc get secret $VCAP_SECRET_NAME -o jsonpath="{.data.store_vcap_services}" | base64 --decode | jq --raw-output '.["user-provided"][]|.credentials|.password'
     

4. Run the following command:

   oc exec $KEEPER_POD -- bash -c "export PGPASSWORD='$PASSWORD' && pg_dump -Fc -h $HOSTNAME -d $DATABASE -U $USERNAME" > ${BACKUP_DIR}
   

   The following lists describe the arguments. You retrieved the values for some of these parameters in the previous step:

   Only for version 5.1.0 or greater:

   Use $KEEPER_POD: Any PostgreSQL 16 pod in your instance.

   For versions below 5.1.0:

   Use $KEEPER_POD: Any PostgreSQL pod in your instance.

   For all versions:

   • ${BACKUP_DIR}: Specify a file where you want to write the downloaded data. Be sure to specify a backup directory in which to store the file. For example, /bu/backup-file-name.dump creates a backup directory named bu.
   • $DATABASE: The store database name that was retrieved from the Store VCAP secret in step 3.
   • $HOSTNAME: The hostname that was retrieved from the Store VCAP secret in step 3.
   • $USERNAME: The username that was retrieved from the Store VCAP secret in step 3.
   • $PASSWORD: The password that was retrieved from the Store VCAP secret in step 3.

   To see more information about the pg_dump command, you can run this command:

   oc exec -it ${KEEPER_POD} -- pg_dump --help
   

5. Take a backup of the secret that contains the encryption key. Ignore this step if the below mentioned secret is not available in that release.

   oc get secret -l service=conversation,app=$INSTANCE-auth-encryption
   oc get secret $INSTANCE-auth-encryption -o yaml > auth-encryption-secret.yaml
   

Restoring data

IBM created a restore tool called pgmig. The tool restores your database backup by adding it to a database that you choose. It also upgrades the schema to the one that is associated with the version of the product where you restore the data. Before the tool adds the backed-up data, it removes the data for all instances in the current service deployment, so any spares are also removed.

Prerequisite:

Setup auth-encryption-secret that you backed up earlier.

oc apply -f auth-encryption-secret.yaml 
oc get secret -l service=conversation,app=$INSTANCE-auth-encryption

1. Install the target IBM Cloud Pak for Data cluster to which you want to restore the data.

   From the web client for the target cluster, create one service instance of for each service instance that was backed up on the old cluster. The target IBM Cloud Pak for Data cluster must have the same number of instances as there were in the environment where you backed up the database.

2. Back up the current database before you replace it with the backed-up database.

   The tool clears the current database before it restores the backup. So, if you might need to revert to the current database, be sure to create a backup of it first.

3. Go to the backup directory that you specified in the ${BACKUP_DIR} parameter in the previous procedure.

4. Run the following command to download the pgmig tool from the GitHub Watson Developer Cloud Community repository.

   In the first command, update <WA_VERSION> to the version that you want to restore. For example, update <WA_VERSION> to 4.6.0 if you want to restore 4.6.0.

   wget https://github.com/watson-developer-cloud/community/raw/master/watson-assistant/data/<WA_VERSION>/pgmig
   chmod 755 pgmig
   

5. Create the following two configuration files and store them in the same backup directory:

   • resourceController.yaml: The Resource Controller file keeps a list of all provisioned instances. See Creating the resourceController.yaml file.

   • postgres.yaml: The PostgreSQL file lists details for the target PostgreSQL pods. See Creating the postgres.yaml file.

6. Get the secret:

   Only for version 5.1.0 or greater, use:

   oc get secret ${INSTANCE}-postgres-16-ca -o jsonpath='{.data.ca\.crt}' | base64 -d | tee ${BACKUP_DIR}/ca.crt | openssl x509 -noout -text
   

   For version below 5.1.0, use:

   oc get secret ${INSTANCE}-postgres-ca -o jsonpath='{.data.ca\.crt}' | base64 -d | tee ${BACKUP_DIR}/ca.crt | openssl x509 -noout -text
   

   • Replace ${INSTANCE} with the name of the instance that you want to back up.
   • Replace ${BACKUP_DIR} with the directory where the postgres.yaml and resourceController.yaml files are located.

7. Copy the files that you downloaded and created in the previous steps to any existing directory on a PostgreSQL pod.

   a. Only for version 5.1.0 or greater:

   Run the following command to find PostgreSQL pods:

     oc get pods | grep ${INSTANCE}-postgres-16
   

   b. For versions below 5.1.0:

   Run the following command to find PostgreSQL pods:

     oc get pods | grep ${INSTANCE}-postgres
   

   c. The files that you must copy are pgmig, postgres.yaml, resourceController.yaml, ca.crt (the secret file that is generated in step 6), and the file that you created for your downloaded data. Run the following commands to copy the files.

   If you are restoring data to a stand-alone IBM Cloud Pak for Data cluster, then replace all references to oc with kubectl in these sample commands.

     oc exec -it ${POSTGRES_POD} -- mkdir /controller/tmp
     oc exec -it ${POSTGRES_POD} -- mkdir /controller/tmp/bu
     oc rsync ${BACKUP_DIR}/ ${POSTGRES_POD}:/controller/tmp/bu/
   

   • Replace ${POSTGRES_POD} with the name of one of the PostgreSQL pods from the previous step.

8. Stop the store deployment by scaling the store deployment down to 0 replicas:

   oc scale deploy ibm-watson-assistant-operator -n ${OPERATOR_NS} --replicas=0
   oc get deployments -l component=store
   

   Make a note of how many replicas there are in the store deployment:

   oc scale deployment ${STORE_DEPLOYMENT} --replicas=0
   

9. Initiate the execution of a remote command in the PostgreSQL pod:

   oc exec -it ${POSTGRES_POD} /bin/bash
   

10. Run the pgmig tool:

    Only for version 5.1.0 or greater:

    cd /controller/tmp/bu
    export PG_CA_FILE=/controller/tmp/bu/ca.crt
    ./pgmig --resourceController resourceController.yaml --target postgres.yaml --source <backup-file-name.dump>
    export ENABLE_ICP=true
    

11. Run the pgmig tool:

    For versions below 5.1.0:

    cd /controller/tmp/bu
    export PG_CA_FILE=/controller/tmp/bu/ca.crt
    ./pgmig --resourceController resourceController.yaml --target postgres.yaml --source <backup-file-name.dump>
    

    • Replace <backup-file-name.dump> with the name of the file that you created for your downloaded data.

    For more command options, see PostgreSQL migration tool details.

    As the script runs, you are prompted for information that includes the instance on the target cluster to which to add the backed-up data. The data on the instance you specify is removed and replaced. If there are multiple instances in the backup, you are prompted multiple times to specify the target instance information.

12. Scale the store deployment back up:

    oc scale deployment ${STORE_DEPLOYMENT} --replicas=${ORIGINAL_NUMBER_OF_REPLICAS}
    oc scale deploy ibm-watson-assistant-operator -n ${OPERATOR_NS} --replicas=1
    

    You might need to wait a few minutes before the data your restored is visible from the web interface.

13. After you restore the data, you must train the backend model. For more information about retraining your backend model, see Retraining your backend model.

accessTokens: 
  - value
  - value2
host: localhost
port: 5000

host: localhost
port: 5432
database: store
username: user
su_username: admin
su_password: password

instance-mappings:
  00000000-0000-0000-0000-001570184978: 00000000-0000-0000-0000-001570194490

Retraining your backend model

Per the number of models in your assistant, you can use one of the following options to retrain your backend model:

• Retrain your backend model manually
• Auto-retrain your backend model

  #!/bin/bash

  calculate_duration() {
    local input_variable="$1"
    DURATION=$(("$NUM_OF_WORKSPACES_TO_TRAIN"*60 / (input_variable * 2) + "$NUM_OF_WORKSPACES_TO_TRAIN" * 2))
  }

  NUM_OF_WORKSPACES_TO_TRAIN=$(oc exec wa-etcd-0 -n cpd -- bash -c '
  password="$( cat /var/run/credentials/pass.key )"
  etcdctl_user="root:$password"
  export ETCDCTL_USER="$etcdctl_user"

  ETCDCTL_API=3 etcdctl --cert=/etc/etcdtls/operator/etcd-tls/etcd-client.crt --key=/etc/etcdtls/operator/etcd-tls/etcd-client.key --cacert=/etc/etcdtls/operator/etcd-tls/etcd-client-ca.crt --endpoints=https://$(hostname).${CLUSTER_NAME}.cpd.svc.cluster.local:2379 get  --prefix  /bluegoat/voyager-nlu/voyager-nlu-slot-wa/workspaces/ --keys-only | sed '/^$/d' | wc -l')

  echo "Number of workspaces to train $NUM_OF_WORKSPACES_TO_TRAIN"

  calculate_duration 5
  DURATION_5=$DURATION

  calculate_duration 10
  DURATION_10=$DURATION

  calculate_duration 15
  DURATION_15=$DURATION
  echo "Approximate duration of the auto retrain all job if you have 5 Training pods: $DURATION_5 seconds"
  echo "Approximate duration of the auto retrain all job if you have 10 Training pods: $DURATION_10 seconds"
  echo "Approximate duration of the auto retrain all job if you have 15 Training pods: $DURATION_15 seconds"

  oc logs $(oc get pod -l component=store-admin --no-headers |awk '{print $1}') | grep "\[RETRAIN-ALL-SUMMARY\] Affected workspaces found"