IBM Cloud storage utilities

Updating the IBM Cloud Block Storage Attacher plug-in

You can upgrade the existing IBM Cloud Block Storage Attacher plug-in to the latest version.

1. Update the Helm repo to retrieve the latest version of all helm charts in this repo.

   helm repo update
   

2. Optional: Download the latest Helm chart to your local machine. Then, extract the package and review the release.md file to find the latest release information.

   helm pull iks-charts/ibmcloud-block-storage-plugin
   

3. Find the name of the Helm chart for the IBM Cloud Block Storage Attacher plug-in.

   helm ls -A
   

   Example output

   <helm_chart_name>    1           Wed Aug  1 14:55:15 2022    DEPLOYED    ibm-block-storage-attacher-1.0.0    default
   

4. Upgrade the IBM Cloud Block Storage Attacher to latest.

   helm upgrade --force --recreate-pods <helm_chart_name> ibm-block-storage-attacher
   

Removing the IBM Cloud Block Volume Attacher plug-in

If you don't want to provision and use the IBM Cloud Block Storage Attacher plug-in in your cluster, you can uninstall the Helm chart.

1. Find the name of the Helm chart for the IBM Cloud Block Storage Attacher plug-in.

   helm list | grep ibm-block-storage-attacher
   

   Example output

   <helm_chart_name>    1           Wed Aug  1 14:55:15 2022    DEPLOYED    ibm-block-storage-attacher-1.0.0    default
   

2. Delete the IBM Cloud Block Storage Attacher plug-in by removing the Helm chart.

   helm uninstall <helm_chart_name> -n <namespace>
   

3. Verify that the IBM Cloud Block Storage Attacher plug-in pods are removed.

   kubectl get pod -n kube-system -o wide | grep attacher
   

   The removal of the pods is successful if no pods are displayed in your CLI output.

4. Verify that the IBM Cloud Block Storage Attacher storage class is removed.

   kubectl get sc | grep attacher
   

The removal of the storage class is successful if no storage class is displayed in your CLI output.

Detaching raw and unformatted Block Storage for Classic from a worker node in a VPC cluster by using the API

You can use a DELETE request to detach storage from a VPC worker node.

Detaching storage from your VPC cluster does not remove your Block Storage for Classic volume or the data that is stored in the volume. You continue to get billed until you manually delete the volume.

1. Identify the storage volume that you want to remove and note the volume ID.

   ibmcloud is volumes
   

2. Get details about the volume. This command returns the worker node ID and attachment ID. Note the worker node ID. In the following command this ID is returned as "Instance name".

   ibmcloud is volume <volume_ID>
   

3. Retrieve a list of your PVs. This command returns a list of your PVs that you can then you use to determine which PVC uses the volume that you want to remove.

   kubectl get pv
   

4. Describe the PV that uses the volume. If you don't know which PV uses the volume that you want to remove, you can run the describe pv command on each PV in your cluster. Note the PVC that uses the PV.

   kubectl describe pv <pv_name>
   

5. Check to see if your storage volume is in use by a pod. The following command shows the pods that mount the volume and the associated PVC. If no pod is returned, the storage is not in use.

   kubectl get pods --all-namespaces -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.volumes[*]}{.persistentVolumeClaim.claimName}{" "}{end}{end}' | grep "<pvc_name>"
   

6. If the pod your volume is using is part of a deployment, delete the deployment. If your pod does not belong to a deployment, delete the pod.

   kubectl delete deployment <deployment_name>
   

   kubectl delete pod <pod_name>
   

7. Delete the PVC and PV.

   kubectl delete pvc <pvc_name>
   

   kubectl delete pv <pv_name>
   

8. Retrieve your IAM token.

   ibmcloud iam oauth-tokens
   

9. Detach storage by using a POST request.

   Example request

   curl -X POST "https://containers.cloud.ibm.com/global/v2/storage/deleteAttachment" -H  "accept: application/json" -H  "Authorization: <IAM_token>" -H  "X-Auth-Resource-Group-ID: <resource_group>" -H  "Content-Type: application/json" -d "{  \"cluster\": \"<cluster_name_or_ID\",  \"volumeAttachmentID\": \"<volume_attachment_ID>\",  \"volumeID\": \"<volume_ID>\",  \"worker\": \"<worker_ID>\"}"
   

   IAM_token

   The IAM OAuth token for your current session. You can retrieve this value by running ibmcloud iam oauth-tokens.

   cluster_name_or_ID

   The unique ID or name that is assigned to your cluster. You can retrieve this ID by running ibmcloud ks cluster ls.

   worker_ID

   The unique ID that is assigned to the worker node where you want to attach your volume. You can retrieve this value by running ibmcloud ks worker ls -c <cluster_name>.

   volume_ID

   The unique ID that is assigned to your Block Storage for Classic volume. You can retrieve a list of your Block Storage for Classic volumes by running ibmcloud is volumes.

   volume_attachment_ID

   The unique ID that is assigned to your volume attachment. You can retrieve this ID by running ibmcloud is volume <volume_ID>.

Reviewing volume attachment details for a VPC worker node by using the API

You can use a GET request to retrieve volume attachment details for a VPC worker node.

1. Retrieve your IAM token.

   ibmcloud iam oauth-tokens
   

2. Retrieve the ID of the resource group where your cluster is deployed.

   ibmcloud ks cluster get <cluster_name_or_ID> | grep "Resource Group ID"
   

3. Retrieve the ID of the worker node for which you want to see volume attachment details. Make sure to select a worker node that is located in the same zone as your Block Storage for Classic instance.

   ibmcloud ks worker ls --cluster <cluster_name_or_ID>
   

4. Review a list of existing volume attachments on a worker node.

   curl -X GET "https://containers.cloud.ibm.com/v2/storage/getAttachments?cluster=<cluster_ID>&worker=<worker_ID>" --header "X-Auth-Resource-Group-ID: <resource_group_ID>" --header "Authorization: <IAM_token>"
   

5. Retrieve the details for a specific attachment.

   curl -X GET "https://containers.cloud.ibm.com/v2/storage/getAttachment?cluster=<cluster_ID>&worker=<worker_ID>&volumeAttachmentID=<volume_attachment_ID>" --header "X-Auth-Resource-Group-ID: <resource_group_ID>" --header "Authorization: <IAM_token>"
   

   IAM_token

   The IAM OAuth token for your current session. You can retrieve this value by running ibmcloud iam oauth-tokens.

   cluster_ID

   The unique ID that is assigned to your cluster. You can retrieve this ID by running ibmcloud ks cluster ls.

   worker_ID

   The unique ID that is assigned to the worker node where you want to attach your volume. You can retrieve this value by running ibmcloud ks worker ls -c <cluster_name>.

   volume_ID

   The unique ID that is assigned to your Block Storage for Classic volume. You can retrieve a list of your Block Storage for Classic volumes by running ibmcloud is volumes.

   volume_attachment_ID

   The unique ID that is assigned to your volume attachment. You can retrieve this ID by running ibmcloud is volume <volume_ID>.

Removing raw Block Storage for VPC from VPC worker nodes by using the CLI

You can remove storage from your worker node by using the ibmcloud ks storage attachment rm command.

Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

1. List your storage volumes and note the ID of the volume that you want to remove.

   ibmcloud is vols
   

2. Get the details of your volume such as the worker-id where the volume is attached. The worker-id is listed as the Instance name in the Volume Attachment Instance Reference section of the command output.

   ibmcloud is vol <volume-ID>
   

   Example output

   Volume Attachment Instance Reference   Attachment type   Instance ID                                 Instance name                                        Auto delete   Attachment ID                               Attachment name      
                                       data              0727_e18c10d7-7f18-48aa-b5ef-5ed163e54198   kube-bsaucubd07dhl66e4tgg-cluster-default-00000a19   false         0727-3bfe90b0-dc2d-498a-946b-8837a5dad7bc   volume-attachment
   

3. List the storage attachments on a worker node in your cluster and make a note of the attachment ID that your want to remove.

   ibmcloud ks storage attachment ls -c <cluster> --worker <worker-id>
   

   Example output:

   Listing volume attachments...
   OK
   ID                                          Name                Status     Type   Volume ID                                   Volume Name                          Worker ID   
   0111-1a111aaa-1111-1111-111a-aaa1a1a11a11   volume-attachment   attached   boot   a001-f11ed1e1-1aa1-11dc-b11d-a0dc111b1111  dissuade-anointer-errand-handbrake   kube-aa1111aa11aaaaa11aa1-cluster-name-default-00000110
   

4. Remove the storage attachment.

   ibmcloud ks storage attachment rm --attachment <attachment-ID> -c <cluster> --worker <worker-ID>
   

5. Verify that your storage was removed from the worker node.

   ibmcloud ks storage attachment ls -c <cluster-ID> --worker <worker-id>
   

Setting up an IBM Cloud Object Storage service instance

Create and configure an IBM Cloud Object Storage service instance to serve as the repository for the data that you want to back up.

1. Create an IBM Cloud Object Storage service instance that uses HMAC credentials.
2. Store your IBM Cloud Object Storage credentials in a Kubernetes secret.
3. Create your first IBM Cloud Object Storage bucket.
   1. In the navigation on the service details page, click Buckets.
   2. Click Create bucket. A dialog box is displayed.
   3. Enter a unique name for your bucket. The name must be unique within IBM Cloud Object Storage across all regions and across all IBM Cloud accounts.
   4. From the Resiliency list, select the level of availability that you want for your data. For more information, see IBM Cloud Object Storage regions and endpoints. For VPC clusters, make a note of the direct endpoint. For example: s3.direct.us.cloud-object-storage.appdomain.cloud.
   5. Change the Location to the region where you want to store your data. Keep in mind that your data might not be allowed to be stored in every region due to legal reasons.
   6. Click Create.
4. Retrieve the IBM Cloud Object Storage host name for your bucket.
   1. Click on your bucket name that you created in the previous step.
   2. In the navigation on the service details page, click Buckets > Configuration.
   3. Note the public URL that you can use to access the data in your bucket.

For more information about configuring your service instance, see the IBM Cloud Object Storage documentation.

Using IBM Cloud Object Storage to back up and restore PVC data

You can use the IBM Cloud Backup Restore Helm chart to back up data in a file storage or block storage PVC to IBM Cloud Object Storage, or restore data from IBM Cloud Object Storage to a PVC in your cluster.

Before you begin:

• Make sure that you have a PVC that you can back up or restore data to. For more information about how to create a PVC, see Adding file storage to apps and Adding block storage to apps.

• If you want to back up a block storage PVC, make sure that your PVC is not mounted to an app. Block storage is mounted with a RWO access mode. This access allows only one pod to be mounted to the block storage at a time. To back up your data, you must remove the app pod that mounts the storage. To check whether a pod is mounted to your PVC, run the following command.

  kubectl get pods --all-namespaces -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.volumes[*]}{.persistentVolumeClaim.claimName}{" "}{end}{end}' | grep "<pvc_name>"
  

• Retrieve your IBM Cloud Object Storage service credentials, the bucket name, and the bucket hostname.

• Follow the instructions to install the Helm client on your local machine and set up the IBM Cloud Helm chart repositories.

• Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

You can deploy the ibm-storage-backup pod or the ibm-storage-restore pod by either editing and applying the values.yaml file of the Helm chart, or by running the helm install command from the CLI.

To back up or restore a PVC by editing the values.yaml file:

1. Download the latest Helm chart version to your local machine.

   helm fetch --untar iks-charts/ibmcloud-backup-restore
   

2. Open the values.yaml file in the nano command line editor.

   nano ibmcloud-backup-restore/values.yaml
   

3. Configure your Helm chart to back up or restore PVC data. You can configure the backup for more than one PVC.

   Example to create a backup pod by configuring the values.yaml file:

   image:
       repository: icr.io/iks-charts/ibmcloud-backup-restore
   pullPolicy: Always
   tag: latest
   ACCESS_KEY_ID: # Example: 10110abab1111bbb111aa1aaa111b1a1
   SECRET_ACCESS_KEY: # Example: a1aba11aaa11b11b11aa1111a1111ba111111111a0b1b11a
   ENDPOINT: # Example: s3.us-east.cloud-object-storage.appdomain.cloud
   BUCKET_NAME: # Example: my-bucket
   BACKUP_NAME: # Example: my_backup
   PVC_NAMES:
       - # Example: my_pvc
   - # Optional example: my_pvc2
   CHART_TYPE: # Example: backup or restore
   BACKUP_TYPE: # Example: incremental
   SCHEDULE_TYPE: # Example: periodic
   SCHEDULE_INFO: # Example: weekly
   

   ACCESS_KEY_ID

   Enter the access key ID of the IBM Cloud Object Storage service credentials that you retrieved earlier.

   SECRET_ACCESS_KEY

   Enter the secret access key of the IBM Cloud Object Storage service credentials that you retrieved earlier.

   ENDPOINT

   Enter the public IBM Cloud Object Storage s3 API endpoint for your bucket that you retrieved earlier.

   BUCKET_NAME

   Backup: Enter the name of the IBM Cloud Object Storage bucket that you created earlier. You use this bucket to store PVC data when you perform a backup. Restore: Enter the name of the IBM Cloud Object Storage bucket where your backup is stored.

   BACKUP_NAME

   Backup: Enter the name of the backup that you want to create in IBM Cloud Object Storage. Restore: Enter the name of the backup that you created with the IBM Cloud Backup Restore Helm chart in IBM Cloud Object Storage. If you have multiple full backups in your IBM Cloud Object Storage service instance, the PVC is restored with the data of the last full backup. If you have incremental backups, the PVC is restored with the data of the last full backup, including all incremental backups up to the day where you start the restore.

   PVC_NAMES

   Backup: Enter the name of the PVC that you want to back up. If you want to back up multiple PVCs, add each PVC to the list of PVCs. To list available PVCs in your cluster that you can back up, run kubectl get pvc. Restore: Enter the name of the PVC to which you want to restore data from IBM Cloud Object Storage. You can restore data to one PVC at a time only. To list available PVCs in your cluster that you can restore data to, run kubectl get pvc.

   CHART_TYPE

   Enter the name of the chart type that you want to deploy. Enter backup to deploy the backup chart. Enter restore to deploy the restore chart.

   BACKUP_TYPE

   Required only for backups. Enter full to create a full backup, or incremental if you want to back up only new or changed files. If you choose incremental, you must specify the SCHEDULING_INFO and SCHEDULING_TYPE option. If you don't specify the BACKUP_TYPE option, a full backup is created by default.

   SCHEDULE_TYPE

   Required only for backups. Enter periodic to create scheduled backups, or leave this option empty to create a one-time backup. If you want to create periodic backups, you must define the backup interval in the SCHEDULE_INFO option.

   SCHEDULE_INFO

   Required only for backups. If you want to create periodic backups, you must decide on the backup schedule. Choose between hourly, daily, or weekly. If you set this option, you must set SCHEDULE_TYPE to periodic.

4. Save and close the values.yaml file.

5. Install the Helm chart with your custom settings in the values.yaml file. When you install the Helm chart and you configure a backup or restore, an ibm-storage-backup or an ibm-storage-restore pod is deployed to your cluster. The backup pod backs up the data from your PVC to IBM Cloud Object Storage and the restore pod restores data to a PVC. Replace <release_name> with a name for your Helm chart. Be sure to install the backup and restore pods in the same zone as the PVC that you want to back up or restore.

   • Install the Helm chart by using the helm install command.

     helm install <release_name> ./ibmcloud-backup-restore -n <namespace>
     

     Example output for backup:

     NAME: <release_name>
     LAST DEPLOYED: Mon Jan 20 09:17:02 2020
     NAMESPACE: default
     STATUS: deployed
     REVISION: 1
     TEST SUITE: None
     NOTES:
     Thank you for installing: ibmcloud-backup-restore.   Your release is named: <release_name>
     
     Please refer Chart README.md file for creating a sample PVC
     Please refer Chart RELEASE.md to see the release details/fixes
     

   • Optional: Install the Helm chart by setting options in the helm install command. You can name your release by specifying the --name parameter.

     helm install <release_name> --set ACCESS_KEY_ID=<access_key_ID> --set SECRET_ACCESS_KEY=<secret_access_key> --set ENDPOINT=<public_bucket_endpoint> --set BUCKET_NAME=<bucket_name> --set BACKUP_NAME=<backup_name> --set PVC_NAMES[0]=<pvc_name1> --set PVC_NAMES[1]=<pvc_name2> --set CHART_TYPE=backup --set BACKUP_TYPE=<backup_type> --set SCHEDULE_TYPE=<schedule_type> --set SCHEDULE_INFO=<schedule_info> ./ibmcloud-backup-restore
     

6. Verify that your data backup or restore completed successfully.

   Backup:

   1. Verify that the ibm-storage-backup pod has a status of Running.

      kubectl get pods -A | grep backup
      

      Example output

      ibm-storage-backup                        1/1     Running             0          64m
      

   2. Review the logs of the ibm-storage-backup pod to make sure that your backup was successful. When you see the ... backup completed message in the event logs, your backup completed successfully.

      kubectl logs ibm-storage-backup
      

      Example output for daily backups.

      [2019-04-18 16:01:51,157] [utilities : 151] [INFO] *****************Start logging to ./Backup.log
      [2019-04-18 16:01:51,158] [backup : 48] [INFO] Starting backup:
      [2019-04-18 16:01:51,158] [configureOS : 66] [INFO] Configuring duplicity with IBM CloudObjectStorage i.e s3.
      [2019-04-18 16:01:51,158] [backup : 62] [INFO] Configuration done!!!
      [2019-04-18 16:01:51,158] [backup : 78] [INFO] Got all required input from config file!!
      [2019-04-18 16:01:52,366] [backup : 119] [WARNING] Incremental backup was not created
      [2019-04-18 16:01:52,366] [backup : 120] [INFO] duplicity  --no-encryption incremental /myvol s3://s3.us-south.cloud-object-storage.appdomain.cloud/mybucket/helm-backup command failed due to Fatal Error: Unable to start incremental backup.  Old signatures not found and incremental specified
      
      [2019-04-18 16:01:52,367] [backup : 121] [INFO] A full backup is required before incremental backups can begin. Creating a one-time full backup and will run incremental backups for scheduled backups.
      [2019-04-18 16:01:54,357] [backup : 129] [INFO] Full backup completed
      [2019-04-18 16:01:54,357] [backup : 130] [INFO] Local and Remote metadata are synchronized, no sync needed.
      Last full backup date: none
      --------------[ Backup Statistics ]--------------
      StartTime 1555603313.31 (Thu Apr 18 16:01:53 2019)
      EndTime 1555603313.32 (Thu Apr 18 16:01:53 2019)
      ElapsedTime 0.01 (0.01 seconds)
      SourceFiles 3
      SourceFileSize 20495 (20.0 KB)
      NewFiles 3
      NewFileSize 20495 (20.0 KB)
      DeletedFiles 0
      ChangedFiles 0
      ChangedFileSize 0 (0 bytes)
      ChangedDeltaSize 0 (0 bytes)
      DeltaEntries 3
      RawDeltaSize 15 (15 bytes)
      TotalDestinationSizeChange 183 (183 bytes)
      Errors 0
      -------------------------------------------------
      
      [2019-04-18 16:01:54,357] [backup : 162] [INFO] Scheduling backup as per configurations, please don't stop this program or run this in background !!!
      [2019-04-18 16:01:54,358] [backup : 166] [INFO] Schedule info is: ['daily']
      [2019-04-18 16:01:54,358] [backup : 172] [INFO] Scheduled for daily!!!
      

7. Verify that your data is successfully backed up or restored.

   • Backup:

     1. Find your IBM Cloud Object Storage service instance in the IBM Cloud resource list.
     2. From the navigation, select Buckets and click on the bucket that you used in your backup configuration. Your backup is displayed as an object in your bucket.
     3. Review the compressed files. You can download the *.gz file, extract the file, and verify the backed-up data.

   • Restore:

     1. Create a deployment.yaml file with a pod that mounts the PVC that contains your restored data. The following example deploys an nginx pod that mounts the PVC on the /test mount directory.

          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: restore
            labels:
            app: nginx
          spec:
            selector:
              matchLabels:
                app: nginx
            template:
              metadata:
                labels:
                  app: nginx
              spec:
                containers:
                  - image: nginx
                    name: nginx
                    volumeMounts:
                    - name: my-volume # Example: my_volume
                      mountPath: /test # Example: /test
                volumes:
                - name: my-volume # Example: my_volume
                  persistentVolumeClaim:
                    claimName: my-actual-pvc-name # Example: my_pvc
        

        spec.containers.image

        The name of the image that you want to use. To list available images in your IBM Cloud Container Registry account, run ibmcloud cr image-list.

        spec.containers.name

        The name of the container that you want to deploy to your cluster.

        spec.containers.volumeMounts.mountPath

        The absolute path of the directory to where the volume is mounted inside the container. Data that is written to the mount path is stored under the root directory in your physical block storage instance. If you want to share a volume between different apps, you can specify volume sub paths for each of your apps.

        spec.containers.volumeMounts.name

        The name of the volume to mount to your pod.

        volumes.name

        The name of the volume to mount to your pod. Typically this name is the same as volumeMounts/name

        volumes.persistentVolumeClaim.claimName

        The name of the PVC that binds the PV that you want to use.

     2. Create the deployment.

        kubectl apply -f deployment.yaml
        

     3. Verify that your pod has a status of Running.

        If you find that your ibm-storage-restore pod does not reach a Completed or CrashLoopBackOff status, restoring your data might have failed. Run kubectl logs ibm-storage-restore to find the root cause for the failure.

        kubectl get pods | grep restore
        

        Example output

        restore-7dfc6f4c78-wkcqp                  1/1     Running             0          3m54s
        

     4. Log in to your pod.

        kubectl exec <pod_name> -it bash
        

     5. Navigate to the mount directory that you specified in your deployment YAML.

        cd <mount_directory>
        

     6. List the files in your mount directory to verify that all your data is restored to the mount directory.

        ls
        

     7. Delete the Helm chart installation from your cluster. This step is required if you restored data to a block storage PVC. Block storage is mounted with a RWO access mode. This access allows only one pod to be mounted to the block storage at a time. Because the ibm-storage-restore pod already mounts the PVC, you must remove the pod to release the PVC so that you can mount the PVC to a different pod in your cluster.

        helm uninstall <release_name> -n <namespace>
        

     8. You successfully restored your backup. You can now mount the PVC that binds the PV to any other pod in your cluster to access the restored files. If the container data that was backed up included a non-root user, you must add non-root permissions to your new container. For more information, see Adding non-root user access to volumes.

Troubleshooting persistent storage when an IBM Cloud Monitoring alert is triggered

When an alert is triggered, review the alert details in IBM Cloud Monitoring and review troubleshooting guides for persistent storage, apps, worker nodes, and clusters to find the root cause of the alert. The alerts that you set up might not be related to a storage volume issue, but to issues that occurred within your app, on the worker node, or in the cluster.

• Troubleshooting persistent storage volumes.
• Troubleshooting worker nodes.
• Troubleshooting apps.
• Troubleshooting clusters.
• Open a support case.