Setting up Block Storage for Classic

Updating the IBM Cloud Block Storage plug-in

You can upgrade the existing IBM Cloud Block Storage plug-in to the latest version.

Before you begin: Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

1. Update the Helm repo to retrieve the latest version of all Helm charts in this repo.

   helm repo update
   

2. Optional: Download the latest Helm chart to your local machine. Then, extract the package and review the release.md file to find the latest release information.

   helm pull iks-charts/ibmcloud-block-storage-plugin --untar
   

3. Find the release name and namespace of the block storage Helm chart that you installed in your cluster.

   helm ls -A
   

   Example output

   NAME            NAMESPACE   REVISION    UPDATED                             STATUS      CHART                                   APP VERSION
   block-plugin    default     1           2022-01-21 09:02:46.11622 -0500 EST deployed        bmcloud-block-storage-plugin-v2.1.5
   

4. Upgrade the IBM Cloud Block Storage plug-in to the latest version. Include the release name and the namespace that you retrieved earlier.

   helm upgrade RELEASE-NAME iks-charts/ibmcloud-block-storage-plugin -n NAMESPACE
   

5. Optional: When you update the plug-in, the default storage class is unset. If you want to set the default storage class to a storage class of your choice, run the following command.

   kubectl patch storageclass STORAGECLASS -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
   

Removing the IBM Cloud Block Storage plug-in

If you don't want to provision and use IBM Cloud Block Storage in your cluster, you can uninstall the Helm chart.

Removing the plug-in does not remove existing PVCs, PVs, or data. When you remove the plug-in, all the related pods and daemon sets are removed from your cluster. You can't provision new block storage for your cluster or use existing block storage PVCs and PVs after you remove the plug-in.

Before you begin:

• Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.
• Make sure that you don't have any PVCs or PVs in your cluster that use block storage.

To remove the plug-in:

1. Find the release name and the namespace of the block storage Helm chart that you installed in your cluster.

   helm ls -A
   

   Example output

   NAME            NAMESPACE   REVISION    UPDATED                             STATUS   CHART                                  APP VERSION
   block-plugin    default     1           2022-01-21 09:02:46.11622 -0500 EST deployed    ibmcloud-block-storage-plugin-v2.1.5
   

2. Delete the IBM Cloud Block Storage plug-in.

   helm uninstall NAME -n kube-system
   

3. Verify that the block storage pods are removed.

   kubectl get pods -n kube-system | grep block
   

   The removal of the pods is successful if no pods are displayed in your CLI output.

4. Verify that the block storage classes are removed. The removal of the storage classes is successful if no storage classes are displayed in your CLI output.

   kubectl get sc | grep block
   

Encrypting volume data by using your own storage class

You can deploy apps that use encrypted volumes by first creating your own storage class.

The following steps explain how to create a custom, encrypted storage class that you can use to create multiple encrypted block storage instances with the same configuration. If you want to create an encrypted PVC by using one of the IBM-provided storage classes, you can do this by referencing the Key Protect credentials directly in your PVC.

1. Decide on a storage configuration.

2. Create your own storage class that provisions an encrypted block storage instance by using one of the IBM-provided storage classes as the basis. You can retrieve the details a storage class by running kubectl get sc <storageclass_name> -o yaml. The following example is based on the ibmc-block-retain-bronze storage class.

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: <name> # Enter the name of the storage class. Example: my_custom_storageclass
   parameters:
     billingType: hourly
     classVersion: "2"
     fsType: ext4
     iopsPerGB: "2"
     sizeRange: '[20-12000]Gi'
     type: Endurance
     encrypted: "true" # Enter "true" to enable encryption.
     encryptionKeySecret: <secret_name> # # #nter the name of the secret that you created earlier.Example: my_secret 
     encryptionKeyNamespace: <namespace> # # #nter the namespace where you created your secret. Example: default
   provisioner: ibm.io/ibmc-block
   reclaimPolicy: Delete
   volumeBindingMode: Immediate
   

3. Create the storage class in your cluster.

   kubectl apply -f storageclass.yaml
   

4. Add Block Storage for Classic to your app by using your own storage class to create a PVC.

5. Verify the encryption of your Block Storage for Classic volumes.

Create a PVC that references your Block Storage for Classic secret

You can provision encrypted Block Storage for Classic by creating a PVC that specifies the Kubernetes secret that holds your Key Protect credentials.

The following steps show how you can reference your Key Protect credentials in your PVC to create an encrypted Block Storage for Classic instance. To create multiple encrypted volumes without specifying the Key Protect credentials in each PVC, you can create a custom, encrypted storage class.

1. Review the provided Block Storage for Classic storage classes to determine which storage class best meets your app requirements. If the provided storage classes don't meet your app requirements, you can create your own customized storage class.

2. Create a PVC configuration file that is named pvc.yaml and that references the Kubernetes secret where you stored the Key Protect service credentials. To create this secret, see Setting up encryption for Block Storage for Classic.

   kind: PersistentVolumeClaim
   apiVersion: v1
   metadata:
     name: <pvc_name> # Enter a name for your PVC.
     annotations:
     volume.beta.kubernetes.io/storage-class: "<storage_class>" # Enter a storage class. To see a list of storageclasses run `kubectl get storageclasses`.
     labels:
       encrypted: "true"
       encryptionKeyNamespace: <namespace> # Enter the namespace where your secret was created.
       encryptionKeySecret: <secret_name> # Enter the name of the secret you created.
   spec:
     accessModes:
       - ReadWriteOnce
       resources:
       requests:
           storage: 20Gi
   

3. Create the PVC in your cluster.

   kubectl apply -f pvc.yaml
   

4. Check the status of your PVC.

   kubectl get pvc
   

5. Wait for your PVC to bind, then create a deployment that uses your PVC.

6. Verify the encryption of your Block Storage for Classic volumes.

Verifying the encryption of your Block Storage for Classic volumes

You can verify the encryption of your volumes by checking the volume mount path.

1. Log in to your app pod. Replace <pod_name> with the name of the pod that mounts your encrypted Block Storage for Classic volume.

   kubectl exec <pod_name> -it bash
   

2. List the file system of your pod.

   df -h
   

3. Review the file system path for your encrypted Block Storage for Classic volume.

   • Encrypted volumes have a path structure of /dev/mapper/<pvc-ID_encrypted>. In this example, the encrypted volume is mounted to the /test file path in the pod.

     Filesystem                                            Size  Used Avail Use% Mounted on
     overlay                                                98G  8.2G   85G   9% /
     tmpfs                                                  64M     0   64M   0% /dev
     tmpfs                                                 2.0G     0  2.0G   0% /sys/fs/cgroup
     /dev/mapper/pvc-a011a111-1111-1111-111a-aaa1a1111a11_encrypted   20G   45M   20G   1% /test
     

   • Unencrypted volumes have a path structure of dev/mapper/<random_string>.

     Filesystem                                     Size  Used Avail Use% Mounted on
     overlay                                         98G   16G   78G  17% /
     tmpfs                                           64M     0   64M   0% /dev
     tmpfs                                          7.9G     0  7.9G   0% /sys/fs/cgroup
     /dev/mapper/3600a09803830476e733f4e477370716e   24G   45M   24G   1% /test
     

Removing your Kubernetes secret doesn't revoke access to the volume data. If you created a pod-only deployment, you must delete the pod. If you created a deployment, you must delete the deployment.

Retrieving the information of your existing block storage

1. Retrieve or generate an API key for your IBM Cloud infrastructure account.

   1. Log in to the IBM Cloud infrastructure portal.
   2. Select Account, then Users, and then User List.
   3. Find your user ID.
   4. In the API KEY column, click Generate to generate an API key or View to view your existing API key.

2. Retrieve the API username for your IBM Cloud infrastructure account.

   1. From the User List menu, select your user ID.
   2. In the API Access Information section, find your API Username.

3. Log in to the IBM Cloud infrastructure CLI plug-in.

   ibmcloud sl init
   

4. Choose to authenticate by using the username and API key for your IBM Cloud infrastructure account.

5. Enter the username and API key that you retrieved in the previous steps.

6. List available block storage devices.

   ibmcloud sl block volume-list
   

   Example output

   id          username              datacenter   storage_type                capacity_gb   bytes_used   lunId   
   11111111    IBM01AAA1111111-1     wdc07        endurance_block_storage     45            -            2      
   

7. Retrieve the volume details. Replace <volume_ID> with the ID of the Block storage volume that you retrieved in step 6.

   ibmcloud sl block volume-detail <volume_ID>
   

   Example output

   ID                         11111111 
   User name                  IBM01AAA1111111-1   
   Type                       endurance_block_storage   
   Capacity (GB)              45   
   LUN Id                     2   
   IOPs                       100   
   Datacenter                 wdc07   
   Target IP                  10.XXX.XX.XXX  
   # of Active Transactions   0   
   Replicant Count            0 
   

8. Make a note of the ID, Capacity, LUN Id, the Datacenter, and Target IP of the volume that you want to mount to your cluster. Note: To mount existing storage to a cluster, you must have a worker node in the same zone as your storage. To verify the zone of your worker node, run ibmcloud ks worker ls --cluster <cluster_name_or_ID>.

Creating a persistent volume (PV) and a matching persistent volume claim (PVC)

1. Optional: If you have storage that you provisioned with a retain storage class, when you remove the PVC, the PV and the physical storage device are not removed. To reuse the storage in your cluster, you must remove the PV first. List existing PVs and look for the PV that belongs to your persistent storage. The PV is in a released state.

   kubectl get pv
   

2. Remove the PV.

   kubectl delete pv <pv_name>
   

3. Verify the PV is removed.

   kubectl get pv
   

4. Create a configuration file for your PV. Include the parameters you retrieved earlier.

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: "block-storage-pv" # Enter a name for your PV. For example, my-static-pv.
     labels:
        failure-domain.beta.kubernetes.io/region: "<region>" # Example us-east.
        failure-domain.beta.kubernetes.io/zone: "<zone>" # Example: wdc04. See /docs/containers?topic=containers-regions-and-zones#zones-sz
   spec:
     capacity:
       storage: "<storage>"
     accessModes:
       - ReadWriteOnce
     flexVolume:
       driver: "ibm/ibmc-block"
       fsType: "<fs_type>" # Enter ext or xfs
       options:
         "Lun": "<Lun_ID>"
         "TargetPortal": "<TargetPortal>"
         "VolumeID": "<VolumeID>"
         "volumeName": "block-storage-pv" # Enter the same value as your PV name from metadata.name
   

   name

   Give your PV a name. For example, block-storage-pv. Note that you must also enter this value in spec.FlexVolume.options as the volumeName.

   labels

   Enter the region and the zone that you retrieved earlier. You must have at least one worker node in the same region and zone as your persistent storage to mount the storage in your cluster. To retrieve your volume details, run ibmcloud sl block volume-list to get the volume ID, then run ibmcloud sl block volume-detail <volume_ID> to get the details of your volume.

   region

   Enter the region where your block storage is located. Note that your cluster and block storage must be in the same region. To find your cluster location, run ibmcloud ks cluster ls. For more information about the available regions and zones, see regions and zones. For example, us-east.

   zone

   Enter the zone where your storage volume is located. To retrieve your volume details, run ibmcloud sl block volume-list to get the volume ID, then run ibmcloud sl block volume-detail <volume_ID> to get the details of your volume. Note that to attach block storage to your cluster, you must have a worker node available in the same zone as the volume that you want to attach. To find the zones of your worker nodes, run ibmcloud ks worker ls -c <cluster>. For example, wdc04.

   storage

   Enter the storage size of the existing block storage volume that you want to attach to your cluster. The storage size must be written in gigabytes, for example, 20Gi (20 GB) or 1000Gi (1 TB). To retrieve your volume details, run ibmcloud sl block volume-list to get the volume ID, and then run ibmcloud sl block volume-detail <volume_ID> to get the details of your volume.

   fsType

   Enter the file system type that is configured for your existing block storage. Choose between ext4 or xfs. If you don't specify this option, the PV defaults to ext4. When the wrong fsType is defined, then the PV creation succeeds, but the mounting of the PV to a pod fails. To retrieve your volume details, run ibmcloud sl block volume-list to get the volume ID, then run ibmcloud sl block volume-detail <volume_ID> to get the details of your volume.

   Lun

   Enter the LUN ID of your block storage volume. To retrieve your volume details, run ibmcloud sl block volume-list to get the volume ID, then run ibmcloud sl block volume-detail <volume_ID> to get the details of your volume.

   TargetPortal

   Enter the IP address of your block storage. To retrieve the TargetPortal parameter, run ibmcloud sl block volume-list to get the volume ID, then run ibmcloud sl block volume-detail <volume_ID> and make a note of the Target IP in the output.

   VolumeId

   Enter the ID of your block storage. To retrieve your volume details, run ibmcloud sl block volume-list.

   volumeName

   Enter the same value as your PV name. For example, block-storage-pv.

5. Create the PV in your cluster.

   kubectl apply -f pv.yaml
   

6. Verify that the PV is created.

   kubectl get pv
   

7. Create another configuration file to create your PVC. In order for the PVC to match the PV that you created earlier, you must choose the same value for storage and accessMode. The storage-class field must be an empty string. If any of these fields don't match the PV, then a new PV is created automatically instead.

   kind: PersistentVolumeClaim
   apiVersion: v1
   metadata:
     name: block-storage-pvc
   spec:
     accessModes:
     - ReadWriteOnce
     resources:
       requests:
         storage: "20Gi"
     storageClassName: ""
   

8. Create your PVC.

   kubectl apply -f static-pvc.yaml
   

9. Verify that your PVC is created and bound to the PV that you created earlier. This process can take a few minutes.

   kubectl describe pvc static-pvc
   

   Example output

   Name:          static-pvc
   Namespace:     default
   StorageClass:  
   Status:        Bound
   

10. Optional Save the following example pod configuration as a file called pod.yaml.

    apiVersion: v1
    kind: Pod
    metadata:
      name: block-storage
      labels:
        app: block-storage
    spec:
      containers:
        - name: block-storage
          image: nginx
          command: ["/bin/sh"]
          args: ["-c", "while true; do date \"+%Y-%m-%d %H:%M:%S\"; sleep 3600; done"]
          workingDir: /home
          imagePullPolicy: Always
          ports:
            - containerPort: 80
          volumeMounts:
            - name: block-storage-pv
              mountPath: /home
      volumes:
        - name: block-storage-pv
          persistentVolumeClaim:
            claimName: block-storage-pvc
    

11. Create the pod in your cluster.

    kubectl create -f pod.yaml
    

12. After the pod is in Running status, get the logs.

    kubectl logs
    

    Example output

    2022-01-21 16:11:00
    

You successfully created a PV and bound it to a PVC. Then, you deployed and app that uses block storage. Cluster users can now mount the PVC to their deployments and start reading from and writing to the persistent volume.

Creating the PVC by using dynamic provisioning when you create a stateful set

Use this option if you want to automatically create the PVC when you create the stateful set.

Before you begin: Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

Complete the following steps to verify that all existing stateful sets in your cluster are fully deployed. If a stateful set is still being deployed, you can't start creating your stateful set. You must wait until all stateful sets in your cluster are fully deployed to avoid unexpected results.

1. List existing stateful sets in your cluster.

   kubectl get statefulset --all-namespaces
   

   Example output

   NAME              DESIRED   CURRENT   AGE
   mystatefulset     3         3         6s
   

2. View the Pods Status of each stateful set to ensure that the deployment of the stateful set is finished.

   kubectl describe statefulset <statefulset_name>
   

   Example output

   Name:               nginx
   Namespace:          default
   CreationTimestamp:  Fri, 05 Oct 2022 13:22:41 -0400
   Selector:           app=nginx,billingType=hourly,region=us-south,zone=dal10
   Labels:             app=nginx
   billingType=hourly
   region=us-south
   zone=dal10
   Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"apps/v1","kind":"StatefulSet","metadata":{"annotations":{},"name":"nginx","namespace":"default"},"spec":{"podManagementPolicy":"Par..."
   Replicas:           3 desired | 3 total
   Pods Status:        0 Running / 3 Waiting / 0 Succeeded / 0 Failed
   Pod Template:
   Labels:  app=nginx
   billingType=hourly
   region=us-south
   zone=dal10
   ...
   

   A stateful set is fully deployed when the number of replicas that you find in the Replicas section of your CLI output equals the number of Running pods in the Pods Status section. If a stateful set is not fully deployed yet, wait until the deployment is finished before you proceed.

3. Create a configuration file for your stateful set and the service that you use to expose the stateful set. The following example shows how to deploy NGINX as a stateful set with three replicas. For each replica, a 20 gigabyte block storage device is provisioned based on the specifications that are defined in the ibmc-block-retain-bronze storage class. All storage devices are provisioned in the dal10 zone. Because block storage can't be accessed from other zones, all replicas of the stateful set are also deployed onto worker nodes that are located in dal10.

   apiVersion: v1
   kind: Service
   metadata:
    name: nginx
    labels:
   app: nginx
   spec:
   ports:
   - port: 80
       name: web
   clusterIP: None
   selector:
       app: nginx
   ---
   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
     name: nginx
   spec:
     serviceName: "nginx"
     replicas: 3
     podManagementPolicy: Parallel
     selector:
       matchLabels:
       app: nginx
       billingType: "hourly"
       region: "us-south" # Enter the region where your cluster is located.
       zone: "dal10"
   template:
     metadata:
     labels:
         app: nginx
         billingType: "hourly"
         region: "us-south"
         zone: "dal10"
     spec:
     containers:
     - name: nginx
       image: nginx
       ports:
       - containerPort: 80
         name: web
       volumeMounts:
       - name: myvol
       mountPath: /usr/share/nginx/html
   volumeClaimTemplates:
   - metadata:
       name: myvol
       spec:
       accessModes:
       - ReadWriteOnce
       resources:
           requests:
           storage: 20Gi
           iops: "300" #required only for performance storage
       storageClassName: ibmc-block-retain-bronze
   

   The following example shows how to deploy NGINX as a stateful set with three replicas. The stateful set does not specify the region and zone where the block storage is created. Instead, the stateful set uses an anti-affinity rule to ensure that the pods are spread across worker nodes and zones. By defining topologykey: failure-domain.beta.kubernetes.io/zone, the Kubernetes scheduler can't schedule a pod on a worker node if the worker node is in the same zone as a pod that has the app: nginx label. For each stateful set pod, two PVCs are created as defined in the volumeClaimTemplates section, but the creation of the block storage instances is delayed until a stateful set pod that uses the storage is scheduled. This setup is referred to as topology-aware volume scheduling.

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: ibmc-block-bronze-delayed
   parameters:
     billingType: hourly
     classVersion: "2"
     fsType: ext4
     iopsPerGB: "2"
     sizeRange: '[20-12000]Gi'
     type: Endurance
   provisioner: ibm.io/ibmc-block
   reclaimPolicy: Delete
   volumeBindingMode: WaitForFirstConsumer
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: nginx
     labels:
       app: nginx
   spec:
     ports:
     - port: 80
       name: web
     clusterIP: None
     selector:
       app: nginx
   ---
   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
     name: web
   spec:
     serviceName: "nginx"
     replicas: 3
     podManagementPolicy: "Parallel"
     selector:
       matchLabels:
         app: nginx
     template:
       metadata:
         labels:
           app: nginx
       spec:
         affinity:
           podAntiAffinity:
             preferredDuringSchedulingIgnoredDuringExecution:
             - weight: 100
               podAffinityTerm:
                 labelSelector:
                   matchExpressions:
                   - key: app
                     operator: In
                     values:
                     - nginx
                 topologyKey: failure-domain.beta.kubernetes.io/zone
         containers:
         - name: nginx
           image: k8s.gcr.io/nginx-slim:0.8
           ports:
           - containerPort: 80
             name: web
           volumeMounts:
           - name: myvol1
             mountPath: /usr/share/nginx/html
           - name: myvol2
             mountPath: /tmp1
     volumeClaimTemplates:
     - metadata:
         name: myvol1
       spec:
         accessModes:
         - ReadWriteOnce # access mode
         resources:
           requests:
             storage: 20Gi
         storageClassName: ibmc-block-bronze-delayed
     - metadata:
         name: myvol2
       spec:
         accessModes:
         - ReadWriteOnce # access mode
         resources:
           requests:
             storage: 20Gi
         storageClassName: ibmc-block-bronze-delayed
   

   name

   Enter a name for your stateful set. The name that you enter is used to create the name for your PVC in the format: <volume_name>-<statefulset_name>-<replica_number>.

   serviceName

   Enter the name of the service that you want to use to expose your stateful set.

   replicas

   Enter the number of replicas for your stateful set.

   podManagementPolicy

   Enter the pod management policy that you want to use for your stateful set.

   • OrderedReady: With this option, stateful set replicas are deployed one after another. For example, if you specified three replicas, then Kubernetes creates the PVC for your first replica, waits until the PVC is bound, deploys the stateful set replica, and mounts the PVC to the replica. After the deployment is finished, the second replica is deployed. For more information about this option, see OrderedReady Pod Management
   • Parallel: With this option, the deployment of all stateful set replicas is started at the same time. If your app supports parallel deployment of replicas, then use this option to save deployment time for your PVCs and stateful set replicas.

   matchLabels

   In the spec selector section, enter all labels that you want to include in your stateful set and your PVC. Labels that you include in the volumeClaimTemplates of your stateful set are not recognized by Kubernetes. Sample labels that you might want to include are:

   • region and zone: If you want all your stateful set replicas and PVCs to be created in one specific zone, add both labels. You can also specify the zone and region in the storage class that you use. If you don't specify a zone and region and you have a multizone cluster, the zone in which your storage is provisioned is selected on a round-robin basis to balance volume requests evenly across all zones.
   • billingType: Enter the billing type that you want to use for your PVCs. Choose between hourly or monthly. If you don't specify this label, all PVCs are created with an hourly billing type.

   labels

   In the spec template metadata section, enter the same labels that you added to the spec.selector.matchLabels section.

   affinity

   In the spec template spec section, specify your anti-affinity rule to ensure that your stateful set pods are distributed across worker nodes and zones. The example shows an anti-affinity rule where the stateful set pod prefers not to be scheduled on a worker node where a pod runs that has the app: nginx label. The topologykey: failure-domain.beta.kubernetes.io/zone restricts this anti-affinity rule even more and prevents the pod to be scheduled on a worker node if the worker node is in the same zone as a pod that has the app: nginx label. By using this anti-affinity rule, you can achieve anti-affinity across worker nodes and zones.

   name

   In the spec volume claim templates metadata section, enter a name for your volume. Use the same name that you defined in the spec.containers.volumeMount.name section. The name that you enter here is used to create the name for your PVC in the format: <volume_name>-<statefulset_name>-<replica_number>.

   storage

   In the spec volume claim templates spec resources requests section, enter the size of the block storage in gigabytes (Gi).

   iops

   In the spec volume claim templates spec resources requests section, if you want to provision performance storage, enter the number of IOPS. If you use an endurance storage class and specify a number of IOPS, the number of IOPS is ignored. Instead, the IOPS that is specified in your storage class is used.

   storageClassName

   In the spec volume claim templates spec section, enter the storage class that you want to use. To list existing storage classes, run kubectl get sc | grep block. If you don't specify a storage class, the PVC is created with the default storage class that is set in your cluster. Make sure that the default storage class uses the ibm.io/ibmc-block provisioner so that your stateful set is provisioned with block storage.

4. Create your stateful set.

   kubectl apply -f statefulset.yaml
   

5. Wait for your stateful set to be deployed.

   kubectl describe statefulset <statefulset_name>
   

   To see the current status of your PVCs, run kubectl get pvc. The name of your PVC is formatted as <volume_name>-<statefulset_name>-<replica_number>.

Static provisioning by using existing PVCs with a stateful set

You can pre-provision your PVCs before creating your stateful set or use existing PVCs with your stateful set.

When you dynamically provision your PVCs when creating the stateful set, the name of the PVC is assigned based on the values that you used in the stateful set YAML file. In order for the stateful set to use existing PVCs, the name of your PVCs must match the name that would automatically be created when using dynamic provisioning.

Before you begin: Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

1. If you want to pre-provision the PVC for your stateful set before you create the stateful set, follow steps 1-3 in Adding block storage to apps to create a PVC for each stateful set replica. Make sure that you create your PVC with a name that follows the following format: <volume_name>-<statefulset_name>-<replica_number>.

   volume_name

   Use the name that you want to specify in the spec.volumeClaimTemplates.metadata.name section of your stateful set, such as nginxvol.

   statefulset_name

   Use the name that you want to specify in the metadata.name section of your stateful set, such as nginx_statefulset.

   replica_number

   Enter the number of your replica, starting with 0.

   For example, if you must create three stateful set replicas, create three PVCs with the following names: nginxvol-nginx_statefulset-0, nginxvol-nginx_statefulset-1, and nginxvol-nginx_statefulset-2.

   Looking to create a PVC and PV for an existing storage device? Create your PVC and PV by using static provisioning.

2. Follow the steps in Dynamic provisioning: Creating the PVC when you create a stateful set to create your stateful set. The name of your PVC follows the format <volume_name>-<statefulset_name>-<replica_number>. Make sure to use the following values from your PVC name in the stateful set specification: spec.volumeClaimTemplates.metadata.name : Enter the <volume_name> of your PVC name.

   metadata.name

   Enter the <statefulset_name> of your PVC name.

   spec.replicas

   Enter the number of replicas that you want to create for your stateful set. The number of replicas must equal the number of PVCs that you created earlier.

   If your PVCs are in different zones, don't include a region or zone label in your stateful set.

3. Verify that the PVCs are used in your stateful set replica pods by listing the pods in your cluster. Identify the pods that belong to your stateful set.

   kubectl get pods
   

4. Verify that your existing PVC is mounted to your stateful set replica. Review the ClaimName in the Volumes section of your CLI output.

   kubectl describe pod <pod_name>
   

   Example output

   Name:           nginx-0
   Namespace:      default
   Node:           10.xxx.xx.xxx/10.xxx.xx.xxx
   Start Time:     Fri, 05 Oct 2022 13:24:59 -0400
   ...
   Volumes:
   myvol:
     Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
     ClaimName:  myvol-nginx-0
   ...
   

Setting up periodic snapshots

You can set up periodic snapshots for your block storage, which is a read-only image that captures the state of the instance at a point in time.

To store the snapshot, you must request snapshot space on your block storage. Snapshots are stored on the existing storage instance within the same zone. You can restore data from a snapshot if a user accidentally removes important data from the volume. \n \n **To create a snapshot for your volume, complete the following steps.

1. Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

2. Log in to the ibmcloud sl CLI.

   ibmcloud sl init
   

3. List existing PVs in your cluster.

   kubectl get pv
   

4. Get the details for the PV for which you want to create snapshot space and note the volume ID, the size, and the IOPS. The size and IOPS are shown in the Labels section of your CLI output.

   kubectl describe pv <pv_name>
   

5. To find the volume ID, review the ibm.io/network-storage-id annotation of your CLI output.

6. Create the snapshot size for your existing volume with the parameters that you retrieved in the previous step.

   ibmcloud sl block snapshot-order <volume_ID> --size <size> --tier <iops>
   

7. Wait for the snapshot size to create. The snapshot size is successfully provisioned when the Snapshot Size (GB) in your CLI output changes from 0 to the size that you ordered.

   ibmcloud sl block volume-detail <volume_ID>
   

8. Create the snapshot for your volume and note the ID of the snapshot that is created for you.

   ibmcloud sl block snapshot-create <volume_ID>
   

9. Verify that the snapshot is created successfully.

   ibmcloud sl block snapshot-list <volume_ID>
   

10. Set the snapshot schedule. For more information on the options available for your snapshot schedule, refer to the CLI documentation.

    ibmcloud sl block snapshot-enable VOLUME_ID <OPTIONS>
    

11. To restore data from a snapshot to an existing volume, run the following command.

    ibmcloud sl block snapshot-restore <volume_ID> <snapshot_ID>
    

Replicating snapshots to another zone

To protect your data from a zone failure, you can replicate snapshots to a block storage instance that is set up in another zone.

Data can be replicated from the primary storage to the backup storage only. You can't mount a replicated block storage instance to a cluster. When your primary storage fails, you can manually set your replicated backup storage to be the primary one. Then, you can mount it to your cluster. After your primary storage is restored, you can restore the data from the backup storage.

Duplicating storage

You can duplicate your block storage instance in the same zone as the original storage instance.

A duplicate has the same data as the original storage instance at the point in time that you create the duplicate. Unlike replicas, use the duplicate as an independent storage instance from the original. To duplicate, first set up snapshots for the volume.

Backing up data to IBM Cloud® Object Storage

You can use the ibm-backup-restore Helm chart to spin up a backup and restore pod in your cluster.

This pod contains a script to run a one-time or periodic backup for any persistent volume claim (PVC) in your cluster. Data is stored in your IBM Cloud® Object Storage instance that you set up in a zone.

Block storage is mounted with an RWO access mode. This access allows only one pod to be mounted to the block storage at a time. To back up your data, you must unmount the app pod from the storage, mount it to your backup pod, back up the data, and remount the storage to your app pod.

To make your data even more highly available and protect your app from a zone failure, set up a second Object Storage instance and replicate data across zones. If you need to restore data from your Object Storage instance, use the restore pod that is provided with the Helm chart.

Copying data to and from pods and containers

You can use the kubectl cp command to copy files and directories to and from pods or specific containers in your cluster.

Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

When you run the kubectl cp command, if you don't specify a container with -c, the command uses the first available container in the pod.

Copy data from your local machine to a pod in your cluster.

kubectl cp <local_filepath>/<filename> <namespace>/<pod>:<pod_filepath>

Copy data from a pod in your cluster to your local machine.

kubectl cp <namespace>/<pod>:<pod_filepath>/<filename> <local_filepath>/<filename>

Copy data from your local machine to a specific container that runs in a pod in your cluster.

kubectl cp <local_filepath>/<filename> <namespace>/<pod>:<pod_filepath> -c CONTAINER

Bronze

Name

ibmc-block-bronze

ibmc-block-retain-bronze

Type

Endurance storage

File system

ext4

IOPS per gigabyte

2

Size range in gigabytes

20-12000 Gi

Hard disk

SSD

Reclaim policy

ibmc-block-bronze: Delete

ibmc-block-retain-bronze: Retain

Silver

Name

ibmc-block-silver

ibmc-block-retain-silver

Type

Endurance storage

File system

ext4

IOPS per gigabyte

4

Size range in gigabytes

20-12000 Gi

Hard disk

SSD

Reclaim policy

ibmc-block-silver: Delete

ibmc-block-retain-silver: Retain

Gold

Name

ibmc-block-gold

ibmc-block-retain-gold

Type

Endurance storage

File system

ext4

IOPS per gigabyte

10

Size range in gigabytes

20-4000 Gi

Hard disk

SSD

Reclaim policy

ibmc-block-gold: Delete

ibmc-block-retain-gold: Retain

Custom

Name

ibmc-block-custom

ibmc-block-retain-custom

Type

Performance File system

ext4

IOPS and size

Size range in gigabytes / IOPS range in multiples of 100

• 20-39 Gi / 100-1000 IOPS
• 40-79 Gi / 100-2000 IOPS
• 80-99 Gi / 100-4000 IOPS
• 100-499 Gi / 100-6000 IOPS
• 500-999 Gi / 100-10000 IOPS
• 1000-1999 Gi / 100-20000 IOPS
• 2000-2999 Gi / 200-40000 IOPS
• 3000-3999 Gi / 200-48000 IOPS
• 4000-7999 Gi / 300-48000 IOPS
• 8000-9999 Gi / 500-48000 IOPS
• 10000-12000 Gi / 1000-48000 IOPS

Hard disk

The IOPS to gigabyte ratio determines the type of hard disk that is provisioned. To determine your IOPS to gigabyte ratio, you divide the IOPS by the size of your storage.

Example: You chose 500Gi of storage with 100 IOPS. Your ratio is 0.2 (100 IOPS/500Gi).

Overview of hard disk types per ratio:

• Less than or equal to 0.3: SATA
• Greater than 0.3: SSD

Reclaim policy

ibmc-block-custom: Delete

ibmc-block-retain-custom: Retain

Creating topology-aware storage

To use block storage in a multizone cluster, your pod must be scheduled in the same zone as your block storage instance so that you can read and write to the volume. Before topology-aware volume scheduling was introduced by Kubernetes, the dynamic provisioning of your storage automatically created the block storage instance when a PVC was created. Then, when you created your pod, the Kubernetes scheduler tried to deploy the pod to the same data center as your block storage instance.

Creating the block storage instance without knowing the constraints of the pod can lead to unwanted results. For example, your pod might not be able to be scheduled to the same worker node as your storage because the worker node has insufficient resources or the worker node is tainted and does not allow the pod to be scheduled. With topology-aware volume scheduling, the block storage instance is delayed until the first pod that uses the storage is created.

To use topology-aware volume scheduling, make sure that you installed the IBM Cloud Block Storage plug-in version 1.2.0 or later.

The following examples show how to create storage classes that delay the creation of the block storage instance until the first pod that uses this storage is ready to be scheduled. To delay the creation, you must include the volumeBindingMode: WaitForFirstConsumer option. If you don't include this option, the volumeBindingMode is automatically set to Immediate and the block storage instance is created when you create the PVC.

Example for Endurance block storage.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ibmc-block-bronze-delayed
  parameters:
    billingType: hourly
    classVersion: "2"
    fsType: ext4
    iopsPerGB: "2"
    sizeRange: '[20-12000]Gi'
    type: Endurance
  provisioner: ibm.io/ibmc-block
  reclaimPolicy: Delete
  volumeBindingMode: WaitForFirstConsumer

Example for Performance block storage.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ibmc-block-performance-storageclass
  labels:
  kubernetes.io/cluster-service: "true"
  provisioner: ibm.io/ibmc-block
  parameters:
    billingType: "hourly"
    classVersion: "2"
    sizeIOPSRange: |-
    "[20-39]Gi:[100-1000]"
    "[40-79]Gi:[100-2000]"
    "[80-99]Gi:[100-4000]"
    "[100-499]Gi:[100-6000]"
    "[500-999]Gi:[100-10000]"
    "[1000-1999]Gi:[100-20000]"
    "[2000-2999]Gi:[200-40000]"
    "[3000-3999]Gi:[200-48000]"
    "[4000-7999]Gi:[300-48000]"
    "[8000-9999]Gi:[500-48000]"
    "[10000-12000]Gi:[1000-48000]"
    type: "Performance"
  reclaimPolicy: Delete
  volumeBindingMode: WaitForFirstConsumer

Specifying the zone and region

If you want to create your block storage in a specific zone, you can specify the zone and region in a customized storage class.

Use the customized storage class if you use the IBM Cloud Block Storage plug-in version 1.0.0 or if you want to statically provision block storage in a specific zone. In all other cases, specify the zone directly in your PVC.

The following .yaml file customizes a storage class that is based on the ibm-block-silver non-retaining storage class: the type is "Endurance", the iopsPerGB is 4, the sizeRange is "[20-12000]Gi", and the reclaimPolicy is set to "Delete". The zone is specified as dal12. To use a different storage class as your base, see the storage class reference.

Create the storage class in the same region and zone as your cluster and worker nodes. To get the region of your cluster, run ibmcloud ks cluster get --cluster <cluster_name_or_ID> and look for the region prefix in the Master URL, such as eu-de in https://c2.eu-de.containers.cloud.ibm.com:11111. To get the zone of your worker node, run ibmcloud ks worker ls --cluster <cluster_name_or_ID>.

Example for Endurance block storage.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ibmc-block-silver-mycustom-storageclass
  labels:
    kubernetes.io/cluster-service: "true"
provisioner: ibm.io/ibmc-block
parameters:
  zone: "dal12"
  region: "us-south"
  type: "Endurance"
  iopsPerGB: "4"
  sizeRange: "[20-12000]Gi"
reclaimPolicy: "Delete"

Example for Performance block storage.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
    name: ibmc-block-performance-storageclass
labels:
  kubernetes.io/cluster-service: "true"
provisioner: ibm.io/ibmc-block
parameters:
  zone: "dal12"
  region: "us-south"
  type: "Performance"
  sizeIOPSRange: |-
  "[20-39]Gi:[100-1000]"
  "[40-79]Gi:[100-2000]"
  "[80-99]Gi:[100-4000]"
  "[100-499]Gi:[100-6000]"
  "[500-999]Gi:[100-10000]"
  "[1000-1999]Gi:[100-20000]"
  "[2000-2999]Gi:[200-40000]"
  "[3000-3999]Gi:[200-48000]"
  "[4000-7999]Gi:[300-48000]"
  "[8000-9999]Gi:[500-48000]"
  "[10000-12000]Gi:[1000-48000]"
reclaimPolicy: "Delete"

Mounting block storage with an XFS file system

The following examples create a storage class that provisions block storage with an XFS file system.

Example for Endurance block storage.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ibmc-block-custom-xfs
labels:
  addonmanager.kubernetes.io/mode: Reconcile
provisioner: ibm.io/ibmc-block
parameters:
  type: "Endurance"
  iopsPerGB: "4"
  sizeRange: "[20-12000]Gi"
  fsType: "xfs"
reclaimPolicy: "Delete"

Example for Performance block storage.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ibmc-block-custom-xfs
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
provisioner: ibm.io/ibmc-block
parameters:
  classVersion: "2"
  type: "Performance"
  sizeIOPSRange: |-
    [20-39]Gi:[100-1000]
    [40-79]Gi:[100-2000]
    [80-99]Gi:[100-4000]
    [100-499]Gi:[100-6000]
    [500-999]Gi:[100-10000]
    [1000-1999]Gi:[100-20000]
    [2000-2999]Gi:[200-40000]
    [3000-3999]Gi:[200-48000]
    [4000-7999]Gi:[300-48000]
    [8000-9999]Gi:[500-48000]
    [10000-12000]Gi:[1000-48000]
  fsType: "xfs"
reclaimPolicy: "Delete"

Understanding your storage removal options

Removing persistent storage from your IBM Cloud account varies depending on how you provisioned the storage and what components you already removed.

Is my persistent storage deleted when I delete my cluster?

During cluster deletion, you have the option to remove your persistent storage. However, depending on how your storage was provisioned, the removal of your storage might not include all storage components. If you dynamically provisioned storage with a storage class that sets reclaimPolicy: Delete, your PVC, PV, and the storage instance are automatically deleted when you delete the cluster. For storage that was statically provisioned or storage that you provisioned with a storage class that sets reclaimPolicy: Retain, the PVC and the PV are removed when you delete the cluster, but your storage instance and your data remain. You are still charged for your storage instance. Also, if you deleted your cluster in an unhealthy state, the storage might still exist even if you chose to remove it.

How do I delete the storage when I want to keep my cluster?

When you dynamically provisioned the storage with a storage class that sets reclaimPolicy: Delete, you can remove the PVC to start the deletion process of your persistent storage. Your PVC, PV, and storage instance are automatically removed. For storage that was statically provisioned or storage that you provisioned with a storage class that sets reclaimPolicy: Retain, you must manually remove the PVC, PV, and the storage instance to avoid further charges.

How does the billing stop after I delete my storage?

Depending on what storage components you delete and when, the billing cycle might not stop immediately. If you delete the PVC and PV, but not the storage instance in your IBM Cloud account, that instance still exists and you are charged for it.

If you delete the PVC, PV, and the storage instance, the billing cycle stops depending on the billingType that you chose when you provisioned your storage and how you chose to delete the storage.

• When you manually cancel the persistent storage instance from the IBM Cloud console or the CLI, billing stops as follows:

  • Hourly storage: Billing stops immediately. After your storage is canceled, you might still see your storage instance in the console for up to 72 hours.
  • Monthly storage: You can choose between immediate cancellation or cancellation on the anniversary date. In both cases, you are billed until the end of the current billing cycle, and billing stops for the next billing cycle. After your storage is canceled, you might still see your storage instance in the console or the CLI for up to 72 hours.
  • Immediate cancellation: Choose this option to immediately remove your storage. Neither you nor your users can use the storage anymore or recover the data.
  • Anniversary date: Choose this option to cancel your storage on the next anniversary date. Your storage instances remain active until the next anniversary date and you can continue to use them until this date, such as to give your team time to make backups of your data.

• When you dynamically provisioned the storage with a storage class that sets reclaimPolicy: Delete and you choose to remove the PVC, the PV and the storage instance are immediately removed. For hourly billed storage, billing stops immediately. For monthly billed storage, you are still charged for the remainder of the month. After your storage is removed and billing stops, you might still see your storage instance in the console or the CLI for up to 72 hours.

What do I need to be aware of before I delete persistent storage?

When you clean up persistent storage, you delete all the data that is stored in it. If you need a copy of the data, make a backup.

I deleted my storage instance. Why can I still see my instance?

After you remove persistent storage, it can take up to 72 hours for the removal to be fully processed and for the storage to disappear from your IBM Cloud console or CLI.

Cleaning up persistent storage

Remove the PVC, PV, and the storage instance from your IBM Cloud account to avoid further charges for your persistent storage.

Before you begin:

• Make sure that you backed up any data that you want to keep.
• Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

To clean up persistent data:

1. List the PVCs in your cluster and note the NAME of the PVC, the STORAGECLASS, and the name of the PV that is bound to the PVC and shown as VOLUME.

   kubectl get pvc
   

   Example output

   NAME                  STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS            AGE
   claim1   Bound     pvc-06886b77-102b-11e8-968a-f6612bb731fb   20Gi       RWO           class       78d
   claim2     Bound     pvc-457a2b96-fafc-11e7-8ff9-b6c8f770356c   4Gi        RWX           class 105d
   claim3      Bound     pvc-1efef0ba-0c48-11e8-968a-f6612bb731fb   24Gi       RWX           class        83d
   

2. Review the ReclaimPolicy and billingType for the storage class.

   kubectl describe storageclass <storageclass_name>
   

   If the reclaim policy says Delete, your PV and the physical storage are removed when you remove the PVC. If the reclaim policy says Retain, or if you provisioned your storage without a storage class, then your PV and physical storage are not removed when you remove the PVC. You must remove the PVC, PV, and the physical storage separately.

   If your storage is charged monthly, you still get charged for the entire month, even if you remove the storage before the end of the billing cycle.

3. Remove any pods that mount the PVC. List the pods that mount the PVC. If no pod is returned in your CLI output, you don't have a pod that uses the PVC.

   kubectl get pods --all-namespaces -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.volumes[*]}{.persistentVolumeClaim.claimName}{" "}{end}{end}' | grep "<pvc_name>"
   

   Example output

   depl-12345-prz7b:    claim1
   

4. Remove the pod that uses the PVC. If the pod is part of a deployment, remove the deployment.

   kubectl delete pod <pod_name>
   

5. Verify that the pod is removed.

   kubectl get pods
   

6. Remove the PVC.

   kubectl delete pvc <pvc_name>
   

7. Review the status of your PV. Use the name of the PV that you retrieved earlier as VOLUME. When you remove the PVC, the PV that is bound to the PVC is released. Depending on how you provisioned your storage, your PV goes into a Deleting state if the PV is deleted automatically, or into a Released state, if you must manually delete the PV. Note: For PVs that are automatically deleted, the status might briefly say Released before it is deleted. Rerun the command after a few minutes to see whether the PV is removed.

   kubectl get pv <pv_name>
   

8. If your PV is not deleted, manually remove the PV.

   kubectl delete pv <pv_name>
   

9. Verify that the PV is removed.

   kubectl get pv
   

10. List the physical storage instance that your PV pointed to and note the id of the physical storage instance.

    ibmcloud sl block volume-list --columns id --columns notes | grep <pv_name>
    

    Example output

    12345678   {"plugin":"ibmcloud-block-storage-plugin-689df949d6-4n9qg","region":"us-south","cluster":"aa1a11a1a11b2b2bb22b22222c3c3333","type":"Endurance","ns":"default","pvc":"block-storage-pvc","pv":"pvc-d979977d-d79d-77d9-9d7d-d7d97ddd99d7","storageclass":"ibmc-block-silver","reclaim":"Delete"}
    

    Understanding the Notes field information:

    "plugin":"ibm-file-plugin-5b55b7b77b-55bb7"

    The storage plug-in that the cluster uses.

    "region":"us-south"

    The region that your cluster is in.

    "cluster":"aa1a11a1a11b2b2bb22b22222c3c3333"

    The cluster ID that is associated with the storage instance.

    "type":"Endurance"

    The type of file or block storage, either Endurance or Performance.

    "ns":"default"

    The namespace that the storage instance is deployed to.

    "pvc":"block-storage-pvc"

    The name of the PVC that is associated with the storage instance.

    "pv":"pvc-d979977d-d79d-77d9-9d7d-d7d97ddd99d7"

    The PV that is associated with the storage instance.

    "storageclass":"ibmc-file-gold"

    The type of storage class: bronze, silver, gold, or custom.

11. Remove the physical storage instance.

    ibmcloud sl block volume-cancel <classic_block_id>
    

12. Verify that the physical storage instance is removed.

The deletion process might take up to 72 hours to complete.

ibmcloud sl block volume-list