Understanding bucket creation and removal

• You can use an existing bucket by specifying the bucket name in your PVC.
• If you provide a bucket name and that bucket doesn't exist, then a bucket with that name is created.
• If you don't provide a bucket name, then a bucket with the naming convention s3fs-<timestamp>-xxx or rclone-<timestamp>-xxx is created, based on the mounter type.
• Buckets are deleted based on reclaim policy defined in your storage class.
  • If reclaimPolicy: Delete is set, the bucket is deleted when the PVC is deleted.
  • If reclaimPolicy: Retain is set, the bucket is retained even after the PVC is deleted.

Restricting nodeserver pod scheduling

By default, the COS CSI driver nodeserver pods are scheduled on all nodes in the cluster. You can use the restrictNodeServerScheduling parameter to restrict nodeserver pod scheduling to only the nodes that are labeled with cos.csi.ibm.io/csi-node=true.

You can configure restrictNodeServerScheduling when you enable the add-on, or update it later by patching the ConfigMap.

• To set restrictNodeServerScheduling when you enable the add-on, include the --param flag in the enable command.

  ibmcloud ks cluster addon enable ibm-object-csi-driver --cluster CLUSTER --param "restrictNodeServerScheduling=true"
  

• To update restrictNodeServerScheduling after the add-on is already enabled, follow these steps.

1. List the nodes in your cluster and determine where you want the COS driver pods to run.

   kubectl get nodes
   

   Example output

   NAME            STATUS   ROLES    AGE    VERSION
   10.241.0.11     Ready    <none>   5d2h   v1.35.5+IKS
   10.241.0.12     Ready    <none>   5d2h   v1.35.5+IKS
   10.241.0.13     Ready    <none>   5d2h   v1.35.5+IKS
   10.241.128.10   Ready    <none>   5d2h   v1.35.5+IKS
   10.241.128.11   Ready    <none>   5d2h   v1.35.5+IKS
   10.241.128.9    Ready    <none>   5d2h   v1.35.5+IKS
   10.241.65.12    Ready    <none>   5d2h   v1.35.5+IKS
   10.241.65.13    Ready    <none>   5d2h   v1.35.5+IKS
   10.241.65.14    Ready    <none>   5d2h   v1.35.5+IKS
   

2. Check that nodeserver pods are currently running on all nodes.

   kubectl get pods -n ibm-object-csi-operator -l app.kubernetes.io/component=node -o wide
   

   Example output

   NAME                        READY   STATUS    RESTARTS   AGE    IP              NODE            NOMINATED NODE   READINESS GATES
   ibm-object-csi-node-2pj2j   3/3     Running   0          145m   172.17.14.10    10.241.0.12     <none>           <none>
   ibm-object-csi-node-7bhwh   3/3     Running   0          145m   172.17.1.72     10.241.65.12    <none>           <none>
   ibm-object-csi-node-7l9hc   3/3     Running   0          145m   172.17.17.6     10.241.128.9    <none>           <none>
   ibm-object-csi-node-cxzt7   3/3     Running   0          145m   172.17.39.72    10.241.0.11     <none>           <none>
   ibm-object-csi-node-dw6qs   3/3     Running   0          145m   172.17.46.77    10.241.128.10   <none>           <none>
   ibm-object-csi-node-rpcvr   3/3     Running   0          145m   172.17.32.198   10.241.65.13    <none>           <none>
   ibm-object-csi-node-swqtg   3/3     Running   0          145m   172.17.16.69    10.241.0.13     <none>           <none>
   ibm-object-csi-node-sxbbs   3/3     Running   0          145m   172.17.26.7     10.241.65.14    <none>           <none>
   ibm-object-csi-node-xm8bt   3/3     Running   0          145m   172.17.20.200   10.241.128.11   <none>           <none>
   

3. Label the nodes where you want nodeserver pods to be scheduled.

   kubectl label nodes NODE-NAME-1 NODE-NAME-2 cos.csi.ibm.io/csi-node=true
   

   Example output

   node/10.241.0.11 labeled
   node/10.241.0.12 labeled
   

4. Enable the restriction by updating the ConfigMap.

   kubectl patch cm managed-addon-ibm-object-csi-driver -n kube-system \
     --type merge -p '{"data":{"restrictNodeServerScheduling":"true"}}'
   

   Example output

   configmap/managed-addon-ibm-object-csi-driver patched
   

5. Verify that nodeserver pods are only scheduled on labeled nodes.

   kubectl get pods -n ibm-object-csi-operator -l app.kubernetes.io/component=node -o wide
   

   Example output

   NAME                        READY   STATUS    RESTARTS   AGE    IP             NODE           NOMINATED NODE   READINESS GATES
   ibm-object-csi-node-cxzt7   3/3     Running   0          145m   172.17.39.72   10.241.0.11    <none>           <none>
   ibm-object-csi-node-7bhwh   3/3     Running   0          145m   172.17.1.72    10.241.65.12   <none>           <none>
   

Setting the maximum volumes per node

By default, the COS CSI driver does not limit the number of volumes that can be mounted on a single node. You can use the maxVolumesPerNode parameter to set a maximum number of volumes per node.

You can configure maxVolumesPerNode when you enable the add-on, or update it later by patching the ConfigMap.

• To set maxVolumesPerNode when you enable the add-on, include the --param flag in the enable command.

  ibmcloud ks cluster addon enable ibm-object-csi-driver --cluster CLUSTER --param "maxVolumesPerNode=VALUE"
  

• To update maxVolumesPerNode after the add-on is already enabled, patch the managed add-on ConfigMap.

  kubectl patch cm managed-addon-ibm-object-csi-driver -n kube-system --type merge -p '{"data":{"maxVolumesPerNode":"VALUE"}}'
  

  Example output

  configmap/managed-addon-ibm-object-csi-driver patched
  

Deploying an app that uses IBM Cloud Object Storage

cos.csi.driver/secret: "<custom-secret>"

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cos-secret-1 # Give your PVC the same name as the secret you created in the previous step.
  namespace: <namespace> # The namespace where you want to create the PVC.
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: <storage_class_name> # The storage class you want to use.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cos-csi-pvc1
  namespace: <namespace> # The namespace where you want to create the PVC.
  annotations:
    cos.csi.driver/secret: "<custom-secret>"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi
  storageClassName: <storage_class_name> # The storage class you want to use.

Disabling the IBM Cloud Object Storage add-on

The existing secrets, PVCs, and deployments are not deleted by disabling the add-on or by patch updates. There are no disruptions to existing customer workloads.

1. Run the following command to disable the add-on.

    ibmcloud ks cluster addon disable ibm-object-csi-driver --cluster CLUSTER
   

   Example output

   Data and resources that you created for the add-on might be deleted when the add-on is disabled. Continue? [y/N]> y
   Disabling add-on ibm-object-csi-driver for cluster XXX...
   OK
   

2. Verify the add-on was removed.

   ibmcloud ks cluster addon ls --cluster CLUSTER
   

Migrating from the Helm plug-in to the cluster add-on

1. Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

2. Get the details of your PVCs and select one to migrate.

   kubectl get pvc --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' | tail -n +2 | while read namespace pvc; do kubectl describe pvc "$pvc" -n "$namespace" | grep 'volume.kubernetes.io/storage-provisioner: ibm.io/ibmc-s3fs' > /dev/null ; if [ $? -eq 0 ]; then echo "PVC: $pvc in Namespace: $namespace uses ibm.io/ibmc-s3fs storage provisioner"; fi; done
   

   Example output

   PVC: pvc-test in Namespace: default uses ibm.io/ibmc-s3fs storage provisioner
   

3. Describe the PVC and get the bucket name.

   kubectl describe pvc <pvc_name> | grep ibm.io/bucket:
   

   Example output

   ibm.io/bucket: test-s3
   

4. Re-create your secret with the bucket name included.

   apiVersion: v1
   kind: Secret
   type: cos-s3-csi-driver
   metadata:
       name: cos-secret-1 # Name your secret.
       namespace: <namespace> # Specify the namespace where you want to create the secret.
   data:
       accessKey: <base64-encoded-HMAC-access-key>
       secretKey: <base64-encoded-HMAC-secret-key>
   stringData:
       bucketName: <bucket-name>
       mountOptions: |
           # uid=3000  # Optional: Run as non-root user. Must match runAsUser in SecurityContext of pod spec.
           key1=value1
           key2=value2
   

5. Find the storage class that was used in your PVC.

   kubectl describe pvc <pvc_name> | grep StorageClass:
   

   Example command for a PVC called test-s3.

   kubectl describe pvc test-s3 | grep StorageClass:
   

   Example output

   StorageClass:  ibmc-s3fs-smart-perf-regional
   

6. Review the new storage classes that are available with the add-on and select a replacement class.

   • If you used a flex class, choose one of the new smart classes.
   • If you used a standard classes, choose one of the new standard classes.
   • The cold and vault classes are no longer available with the add-on; choose a smart or standard class instead.

7. Review the details of your PVC.

   kubectl describe pvc test-s3
   

   Example output

   Name:          pvc-test
   Namespace:     default
   StorageClass:  ibmc-s3fs-smart-perf-regional
   Status:        Bound
   Volume:        pvc-c625474d-31f0-4929-bc3e-feace1fb42fb
   Labels:        <none>
   Annotations:   ibm.io/auto-create-bucket: true
               ibm.io/auto-delete-bucket: true
               ibm.io/bucket: bha-test-s23
               ibm.io/secret-name: satstoragesecret
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.kubernetes.io/storage-provisioner: ibm.io/ibmc-s3fs
   Finalizers:    [kubernetes.io/pvc-protection]
   Capacity:      3Gi
   Access Modes:  RWO
   VolumeMode:    Filesystem
   Used By:       test-pod
   Events:        <none>
   

8. Create a replacement PVC that uses a new storage class and references the secret you created earlier.

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
   name: cos-csi-pvc1
   namespace: <namespace> # The namespace where you want to create the PVC.
   annotations:
       cos.csi.driver/secret: "cos-secret-1"  # Secret created in step 4
   spec:
   accessModes:
   - ReadWriteOnce
   resources:
       requests:
       storage: 256Mi
   storageClassName: <storage_class_name> # The storage class you picked based on old storage class mapping.
   

9. Verify the PVC is Bound.

   kubectl get pvc
   

10. Get the details of your app.

    kubectl get pods
    

11. Scale down your app to zero.

    kubectl scale deployment --replicas=0 my-app
    

12. Create a replacement deployment that references the PVC you created in the previous step.

13. After the new deployment is running, you can delete the old deployment.

14. Repeat these steps for each PVC that you want to migrate.

IBM Cloud Object Storage cluster add-on storage classes

The IBM Cloud Object Storage cluster add-on provides storage classes for the s3fs and rclone mounters. Choose a storage class that fits your data access requirements. The storage class determines the bucket class, reclaim policy, and default mount behavior for the bucket that is created for your workload.

Standard

Use for hot data that is accessed frequently. Common use cases are web or mobile apps.

Vault

Use for workloads or cool data that are accessed infrequently, such as once a month or less. Common use cases are archives, short-term data retention, digital asset preservation, tape replacement, and disaster recovery.

Cold

Use for cold data that is rarely accessed (every 90 days or less), or inactive data. Common use cases are archives, long-term backups, historical data that you keep for compliance, or workloads and apps that are rarely accessed.

Smart

Use for workloads and data that do not follow a specific usage pattern, or when the usage pattern is difficult to predict.

Decide on the level of resiliency for the data that is stored in your bucket. For more information, see Regions and endpoints.

Cross-region

Your data is stored across three regions within a geolocation for highest availability. If you have workloads that are distributed across regions, requests are routed to the nearest regional endpoint. The IBM Cloud Object Storage endpoint for the geolocation is automatically set based on the location that your cluster is in. For example, if your cluster is in US South, then your storage classes are configured to use the US GEO endpoint for your buckets. Choose a storage class with cross-region in its name.

Regional

Your data is replicated across multiple zones within one region. If you have workloads that are located in the same region, you see lower latency and better performance than in a cross-regional setup. The regional endpoint is automatically set based on the location that your cluster is in. For example, if your cluster is in US South, then your storage classes are configured to use US South as the regional endpoint for your buckets. Choose a storage class without cross-region in its name.

To review the detailed bucket configuration for a storage class, run oc describe storageclass <storageclass_name> or kubectl describe storageclass <storageclass_name>.