Installing the IBM Cloud Object Storage cluster add-on

Prerequisites:

  • The IBM Cloud Object Storage add-on requires at least 0.3 vCPU and 360 MB of memory.
  • The add-on is available for Red Hat CoreOS (RHCOS) and Ubuntu worker nodes. If your cluster has both RHEL and RHCOS nodes, then the add-on is deployed only on the RHCOS nodes.
  • Set up an IBM Cloud Object Storage instance.
  • Optional If you plan to use bucket versioning, your service credentials must have Manager or Writer permissions to enable or disable bucket versioning on the bucket. For more information, see Getting started with versioning.

Understanding bucket creation and removal

  • You can use an existing bucket by specifying the bucket name in your PVC.
  • If you provide a bucket name and that bucket doesn't exist, then a bucket with that name is created.
  • If you don't provide a bucket name, then a bucket with the naming convention temp-xxx is created.
  • Buckets are deleted based on reclaim policy defined in your storage class.
    • If reclaimPolicy: Delete is set, the bucket is deleted when the PVC is deleted.
    • If reclaimPolicy: Retain is set, the bucket is retained even after the PVC is deleted.

Enabling the IBM Cloud Object Storage add-on from the CLI

Before you begin: Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

  1. List the add-ons and find the version you want to install.
    ibmcloud ks cluster addon versions
  2. Review the add-on options.
    ibmcloud ks cluster addon options --addon ibm-object-csi-driver [--version VERSION]
  3. Install the add-on.
    ibmcloud ks cluster addon enable ibm-object-csi-driver --cluster CLUSTER [--version VERSION]
  4. Verify the installation.
    ibmcloud ks cluster addon ls --cluster CLUSTER

    OK
Name                    Version   Health State   Health Status
ibm-object-csi-driver   1.0       normal         Addon Ready. For more info: http://ibm.biz/addon-state (H1500)
  5. List the available storage classes.
    kubectl get sc | grep object

    ibm-object-storage-smart-rclone             cos.s3.csi.ibm.io   Delete          Immediate           false                  17h
ibm-object-storage-smart-rclone-retain      cos.s3.csi.ibm.io   Retain          Immediate           false                  17h
ibm-object-storage-smart-s3fs               cos.s3.csi.ibm.io   Delete          Immediate           false                  17h
ibm-object-storage-smart-s3fs-retain        cos.s3.csi.ibm.io   Retain          Immediate           false                  17h
ibm-object-storage-standard-rclone          cos.s3.csi.ibm.io   Delete          Immediate           false                  17h
ibm-object-storage-standard-rclone-retain   cos.s3.csi.ibm.io   Retain          Immediate           false                  17h
ibm-object-storage-standard-s3fs            cos.s3.csi.ibm.io   Delete          Immediate           false                  17h
ibm-object-storage-standard-s3fs-retain     cos.s3.csi.ibm.io   Retain          Immediate           false                  17h

Deploying an app that uses IBM Cloud Object Storage

Create a Kubernetes secret that contains your COS credentials.

  1. Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

  2. Save the following configuration as a file called secret.yaml. Provide either IAM credentials or HMAC, but not both.

    • For IAM credentials, use a combination of apiKey and serviceId from Object Storage.
    • For HMAC credentials, use accessKey and secretKey from Object Storage.
    apiVersion: v1
kind: Secret
type: cos-s3-csi-driver
metadata:
    name: cos-secret-1 # Name your secret. This same name is used for the PVC in the following steps.
    namespace: <namespace> # Specify the namespace where you want to create the secret.
data:
    apiKey: <base64-encoded-COS-Service-Instance-apikey>
    serviceID: <base64-encoded-COS-resource_instance_id>
    accessKey: <base64-encoded-HMAC-access_key_id>
    secretKey: <base64-encoded-HMAC-secret_access_key>
    kp-root-key-crn: <CRN> # Key Protect or HPCS root key crn in base64 encoded format
stringData:
    bucketName: <bucket-name> # Optional. If you don't provide a bucket name, a bucket with the naming convention s3fs-timestamp-xxx or rclone-timestamp-xxx is created.
    bucketVersioning: false # Bucket versioning is set to false by default. Set to true to enable bucket versioning. Set to false to disable versioning for a bucket where versioning is enabled.
# uid: "3000" # Optional: Provide a uid to run as non root user. This must match runAsUser in SecurityContext of pod spec.
mountOptions: |
    # Review or update the following default s3fs mount options
    #multipart_size=52
    #multireq_max=20
    #max_dirty_data=5120
    #parallel_count=20
    #max_stat_cache_size=100000
    #retries=5
    #kernel_cache
    
    # Review or update the following default rclone mount options
    #acl=private
    #bucket_acl=private
    #upload_cutoff=100Mi
    #chunk_size=16Mi
    #max_upload_parts=1000
    #upload_concurrency=8
    mountOptions
    You can customize the mount options for either s3fs or rclone by editing the mountOptions in your secret. Align the options that you specify with the storage class that your PVC uses. To review the default values for a storage class, run oc describe storageclass <storageclass_name> or kubectl describe storageclass <storageclass_name>. For more information, see the s3fs mount options and the rclone mount options.

    Currently, the add-on is enabled to support a fixed set of mount options with proper validation for each mount option. If you want to use any other mount options that are not in the validation list, contact support to enable those options.

  3. Encode all the secret data parameters to base64.

    echo -n "<value>" | base64

  4. Update the secret.yaml with the base64 encoded values.

  5. Create the secret.

    kubectl apply -f secret.yaml

Create a PVC

You can either use a single secret across multiple PVCs or one secret per PVC.

You can manage this behavior by using the following annotations in the PVC yaml. These annotations help the driver map the PVC to the correct secret.

cos.csi.driver/secret: "<custom-secret>"

Make sure that your secret, PVC, and pods are all in the same namespace

Example PVC for a 1-to-1 secret to PVC mapping by giving your PVC the same name as the secret you created earlier.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cos-secret-1 # Give your PVC the same name as the secret you created in the previous step.
  namespace: <namespace> # The namespace where you want to create the PVC.
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: <storage_class_name> # The storage class you want to use.

Example PVC for using 1 secret to many PVCs by using annotations to specify the secret.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cos-csi-pvc1
  namespace: <namespace> # The namespace where you want to create the PVC.
  annotations:
    cos.csi.driver/secret: "<custom-secret>"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi
  storageClassName: <storage_class_name> # The storage class you want to use.

  1. Choose one of the previous examples and customize it for your use case. For a list of storage classes, see the Storage class reference.

  2. Create the PVC.

    kubectl apply -f pvc.yaml

Create a deployment

  1. Save the following configuration to a file called dep.yaml.

    apiVersion: apps/v1
kind: Deployment
metadata:
  name: <name>
  labels:
    app: <name>
spec:
  replicas: 1
  selector:
    matchLabels:
      app: <name>
  template:
    metadata:
      labels:
    app: <name>
    spec:
      containers:
      - name: app-frontend
        image: <image> # Enter your app image.
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: <path_you_want_to_mount_the_volume_on> # For example `/dev`
          name: cos-csi-volume
      volumes:
      - name: cos-csi-volume
        persistentVolumeClaim:
          claimName: <pvc_name> # Enter the name of the PVC you created earlier.

  2. Create the deployment.

    kubectl apply -f dep.yaml

Disabling the IBM Cloud Object Storage add-on

The existing secrets, PVCs, and deployments are not deleted by disabling the add-on or by patch updates. There are no disruptions to existing customer workloads.

  1. Run the following command to disable the add-on.
     ibmcloud ks cluster addon disable ibm-object-csi-driver --cluster CLUSTER
    Example output
    Data and resources that you created for the add-on might be deleted when the add-on is disabled. Continue? [y/N]> y
Disabling add-on ibm-object-csi-driver for cluster XXX...
OK
  2. Verify the add-on was removed.
    ibmcloud ks cluster addon ls --cluster CLUSTER

Migrating from the Helm plug-in to the cluster add-on

  1. Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

  2. Get the details of your PVCs and select one to migrate.

    kubectl get pvc --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' | tail -n +2 | while read namespace pvc; do kubectl describe pvc "$pvc" -n "$namespace" | grep 'volume.kubernetes.io/storage-provisioner: ibm.io/ibmc-s3fs' > /dev/null ; if [ $? -eq 0 ]; then echo "PVC: $pvc in Namespace: $namespace uses ibm.io/ibmc-s3fs storage provisioner"; fi; done

    Example output

    PVC: pvc-test in Namespace: default uses ibm.io/ibmc-s3fs storage provisioner

  3. Describe the PVC and get the bucket name.

    kubectl describe pvc <pvc_name> | grep ibm.io/bucket:

    Example output

    ibm.io/bucket: test-s3

  4. Re-create your secret with the bucket name included.

    apiVersion: v1
kind: Secret
type: cos-s3-csi-driver
metadata:
    name: cos-secret-1 # Name your secret.
    namespace: <namespace> # Specify the namespace where you want to create the secret.
data:
    accessKey: <base64-encoded-HMAC-access-key>
    secretKey: <base64-encoded-HMAC-secret-key>
stringData:
    bucketName: <bucket-name>
# uid: "3000" # Optional: Provide a uid to run as non root user. This must match runAsUser in SecurityContext of pod spec.
mountOptions: |
    key1=value1
    key2=value2

  5. Find the storage class that was used in your PVC.

    kubectl describe pvc <pvc_name> | grep StorageClass:

    Example command for a PVC called test-s3.

    kubectl describe pvc test-s3 | grep StorageClass:

    Example output

    StorageClass:  ibmc-s3fs-smart-perf-regional

  6. Review the new storage classes that are available with the add-on and select a replacement class.

    • If you used a flex class, choose one of the new smart classes.
    • If you used a standard classes, choose one of the new standard classes.
    • The cold and vault classes are no longer available with the add-on; choose a smart or standard class instead.

  7. Review the details of your PVC.

    kubectl describe pvc test-s3

    Example output

    Name:          pvc-test
Namespace:     default
StorageClass:  ibmc-s3fs-smart-perf-regional
Status:        Bound
Volume:        pvc-c625474d-31f0-4929-bc3e-feace1fb42fb
Labels:        <none>
Annotations:   ibm.io/auto-create-bucket: true
            ibm.io/auto-delete-bucket: true
            ibm.io/bucket: bha-test-s23
            ibm.io/secret-name: satstoragesecret
            pv.kubernetes.io/bind-completed: yes
            pv.kubernetes.io/bound-by-controller: yes
            volume.kubernetes.io/storage-provisioner: ibm.io/ibmc-s3fs
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      3Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       test-pod
Events:        <none>

  8. Create a replacement PVC that uses a new storage class and references the secret you created earlier.

    apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: cos-csi-pvc1
namespace: <namespace> # The namespace where you want to create the PVC.
annotations:
    cos.csi.driver/secret: "cos-secret-1"  # Secret created in step 4
spec:
accessModes:
- ReadWriteOnce
resources:
    requests:
    storage: 256Mi
storageClassName: <storage_class_name> # The storage class you picked based on old storage class mapping.

  9. Verify the PVC is Bound.

    kubectl get pvc

  10. Get the details of your app.

    kubectl get pods

  11. Scale down your app to zero.

    kubectl scale deployment --replicas=0 my-app

  12. Create a replacement deployment that references the PVC you created in the previous step.

  13. After the new deployment is running, you can delete the old deployment.

  14. Repeat these steps for each PVC that you want to migrate.

IBM Cloud Object Storage cluster add-on storage classes

The IBM Cloud Object Storage cluster add-on provides storage classes for the s3fs and rclone mounters. Choose a storage class that fits your data access requirements. The storage class determines the bucket class, reclaim policy, and default mount behavior for the bucket that is created for your workload.

Standard
Use for hot data that you access frequently, such as data for web or mobile apps.
Smart
Use for workloads and data that do not follow a specific usage pattern, or when the usage pattern is difficult to predict.
COS cluster add-on storage classes
Name Bucket class Mounter Reclaim policy Binding mode
ibm-object-storage-smart-rclone Smart rclone Delete Immediate
ibm-object-storage-smart-rclone-retain Smart rclone Retain Immediate
ibm-object-storage-smart-s3fs Smart s3fs Delete Immediate
ibm-object-storage-smart-s3fs-retain Smart s3fs Retain Immediate
ibm-object-storage-standard-rclone Standard rclone Delete Immediate
ibm-object-storage-standard-rclone-retain Standard rclone Retain Immediate
ibm-object-storage-standard-s3fs Standard s3fs Delete Immediate
ibm-object-storage-standard-s3fs-retain Standard s3fs Retain Immediate

To review the detailed bucket configuration for a storage class, run oc describe storageclass <storageclass_name> or kubectl describe storageclass <storageclass_name>.

Storage class parameters

All cluster add-on storage classes include the following core parameters.

Core parameters for COS cluster add-on storage classes
Parameter Description
client Identifies the client type that the driver uses. The add-on storage classes use awss3.
cosEndpoint Defines the IBM Cloud Object Storage endpoint for the bucket region.
csi.storage.k8s.io/node-publish-secret-name References the name of the secret that contains your IBM Cloud Object Storage credentials.
csi.storage.k8s.io/node-publish-secret-namespace References the namespace of the secret that contains your IBM Cloud Object Storage credentials.
locationConstraint Defines the bucket class and region, such as au-syd-smart or au-syd-standard.
mounter Specifies whether the storage class uses the s3fs or rclone mounter.

Default s3fs storage class mount options

The s3fs storage classes use the following default mount options.

Mount option Description
multipart_size=52 Sets the part size, in MB, for each multipart request.
multireq_max=20 Sets the maximum number of parallel requests for listing objects.
max_dirty_data=5120 Flushes dirty data to S3 after a specified number of MB are written. The minimum supported value is 50. A value of -1 disables this behavior.
parallel_count=20 Sets the number of parallel requests for uploading large objects. s3fs uploads large objects by using multipart requests and sends requests in parallel.
max_stat_cache_size=100000 Sets the maximum number of entries in the stat cache and symbolic link cache.
retries=5 Sets the number of times to retry a failed S3 transaction.
kernel_cache Enables the kernel buffer cache for the volume mount point. Data that is read from IBM Cloud Object Storage is stored in the kernel cache to help provide faster read access. Kernel cache is enabled for the standard and smart s3fs storage classes.
{: caption="Default mount options for COS add-on s3fs storage classes" caption-side="bottom"}

Default rclone storage class mount options

The rclone storage classes use the following default mount options.

Mount option Description
acl=private Ensures that uploaded objects are not publicly accessible.
bucket_acl=private Sets the default ACL for buckets that rclone creates to private.
upload_cutoff=100Mi Uploads files larger than 100 MiB by using multipart upload. Smaller files are uploaded in a single request.
chunk_size=16Mi Sets the size of each part in a multipart upload.
max_upload_parts=1000 Sets the maximum number of parts per multipart upload and indirectly caps the maximum supported file size with the configured chunk_size. With chunk_size=16Mi, the maximum file size is 16 GiB.
upload_concurrency=8 Sets the number of parts that are uploaded in parallel during a multipart upload.
{: caption="Default mount options for COS add-on rclone storage classes" caption-side="bottom"}