IBM Cloud Docs
Subscribing to Object Storage events

Subscribing to Object Storage events

With this tutorial, you can learn how to subscribe to Object Storage events by using the IBM Cloud® Code Engine CLI.

Oftentimes in distributed environments you want your applications or jobs to react to messages (events) that are generated from other components, which are usually called event producers. With Code Engine, your applications or jobs can receive events of interest by subscribing to event producers. Event information is received as POST HTTP requests for applications and as environment variables for jobs.

Before you begin

All Code Engine users are required to have a Pay-as-you-Go account. Tutorials might incur costs. Use the Cost Estimator to generate a cost estimate based on your projected usage. For more information, see Code Engine pricing.

Determine your Object Storage bucket and region

The Object Storage event producer generates events based on operations on objects in IBM Cloud Object Storage buckets.

  1. Install the Object Storage plug-in CLI.

    ibmcloud plugin install cloud-object-storage

  2. Create an Object Storage resource instance. For example, create an Object Storage resource that is named mycloud-object-storage that uses the IBM Cloud Lite service plan.

    ibmcloud resource service-instance-create mycloud-object-storage cloud-object-storage lite global

  3. Display the details of the Object Storage resource instance that you created. Use the details to get the CRN (Cloud Resource Name) from your Object Storage instance. The CRN identifies which Object Storage instance you want to use. The CRN is the value of the ID field in the output of the ibmcloud resource service-instance COS_INSTANCE_NAME command.

    ibmcloud resource service-instance mycloud-object-storage

    Example output

    Name:                  mycloud-object-storage
ID:                    crn:v1:bluemix:public:cloud-object-storage:global:a/ab9d57f699655f028880abcd2ccdb524:910b727b-abcd-4a73-abcd-77c68bfeabcd::
GUID:                  910b727b-abcd-4a73-abcd-77c68bfeabcd
Location:              global
Service Name:          cloud-object-storage
Service Plan Name:     lite
Resource Group Name:   Default
State:                 active
Type:                  service_instance
Sub Type:
Created at:            2020-10-14T19:09:22Z
Created by:            user@us.ibm.com
Updated at:            2020-10-14T19:09:22Z
[...]

    If you do not know your Object Storage instance name, run ibmcloud resource service-instances --service-name cloud-object-storage to see a list of Object Storage instances.

    For more information about Object Storage instances, see Getting started with IBM Cloud Object Storage.

  4. Configure your Object Storage CRN that you found with the previous step to specify an Object Storage instance to work with. Be sure to copy the entire ID, starting with crn:. This examples uses the --force option to force the configuration to use the specified CRN, which might be helpful if you have more than one Object Storage instance.

    ibmcloud cos config crn --crn CRN --force

    Example output

    Saving new Service Instance ID...
OK
Successfully stored your service instance ID.

  5. Identify a bucket to subscribe to. To see a list of buckets that are associated with your Object Storage instance,

    ibmcloud cos buckets

    To create a bucket,

    ibmcloud cos bucket-create -bucket BUCKET_NAME

  6. Identify the location and plan of the Object Storage bucket; for example use the mybucket bucket.

    ibmcloud cos bucket-location-get --bucket mybucket

    Example output

    Details about bucket mybucket:
Region: us-south
Class: Standard

Your Object Storage bucket must be a regional bucket that is located in the same region as your Code Engine project.

Assigning the Notifications Manager role to Code Engine

Before you can create an Object Storage subscription, you must assign the Notifications Manager role to a Code Engine project. As a Notifications Manager, Code Engine can view, modify, and delete notifications for an Object Storage bucket.

Only account administrators can assign the Notifications Manager role.

  1. Identify the Code Engine project that you want to use. You can use the ibmcloud ce project list command to display a list of projects. Use the ibmcloud ce project select command to select your project as the current context. For example, to select a project named myproject

    ibmcloud ce project select -n myproject

  2. Assign the Notification Manager role by using the ibmcloud iam authorization-policy-create command.

    For example, to assign the Notifications Manager role to a project named myproject for an Object Storage instance named mycosinstance,

    ibmcloud iam authorization-policy-create codeengine cloud-object-storage "Notifications Manager" --source-service-instance-name PROJECT --target-service-instance-name COS-INSTANCE

    After you assign the Notifications Manager role to your project, you can then create Object Storage subscriptions for any regional buckets in your Object Storage instance that are in the same region as your project.

    The following table summarizes the options that are used with the iam authorization-policy-create command in this example. For more information about the command and its options, see the ibmcloud iam authorization-policy-create command.

    iam authorization-policy-create command components
    Command option Description
    codeengine The source service that can be authorized to access.
    cloud-object-storage The target service that the source service can be authorized to access.
    Notifications Manager The roles that provide access for the source service.
    source-service-instance-name The name of the codeengine project that you want to authorize to access.
    target-service-instance-name The name of the cloud-object-storage instance that you want to access.

  3. Verify that the Notifications Manager role is set.

    ibmcloud iam authorization-policies

    Example output

    ID:                        abcd1234-a123-b456-bdd9-849e337c4460
Source service name:       codeengine
Source service instance:   1234abcd-b456-c789-a7c5-ef82e56fb24c
Target service name:       cloud-object-storage
Target service instance:   a1b2c3d4-cbad-567a-8cea-77c68bfe97c9
Roles:                     Notifications Manager

Create your app (or job)

While events can be used to trigger apps or jobs, this tutorial uses an app.

Create an application that is named cos-app with the ibmcloud ce app create command by using an image that is called cos-listen. This app logs each event as it arrives. This image is built from cos-listen.go, available from the Samples for IBM Cloud Code Engine GitHub repo.

ibmcloud ce app create --name cos-app --image icr.io/codeengine/cos-listen

Run ibmcloud ce application get --name cos-app to verify that your app is in a Ready state. The application is in a ready state if the status summary reflects that the application deployed successfully.

For more information about this app, see the IBM Cloud Object Storage readme file.

Create a subscription

After your app is ready, you can create an Object Storage subscription so you can start receiving Object Storage events with the ibmcloud ce sub cos create command.

For example, create an Object Storage subscription that is called cos-sub. This subscription forwards any type of bucket operation from the mybucket bucket to an application that is called cos-app.

ibmcloud ce sub cos create --name cos-sub --destination cos-app --bucket mybucket --event-type all

Run the ibmcloud ce sub cos get -n cos-sub command to find information about your subscription.

Example output

By default, the ibmcloud ce sub cos get command returns two parts. The first part includes Object Storage subscription-related information such as subscription name, destination, prefix, suffix, and event type. The second part includes resource-related event information about the Object Storage subscription that can be used for debugging purposes. By default, event information is available for 1 hour after it occurs.

Getting COS event subscription 'cos-sub'...
OK
Name:          cos-sub
ID:            abcdefgh-abcd-abcd-abcd-1a2b3c4d5e6f
Project Name:  myproject
Project ID:    01234567-abcd-abcd-abcd-abcdabcd1111
Age:           4m16s
Created:       2021-02-01T13:11:31-05:00
Destination:  App:cos-app
Bucket:       mybucket
EventType:    all
Ready:        true

Conditions:
    Type            OK    Age  Reason
    CosConfigured   true  38s
    Ready           true  38s
    ReadyForEvents  true  38s
    SinkProvided    true  38s

Events:
    Type    Reason          Age  Source                Messages
    Normal  CosSourceReady  39s  cossource-controller  CosSource is ready

By default, the subscription cos create command first checks to see whether the destination application exists. If the destination check fails because the app name that you provided does not exist in your project, the subscription cos create command returns an error. If you want to create a subscription without first creating the application, use the --force option. By using the --force option, the command bypasses the destination check. Note that the Ready field of the subscription shows false until the destination app is created. Then, the subscription moves to a Ready: true state automatically.

After the subscription is created, but before the subscription cos create command reports any results, the subscription cos create command repeatedly polls the subscription for its status to verify its readiness. This continuous polling for status lasts for 15 seconds by default before it times out. If the subscription status returns as Ready:true, it reports success, otherwise it reports an error. You can change the amount of time that the subscription cos create command waits before it times out by using the --wait-timeout option. You can also bypass the status polling by setting the --no-wait option to false.

For more information about headers and body, see HTTP headers and body information for events.

Note that subscriptions can affect how an application scales. For more information, see Configuring application scaling.

Testing your subscription

  1. Upload a .txt file to your bucket. For example, you can use the ibmcloud cos object-put command to upload the sample.txt object to a bucket with sample as the value for --key.

    ibmcloud cos object-put --bucket mybucket --key sample --body sample.txt

  2. View the processed event by using the ibmcloud ce app logs command.

    ibmcloud ce app logs --name cos-app

    Example output

    This command returns log files that include information about the event that was forwarded to your destination app. From the following output, you can see that a Write operation was performed on the sample object in the bucket named mybucket.

    Body: {"bucket":"mybucket","endpoint":"","key":"sample","notification":{"bucket_name":"mybucket","content_type":"text/plain","event_type":"Object:Write","format":"2.0","object_length":"1960","object_name":"sample","request_id":"103dd6f7-dd7b-4f49-86db-c2ff4b678b0a","request_time":"2021-02-11T16:57:42.373Z"},"operation":"Object:Write"}

Update your Object Storage subscription

Now you know that your Object Storage subscription was created successfully and that the Object Storage subscription is ready to serve events, you can update the Object Storage subscription with the ibmcloud ce sub cos update command. For example, you can change your subscription to run only when specific operations happen on a subset of objects in the bucket.

  1. Update the Object Storage subscription to forward events only when delete operations happen on files with a name prefix of test.

    ibmcloud ce sub cos update --name cos-sub --event-type delete --prefix test

  2. Run the ibmcloud ce sub cos get command to find information about your subscription.

    ibmcloud ce sub cos get --name cos-sub

    Example output

    In this output, you can see that the updated values for Prefix and EventType are displayed.

    Getting COS event subscription 'cos-sub'...
OK
Name:          cos-sub
ID:            abcdefgh-abcd-abcd-abcd-1a2b3c4d5e6f
Project Name:  myproject
Project ID:    01234567-abcd-abcd-abcd-abcdabcd1111
Age:           4m16s
Created:       2021-02-01T13:11:31-05:00
Destination:  App:cos-app
Bucket:       mybucket
EventType:    delete
Prefix:       test
Ready:        true

Conditions:
    Type            OK    Age  Reason
    CosConfigured   true  24m
    Ready           true  24m
    ReadyForEvents  true  24m
    SinkProvided    true  24m

Events:
    Type    Reason          Age               Source                Messages
    Normal  CosSourceReady  9s (x2 over 24m)  cossource-controller  CosSource is ready

  3. Delete an object from your bucket that has the test prefix. For example, delete a file with test2.txt for the name (or key). You can use the ibmcloud cos object-delete command to delete an object from your bucket or use the Object Storage console.

  4. View the processed event by using the ibmcloud ce app logs command.

    ibmcloud ce app logs --name cos-app

    Example output

    This command returns log files that include information about the event that was forwarded to your destination app. From the following output, you can see that a Delete operation was performed on the .txt object in the bucket named mybucket.

    Body: {"bucket":"mybucket","endpoint":"",""key":"test2.txt","notification":{"bucket_name":"mybucket","event_type":"Object:Delete","format":"2.0","object_length":"41","object_name":"test2.txt","request_id":"c1099857-f1f3-4d74-9ac4-8d374582f77d","request_time":"2021-09-15T15:22:01.205Z"},"operation":"Object:Delete"}

Clean up for Object Storage tutorial

Ready to delete your Object Storage subscription and your app? You can use the ibmcloud ce app delete and the ibmcloud ce sub cos delete commands.

To remove your subscription,

ibmcloud ce sub cos delete --name cos-sub

To remove your application,

ibmcloud ce app delete --name cos-app

Ready to delete your Object Storage bucket and service instance? You can use the ibmcloud cos bucket-delete command to remove your bucket. To remove your Object Storage service instance, use the ibmcloud resource service-instance-delete command.