Event Streams

Setting up an Event Streams package outside IBM Cloud

If you want to set up your Event Streams outside of IBM Cloud, you must manually create a package binding for your Event Streams service. You need the Event Streams service credentials and connection information.

Create a package binding that is configured for your Event Streams service.

ibmcloud fn package bind /whisk.system/messaging myMessageHub -p kafka_brokers_sasl "[\"kafka01-prod01.messagehub.services.us-south.bluemix.net:9093\", \"kafka02-prod01.messagehub.services.us-south.bluemix.net:9093\", \"kafka03-prod01.messagehub.services.us-south.bluemix.net:9093\"]" -p user <your Event Streams user> -p password <your Event Streams password> -p kafka_admin_url https://kafka-admin-prod01.messagehub.services.us-south.bluemix.net:443

Creating a trigger for an Event Streams package outside IBM Cloud

If you want to set up your Event Streams outside of IBM Cloud, you must manually create a package binding for your Event Streams service. You need the Event Streams service credentials and connection information.

1. Create a package binding that is configured for your Event Streams service.

   ibmcloud fn package bind /whisk.system/messaging myMessageHub -p kafka_brokers_sasl "
   [\"broker-1-9eyy8dkv3rrj0wdn.kafka.svc01.us-south.eventstreams.cloud.ibm.com:9093\", \"broker-1-9eyy8dkv3rrj0wdn.kafka.svc02.us-south.eventstreams.cloud.ibm.com:9093\", \"broker-1-9eyy8dkv3rrj0wdn.kafka.svc03.us-south.eventstreams.cloud.ibm.com:9093\"]" -p user <your Event Streams user> -p password <your Event Streams password> -p kafka_admin_url https://9eyy8dkv3rrj0wdn.svc01.us-south.eventstreams.cloud.ibm.com
   

2. Now you can create a trigger by using your new package that is fired when new messages are posted to your Event Streams topic.

   ibmcloud fn trigger create MyMessageHubTrigger -f myMessageHub/messageHubFeed -p topic mytopic -p isJSONData true
   

Listening for messages

After a trigger is created, the system monitors the specified topic in your messaging service. When new messages are posted, the trigger is fired.

The payload of that trigger contains a messages field, which is an array of messages that are posted from the last time the trigger is fired. Each message object in the array contains the following fields:

• topic
• partition
• offset
• key
• value

In Kafka terms, the fields are self-evident. However, key has a feature that is called isBinaryKey that allows the key to transmit binary data. Additionally, the value requires special consideration. Fields isJSONData and isBinaryValue are available to handle JSON and binary messages. These fields, isJSONData, and isBinaryValue, cannot be used together.

As an example, if isBinaryKey was set to true when the trigger was created, the key is encoded as a Base64 string when returned from they payload of a fired trigger.

If a key of Some key is posted with isBinaryKey set to true, the trigger payload resembles the following example.

{
    "messages": [
        {
            "partition": 0,
            "key": "U29tZSBrZXk=",
            "offset": 421760,
            "topic": "mytopic",
            "value": "Some value"
        }
    ]
}

If the isJSONData parameter was set to false (or not set at all) when the trigger was created, the value field is the raw value of the posted message. However, if isJSONData was set to true when the trigger was created, the system attempts to parse this value as a JSON object, on a best-effort basis. If parsing is successful, then the value in the trigger payload is the resulting JSON object.

If a message of {"title": "Some string", "amount": 5, "isAwesome": true} is posted with isJSONData set to true, the trigger payload might look something like the following example.

{
    "messages": [
    {
        "partition": 0,
        "key": null,
        "offset": 421760,
        "topic": "mytopic",
        "value": {
          "amount": 5,
          "isAwesome": true,
          "title": "Some string"
        }
    }
    ]
}

However, if the same message content is posted with isJSONData set to false, the trigger payload would look like the following example.

{
    "messages": [
    {
        "partition": 0,
        "key": null,
        "offset": 421761,
        "topic": "mytopic",
        "value": "{\"title\": \"Some string\", \"amount\": 5, \"isAwesome\": true}"
    }
    ]
}

Similar to isJSONData, if isBinaryValue was set to true during trigger creation, the resultant value in the trigger payload is encoded as a Base64 string.

If a value of Some data is posted with isBinaryValue set to true, the trigger payload might look something like the following example.

{
    "messages": [
    {
        "partition": 0,
        "key": null,
        "offset": 421760,
        "topic": "mytopic",
        "value": "U29tZSBkYXRh"
    }
    ]
}

If the same message is posted without isBinaryData set to true, the trigger payload would resemble the following example.

{
    "messages": [
    {
        "partition": 0,
        "key": null,
        "offset": 421760,
        "topic": "mytopic",
        "value": "Some data"
    }
    ]
}

Messages are batched

Notice that the trigger payload contains an array of messages. If these messages are produced to your messaging system quickly, the feed attempts to batch up the posted messages into a single firing of your trigger. Batch processing allows the messages to be posted to your trigger more rapidly and efficiently.

Keep in mind when you are coding actions that are fired by your trigger, that the number of messages in the payload is technically unbounded, but is always greater than 0. See the following example of a batched message (note the change in the offset value).

{
    "messages": [
        {
        "partition": 0,
        "key": null,
        "offset": 100,
        "topic": "mytopic",
        "value": {
            "amount": 5
        }
        },
        {
        "partition": 0,
        "key": null,
        "offset": 101,
        "topic": "mytopic",
        "value": {
            "amount": 1
        }
        },
        {
        "partition": 0,
        "key": null,
        "offset": 102,
        "topic": "mytopic",
        "value": {
            "amount": 999
        }
        }
    ]
}