Integrating IBM Cloud Logs with Event Streams

You can use IBM® Event Streams for IBM Cloud® with IBM Cloud Logs to stream and manage log data. IBM® Event Streams for IBM Cloud® is a scalable, managed Apache Kafka service that lets applications send data by creating messages and sending them to a topic. Applications can subscribe to these topics to receive messages, enabling real-time data processing and analytics.

When you update the streaming configuration after it is enabled, the changes are picked automatically. It takes a few minutes for the changes to be applied.

Prereqs

An IBM® Event Streams for IBM Cloud® instance is provisioned in the same account as the IBM Cloud Logs instance from where you plan to stream data.

Check the limitations of the service plans. For more information, see Limits and quotas.
You have permissions to create a topic in the IBM® Event Streams for IBM Cloud® instance.

To create a topic in Event Streams, you must have the manager role for the Event Streams instance. This role includes the messagehub.topic.manage IAM action role that allows an app or user to create or delete topics.
You have permissions to configure streaming in your IBM® Cloud Logs instance.

You must have the manager service role.

Create the service to service authorization

Authorization permissions to send data from an IBM® Cloud Logs instance to a topic that is defined in an Event Streams instance is done by configuring a service to service authorization in IAM in IBM Cloud.

To configure the service to service authorization, see Creating a S2S authorization to work with the Event Streams service.

Create a topic

In your IBM® Event Streams for IBM Cloud® instance, you must define a topic where the IBM Cloud Logs instance stream data.

Complete the following steps to create an Event Streams topic:

Click the Menu icon > Resource list.
Look for the Event Streams instance that you plan to use, and select it.
In the Event Streams instance console, click Topics.
Click Create topic.
Enter a topic name and click Next.
Enter the number of partitions and click Next.

One or more partitions make up a topic. A partition is an ordered list of messages. Partitions are distributed across the brokers in order to increase the scalability of your topic. You can also use them to distribute messages across the members of a consumer group.
Select a Message retention.

Message retention defines how long messages are retained before they are deleted. If your messages are not read by a consumer within this time, they will be missed.
Click Create Topic.

Configure streaming

Configure your IBM Cloud Logs instance to stream data to a topic in your IBM® Event Streams for IBM Cloud® instance.

Complete the following steps:

Log in to your IBM Cloud account.
Access your IBM Cloud Logs instance
Click the Data pipeline icon > Streams.
Click + Add stream.
Configure the stream:
- For Stream name, enter a name of your choice.
- For Stream Url, enter the Bootstrap server list. Complete the following steps to get the list:
  
  Click the Menu icon > Resource list.
  
  Look for the Event Streams instance that you plan to use, and select it.
  
  Select Topics. Then, click Connect to this topic
  
  In the Resources section make a copy of the provided Bootstrap server list.
  
  Connect dialog
- For Topic name, enter the name of your IBM® Event Streams for IBM Cloud® topic.
- To limit the data that will be sent to the topic, enter a query that matches the logs you want sent. For example, $m.severity == ERROR
  
  For more information on how to define data rules, see Configuring streaming data rules.

The following image shows a sample configuration:

add-streams — Example stream configuration

Validate the configuration

Validate that you can stream a sample test log line to the topic before starting streaming.

Complete the following steps:

Use a Kafka tool to connect securely to the IBM® Event Streams for IBM Cloud® instance. For example, you can use Use Kafka Connect or ksqlDB.
Send a sample log to verify the connection.

Click Send sample log to test the connection. You should see the sample message in your console within a minute.

Start streaming

After you verify that the sample log is successfully received, set the streaming configuration to active and click Create stream.

It takes a few minutes before a streaming configuration starts streaming once you enable the configuration.

Your IBM® Event Streams for IBM Cloud® will now filter and receive messages according to the specified query.