---
name: cloud-logs-streaming_config
title: Integrating IBM Cloud Logs with Event Streams
description: You can use IBM® Event Streams for IBM Cloud® with IBM Cloud Logs to stream and manage log data. IBM® Event Streams for IBM Cloud® is a scalable, managed Apache Kafka service that lets applications send data by creating messages and sending them to a topic. Applications can subscribe to these topics to receive messages, enabling real-time data processing and analytics.
last-updated: 2025-04-28
---

> ## Documentation Index
> The table of contents for this documentation set is at https://cloud.ibm.com/docs/cloud-logs?format=markdown
> The index for all IBM Cloud docs is at: https://cloud.ibm.com/docs/llms.txt
> Use these files to discover more information as needed.

# Integrating IBM Cloud Logs with Event Streams
{: #streaming-config}

You can use IBM® Event Streams for IBM Cloud® with IBM Cloud Logs to stream and manage log data. IBM® Event Streams for IBM Cloud® is a scalable, managed Apache Kafka service that lets applications send data by creating messages and sending them to a topic. Applications can subscribe to these topics to receive messages, enabling real-time data processing and analytics.
{: shortdesc}

When you update the streaming configuration after it is enabled, the changes are picked automatically. It takes a few minutes for the changes to be applied.
{: note}

## Prereqs
{: #streaming-config-prereqs}

- An IBM® Event Streams for IBM Cloud® instance is provisioned in the same account as the IBM Cloud Logs instance from where you plan to stream data.

    Check the limitations of the service plans. For more information, see [Limits and quotas](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-kafka_quotas&format=markdown).
    {: important}

- You have permissions to create a topic in the IBM® Event Streams for IBM Cloud® instance.

    To create a topic in Event Streams, you must have the **manager** role for the Event Streams instance. This role includes the **messagehub.topic.manage** IAM action role that allows an app or user to create or delete topics.

- You have permissions to configure streaming in your IBM® Cloud Logs instance.

    You must have the **manager** service role.



## Create the service to service authorization
{: #streaming-config-1}
{: step}

Authorization permissions to send data from an IBM® Cloud Logs instance to a topic that is defined in an Event Streams instance is done by configuring a service to service authorization in IAM in IBM Cloud.

To configure the service to service authorization, see [Creating a S2S authorization to work with the Event Streams service](https://cloud.ibm.com/docs/cloud-logs?topic=cloud-logs-iam-service-auth-es&format=markdown).


## Create a topic
{: #streaming-config-2}
{: step}

In your IBM® Event Streams for IBM Cloud® instance, you must define a topic where the IBM Cloud Logs instance stream data.

Complete the following steps to create an Event Streams topic:

1. Click the **Menu** icon ![Menu icon](../icons/icon_hamburger.svg) > **Resource list**.

2. Look for the Event Streams instance that you plan to use, and select it.

3. In the Event Streams instance console, click **Topics**.

4. Click **Create topic**.

5. Enter a topic name and click **Next**.

6. Enter the number of partitions and click **Next**.

    One or more partitions make up a topic. A partition is an ordered list of messages. Partitions are distributed across the brokers in order to increase the scalability of your topic. You can also use them to distribute messages across the members of a consumer group.
    {: note}

7. Select a **Message retention**.

    **Message retention** defines how long messages are retained before they are deleted. If your messages are not read by a consumer within this time, they will be missed.
    {: important}

8. Click **Create Topic**.


## Configure streaming
{: #streaming-config-3}
{: step}

Configure your IBM Cloud Logs instance to stream data to a topic in your IBM® Event Streams for IBM Cloud® instance.

Complete the following steps:

1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login){: external}.

2. [Access your IBM Cloud Logs instance](https://cloud.ibm.com/docs/cloud-logs?topic=cloud-logs-instance-launch&format=markdown#instance-launch-cloud-ui)

3. Click the **Data pipeline icon** ![Data pipeline icon](/icons/data-pipeline.svg "Data pipeline") > **Streams**.

4. Click **+ Add stream**.

5. Configure the stream:

   * For **Stream name**, enter a name of your choice.

   * For **Stream Url**,  enter the _Bootstrap server_ list. Complete the following steps to get the list:

        Click the **Menu** icon ![Menu icon](../icons/icon_hamburger.svg) > **Resource list**.

        Look for the Event Streams instance that you plan to use, and select it.

        Select **Topics**. Then, click **Connect to this topic**

        In the **Resources** section make a copy of the provided _Bootstrap server_ list.

        ![connect-to-service-page](images/es-connect-page.png){: caption="Connect dialog" caption-side="bottom"}

   * For **Topic name**, enter the name of your IBM® Event Streams for IBM Cloud® topic.

   * To limit the data that will be sent to the topic, enter a query that matches the logs you want sent. For example, `$m.severity == ERROR`

       For more information on how to define data rules, see [Configuring streaming data rules](https://cloud.ibm.com/docs/cloud-logs?topic=cloud-logs-streaming_rules&format=markdown).

The following image shows a sample configuration:

![add-streams](images/es-add-streams.png){: caption="Example stream configuration" caption-side="bottom"}


## Validate the configuration
{: #streaming-config-4}
{: step}

Validate that you can stream a sample test log line to the topic before starting streaming.

Complete the following steps:

1. Use a Kafka tool to connect securely to the IBM® Event Streams for IBM Cloud® instance. For example, you can use [Use Kafka Connect or ksqlDB](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-quick_setup_guide&interface=ui&format=markdown#kafka_connect_ksql).

2. Send a sample log to verify the connection.

    Click **Send sample log** to test the connection. You should see the sample message in your console within a minute.



## Start streaming
 {: #streaming-config-activate}
 {: step}

After you verify that the sample log is successfully received, set the streaming configuration to active and click **Create stream**.

It takes a few minutes before a streaming configuration starts streaming once you enable the configuration.

Your IBM® Event Streams for IBM Cloud® will now filter and receive messages according to the specified query.