Getting started with Event Streams Gen 2

Gen 2

This tutorial guides you through the steps to quickly start using Event Streams by provisioning an instance, creating a topic and a credential, and then producing and consuming data. Additionally, you'll learn how to connect IBM Cloud® Monitoring and IBM Cloud® Activity Tracker, and optionally how to use Kafka Connect or ksqlDB. Finally, you'll also find out how to get help with Event Streams.

Select your interface using the tabs at the start of the page.

Follow these steps to complete the tutorial:

Before you begin

Before you get started, we highly recommend that you read the following information to better understand Apache Kafka, which Event Streams is built on:

Step 1: Select your plan

Event Streams offers different plans according to the generation of the platform, see Choosing your plan. For the Gen2 platform the following plan can be selected.

  • The Enterprise Gen2 plan offers pay-as-you-go access to an isolated single-tenant Event Streams service running on the Gen2 (VPC) platform. In addition to a selection of throughput and storage options, this plan also offers user-managed encryption private endpoints and meets a higher number of regulatory compliance standards. The Enterprise Gen2 plan is the best choice if data isolation, scalability, security, guaranteed performance, and increased retention are important considerations.

Step 2: Provision an Event Streams instance by using the console

  1. Log in to the IBM Cloud console.

  2. Search for Event Streams in the Catalog and click on the Event Streams tile.

  3. Select a Location that supports the Gen2 platform.

  4. Select the Enterprise Gen2 plan from the Select a pricing plan section.

  5. Enter a name for your service. You can use the default value.

  6. Click Create. The Event Streams Resource list page opens.

  7. When your instance has been created, click on the instance name to view more information.

Step 3: Create a topic and select number of partitions by using the console

For guidance about the settings that you can modify when creating topics, see topic configuration.

  1. From your newly provisioned instance, navigate to Topics using the menu on the left.

  2. Click the Create topic button and an enter a topic name. Click Next. Topic names are restricted to a maximum of 200 characters.

  3. Select the number of partitions.

    One or more partitions make up a topic. A partition is an ordered list of messages. 1 partition is sufficient for getting started, but production systems often have more.

    Partitions are distributed across the brokers to increase the scalability of your topic. You can also use them to distribute messages across the members of a consumer group.

    Click Next.

  4. Set the message retention period. This is how long messages are retained before they are deleted. If your messages are not read by a consumer within this time, they will be missed. The default retention period for messages is 24 hours. The minimum is 1 hour and the maximum is 30 days. Specify this value as multiples of hours.

    Click Create topic.

Working with topics using the console

After you create topics, you can use the console to list topics.

List topics

From your Event Streams instance, navigate to Topics from the menu on the left.

From the Topics page, you can view the following information about your topics: Name, Partitions, Retention time, Retention size, and Cleanup policy.

Step 4: Create a service credential by using the console

To allow you to connect to your Event Streams instance, create a service key by using the IBM Cloud console:

  1. Locate your Event Streams service in the Resource list.
  2. Click your service tile.
  3. Click Service credentials.
  4. Click New credential.
  5. Complete the details for your new credential like a name and role and click Add. A new credential appears in the credentials list.
  6. Expand the new credential's section to reveal the details in JSON format.

Step 5: Using your instance to produce and consume data

To connect to Event Streams, a Virtual Private Endpoint (VPE) for VPC must be created to establish a secure private connection.

Follow the instructions to connect to your instance to create a VPE and run a producer and consumer application to send and recieve data.

Step 6: Connect IBM Cloud Monitoring for operational visibility by using the console

You can use IBM Cloud Monitoring to get operational visibility into the performance and health of your applications, services, and platforms. IBM Cloud Monitoring provides administrators, DevOps teams, and developers full stack telemetry with advanced features to monitor and troubleshoot, define alerts, and design custom dashboards.

For more information about how to use Monitoring with Event Streams, see:

Step 7: Connect IBM Cloud® Activity Tracker to audit service activity

IBM Cloud Activity Tracker allows you to view, manage, and audit service activity to comply with corporate policies and industry regulations. Activity Tracker records user-initiated activities that change the state of a service in IBM Cloud. Use Activity Tracker to track how users and applications interact with the Event Streams service on the Standard and Enterprise plans.

To get up and running with Activity Tracker, see Getting Started with Activity Tracker.

Activity Tracker can have only one instance per location. To view events, you must access the web UI of the Activity Tracker service in the same location where your service instance is available. For more information, see Launch the web UI.

For more information about events specific to Event Streams, see:

Events are formatted according to the Cloud Auditing Data Federation (CADF) standard.

Step 8: (Optional) Use Kafka Connect or ksqlDB

Kafka Connect

Kafka Connect is part of the Apache Kafka project and allows you to connect external systems to Kafka. It consists of a runtime that can run connectors to copy data to and from a cluster.

For more information, see Using Kafka Connect with Event Streams.

Kafka Connect is not part of the managed Event Streams service.

ksqlDB

You can use KSQL with the Event Streams Enterprise plan for stream processing.

ksqlDB is a purpose-built database for event streaming. Use it to build end-to-end event streaming applications quickly with a purpose-built stream processing database for Apache Kafka.

First complete these setup steps. Then the quickest and easiest way to run ksqlDB with Event Streams is to use a docker container as described in ksqlDB quickstart.

Step 9: Get help

For a general overview of how to get help with Event Streams and where to get support, see Getting help and support.

FAQs details answers to some of the common questions about Event Streams.

If you're experiencing a problem with Event Streams, here's a list of the information you need to gather before you open a case Reporting a problem to the Event Streams team - Enterprise Gen 2 plan.