Connecting to Event Streams

To connect to Event Streams, a Virtual Private Endpoint (VPE) for VPC must be created to establish a secure private connection. Credentials must then be created to authenticate with IAM. For applications not running within VPC, additional networking setup is required.

The following information gives an overview of the steps required, using the example of a Kafka client application running on a VPC VSI, but could be applied to any of the VPC application platforms. For apps running externally to a VPC, further information is provided for the additional steps required.

Creating a VPE

To enable an application deployed in an IBM Cloud® Virtual Private Cloud to access your Enterprise instance over the private network, a virtual private endpoint (VPE) must be created in the VPC.

  1. In the IBM Cloud console, click the menu icon and select VPC infrastructure > Network > Virtual private endpoint gateways.
  2. Create a VPE for your Event Streams instance by using the guidance in About virtual private endpoint gateways.
  3. After you create your VPE, it might take a few minutes for the new VPE and pDNS to complete the process and begin working for your VPC. Completion is confirmed when you see an IP address set in the details view of the VPE.
  4. Take a note of this VPE IP address.

Creating a service credential

All plans use IAM for authentication. You don't need to understand IAM to get started but some knowledge is recommended if you want to secure your Event Streams service. For more information, see Managing access to your Event Streams resources. To complete the following steps and be authorized to create topics, your application must have a Manager access role. By default, the owner of the account that contains the service instance has this role.

A service credential enables an API key to be created with the required access role. To create a service credential by using the IBM Cloud console, complete the following steps.

  1. Locate your Event Streams service on the dashboard.
  2. Click your service tile.
  3. Click Service credentials.
  4. Click New credential.
  5. Complete the details for your new credential. Choose a name and a role and click Add. A new credential appears in the credentials list.
  6. Click the new credential by using View credentials to reveal the details in JSON format.

To create a service key by using the IBM Cloud CLI, complete the following steps.

  1. Locate your service:

    ibmcloud resource service-instances

  2. Create a service key:

    ibmcloud resource service-key-create <key_name> <key_role> --instance-name <your_service_name>

  3. Print the service key:

    ibmcloud resource service-key <key_name>

Connect from a VSI

The following information shows the steps needed to connect to a VPE from an application running on a VSI.

Create an IBM Cloud VPC

Set up a Virtual Private Cloud in your region. Make sure to select 'Allow SSH' in the options for the VPC before creating. Create your VPC

Keep resources in the same region to avoid issues.

Create an SSH key

  1. Create an SSH key in the same region as the VPC.

  2. Once your key is ready, move it to the .ssh directory on your local machine to follow best practices for secure SSH key management.

  3. Update the key’s permissions to make it read-only for the file owner. On Unix-like systems such as macOS, run the following command:

    $ chmod 400 <COPY LOCAL LOCATION OF THE SSH KEY>

Create a Virtual Server Instance (VSI) in the VPC

Create a VSI in the same region, make sure to:

  • Choose Ubuntu Linux for operating system. (You can use the smallest profile.)
  • Select the SSH Key created above.
  • Select the VPC created above.

Reserve a floating IP for your VSI

Reserve a floating IP address. Make sure the correct region and zone is selected, and bind it to the VSI created in the previous step.

This enables the VSI to be connected to from the internet. If required, the security group associated with the VSI can be modified to only allow connections from specific IP addresses, for example, the specific IP address of the machine you connect from.

Log in to your VSI

In your terminal, SSH in to your VSI with the following command.

If you saved the private SSH key in a different directory, replace the file path in the command accordingly.

$ ssh -i ~/.ssh/<NAME OF THE SSH KEY> root@<FLOATING IP ADDRESS>

Your local terminal session should now be connected to your virtual server. Continue using this session for the next steps.

Install and use the Kafka console tools

The following steps show how to install and use the console tools to create a topic and then produce and consume a message.

Accessing an Enterprise instance from an external network

Solutions such as Direct Link 2.0 or Transit Gateway can be utilized to establish a connection between an external network, such as an on-premise data center or Classic infrastructure, and the IBM Cloud VPE. However, when dealing with workloads operating on an external network, it is essential to consider additional factors to ensure a successful connection to Kafka. Note that these considerations do not apply to HTTP workloads.

The private endpoint details allocated to your instance (as described in the service key) must be resolvable and routable from the network that the workload is running in. It is not possible to specify alternative hostname entries in the workload's bootstrap.servers properties as a way to route traffic from the external network.

The reason for this behavior is Kafka's two-step connection process. In the initial step, the hostnames that are provided in the client's bootstrap.servers property are used to establish the first bootstrap connection. However, the server then responds to the client with the actual endpoint hostname details it uses. These hostname details are the private endpoint details originally allocated to your instance and cannot be changed. Hence, the details must be resolvable and routable directly from the external network.

For more information, see Accessing a VPE externally from a VPC.

What to do next

Now you have connection and credential information, you can choose a Kafka client. For more information, see Using the Kafka API.

Further Information