IBM Cloud Docs
Connecting IBM® MQ to Event Streams by using the Kubernetes service

Connecting IBM® MQ to Event Streams by using the Kubernetes service

Get the Kafka Connect runtime to run in an IBM Cloud® Kubernetes Service cluster. Then, start the IBM MQ Source Connector to copy messages from an IBM MQ source queue to a destination Kafka topic in Event Streams.

The IBM MQ Source Connector connects to an IBM MQ queue manager and consumes MQ message data from an MQ queue. The Connector converts each MQ message into a Kafka record and sends the message to an Event Streams Kafka topic.

Step 1. Install the prerequisites

Ensure you have the following software and services installed:

  • An Event Streams instance - Standard or Enterprise plan.

  • An instance of IBM MQ on Cloud or IBM MQ Version 8, or later.

    You can configure the IBM MQ Connector to authenticate with IBM MQ by using a user identifier and password. We recommend that you grant the following permissions only to the identity associated with an instance of the MQ bridge:

    • CONNECT authority. The IBM MQ Connector must be able to connect to the MQ queue manager.
    • GET authority for the queue that the IBM MQ Connector is configured to consume from.

  • An Kubernetes Service cluster. You can provision one for testing purposes at no cost.

    You also need CLI access to your cluster. For more information, see Setting up the CLI and API.

  • A recent version of kubectl

  • Git

Step 2. Clone the kafka-connect repositories

Clone the following two repositories that contain the required files:

Step 3. Create your Kafka Connect configuration

  1. You must set up this configuration only once. Event Streams stores it for future use.

    From the event-streams-samples project, navigate to the kafka-connect/IKS directory, edit the connect-distributed.properties file, and replace <BOOTSTRAP_SERVERS>; in one place and <APIKEY> in three places with your Event Streams credentials.

    Provide <BOOTSTRAP_SERVERS> as a comma-separated list. If they are not valid, you get an error.

    Your <APIKEY> appears in clear text on your machine but is secret when pushed to Kubernetes Service.

    Kafka Connect can run multiple workers for reliability and scalability reasons. If your Kubernetes Service cluster has more than one node and you want multiple Connect workers, edit the kafka-connect.yaml file, and edit the entry replicas: 1.

  2. Then, run the following commands.

    To create a secret, run the following command:

    kubectl create secret generic connect-distributed-config --from-file=connect-distributed.properties

    To create a configmap, run the following command:

    kubectl create configmap connect-log4j-config --from-file=connect-log4j.properties

Step 4. Deploy Kafka Connect

Apply the configuration in the kafka-connect.yaml file by running the following command:

kubectl apply -f ./kafka-connect.yaml

Step 5. Validate Kafka Connect is running

To validate that Kafka Connect is running, port forward to the kafkaconnect-service on port 8083, as in the following example.

kubectl port-forward service/kafkaconnect-service 8083

Keep the terminal that you used for port forwarding open, and use another terminal for the next steps.

The Connect REST API is then available at http://localhost:8083. If you want more information about the API, see Kafka Connect REST Interface.

So, you now have the Kafka Connect runtime that is deployed and running in Kubernetes Service. Next, let's configure and start the IBM MQ Connector.

Step 6. Configure the mq-source JSON file

Edit the mq-source.json file that is located in kafka-connect-mq-source/config so that, at a minimum, the required properties are completed with your information.

mq-source.json file properties

Replace the placeholders in the mq-source.json file with your own values.

TOPIC
Required. Name of the destination Kafka topic.
QUEUE_MANAGER
Required. Name of the source IBM MQ queue manager.
QUEUE
Required. Name of the source IBM MQ queue.
CHANNEL_NAME
Required (unless you're using bindings or a CCDT file). Name of the server-connection channel.
CONNECTION_NAME_LIST
Required (unless you're using bindings or a CCDT file). A list of one or more host(port) pairs for connecting to the queue manager. Separate entries with a comma.

Step 7. Start the connector with its configuration

Run the following command to start the IBM MQ Connector with the configuration that you provided in the previous step.

curl -X POST -H "Content-Type: application/json" http://localhost:8083/connectors --data "@./mq-source.json"

Step 8. Monitor your connector

You can check your connector by going to the following address:

http://localhost:8083/connectors/mq-source/status

If the state of the connector is not running, restart the connector.