Streaming data

Configure streaming

Consider the following information when streaming data to an Event Streams instance:

• You must have manager role to configure streaming in the Activity Tracker instance. This role includes the logdnaat.dashboard.manage IAM action role that allows a user to perform admin tasks such as configure streaming.

• When you configure streaming, the Activity Tracker instance and the Event Streams instance must be provisioned in the same account.

• To connect the Activity Tracker instance to the Event Streams instance, you need the following information:

  • Endpoint URLs to call the APIs

  • Credentials for authentication

• If you configure the account to restrict access to configured IP addresses via IAM settings, or if the account limits the network locations that connections are accepted from via context based restrictions rules (CBR) for the Event Streams service, you must allowlist the Activity Tracker CIDR blocks in the account. For more information, see Activity Tracker CIDR blocks and Event Streams - Restricting network access.

• To create a topic in Event Streams, you must have manager role. This role includes the messagehub.topic.manage IAM action role that allows an app or user to create or delete topic.

• The credential that Activity Tracker uses to publish data in Event Streams must have writer role. This role includes the messagehub.topic.write IAM action role that allows an app or service to write data to 1 or more topics.

Consider the following information when streaming data to an Activity Tracker instance:

• The Activity Tracker instance data that will receive data must be configured with a paid service plan. Activity Tracker instances on the Lite plan cannot receive streamed data.

• You must have manager role to configure streaming in the Activity Tracker instance. This role includes the logdnaat.dashboard.manage IAM action role that allows a user to perform admin tasks such as configure streaming.

• When you configure streaming, the source Activity Tracker instance and the destination Activity Tracker instance can be provisioned in the same account or in different accounts.

• To connect the source Activity Tracker instance to the destination Activity Tracker instance, you need the following information:

  • Destination Activity Tracker ingestion URL

  • Ingestion key for the destination Activity Tracker for authentication

• If you have any regulatory restriction to keep data within specific regions, make sure streaming is only configured to a valid destination.

Monitor streaming

To monitor streaming, you can use the following services:

• IBM Cloud Monitoring service to monitor streaming to an Event Streams instance:

  Event Streams is integrated with the Monitoring service. Monitoring provides a default template that you can customize to monitor the Event Streams instance, how data is streamed out of Activity Tracker and consumed by any application or service that is subscribed to Event Streams.

  For more information, see Monitoring streaming by using IBM Cloud Monitoring.

• IBM Cloud Activity Tracker:

  Streaming generates Activity Tracker events with the action logdnaat.streaming-logs.send to notify of failures sending data. There are different reasons for failure such as invalid credentials and topic deleted.

  For more information, see Monitoring streaming by using Activity Tracker.

Conditional streaming

You can configure exclusion rules to filter out data from streaming. For more information, see Configuring conditional streaming.

• You configure streaming exclusion rules through Settings > Streaming > Exclusion rules.
• The exclusion rules that you define for streaming are different from the exclusion rules that you can define at the instance level through Settings > Usage > Exclusion rules.

When you define exclusion rules, either at the instance level, or for streaming, they are applied as follows:

• Exclusion rules that you define at the instance level are applied first.
• Only the data that is retained and available for search is in scope of the exclusion rules that you define for streaming.
• After a streaming exclusion rule is active, data that matches the filter criteria is not streamed.
• Conditions that are applied by a query are enforced.

Activity Tracker events

The following Activity Tracker events are generated when you configure streaming: