IBM Cloud Docs
Enriching data

Enriching data

You can easily enrich your log data with IBM Cloud Logs. You can automatically add fields to your JSON logs based on specific matches in your log data by using a pre-defined custom data source of your own. This way, you can enhance your log data with business, operations, or security information that is not available at run time.

You can enrich your logs in two possible ways:

  • Select a log key to be used to look up a key value and enrich the logs automatically during ingestion. The logs are saved with the enriched fields. The advantages of this mode are:

    • Logs are automatically enriched.

    • The logs include the enrichment data, which can be consumed everywhere (for example, in any query and also by third-party products that read the logs from the bucket).

  • Use the DataPrime query enrich to look up a value in a table and enrich the log dynamically for the query. The advantages of this mode are:

    • You can enrich old logs already ingested into IBM Cloud Logs.

    • The enrichment does not increase the size of the stored logs, since the enrichment is done dynamically, and is only used for the query results.

Data enrichment use cases

Some example use cases where enrichment can be helpful are:

Monitoring

In this example, assume we have logs with a UUID representing a customer. However, no field exists in the log with the customer name.

You can enrich the log by adding a field containing the customer name so you can visualize and search the logs base on the name. With custom enrichment, you create the enrichment by setting up a CSV file to map each UUID to a customer name.

Security

In this example, the logs contain a field with a domain name that represents where an application is accessed. You want to create an alert that creates a notification if an attempt to access the application is made from an unauthorized domain.

You can create a CSV file with a list of allowlisted domains so each log is enriched with a field (domain_enriched) with the word allowed for domains in the list. You can then create an alert for logs that do not contain this field (for example NOT domain_enriched:allowed).

Geo Enrichment

Logs can include IP information.

With the geo enrichment, you can automatically add IP-based geographical information to your logs as new fields that can be queried, visualized, and reported.

For example, you can add the country name, city name, continent name, postal code, and location geo point.

If you don’t have IP fields set, or your data isn’t JSON-formatted, you can use IBM Cloud Logs rules to extract the IP addresses that are found in your log records by using the Extract or Parse rules.

After you define the IP field, IBM Cloud Logs adds geographical information to the logs based on the selected fields.

These fields are not added if the enrichment database does not have the queried IP.