IBM Cloud Logs log parsing rules

Rules groups

Parsing rules are organized inside Rule Groups. Each group has a name and a set of rules with a logical relationship between them. Logs are processed according to the order of rule group (from the beginning to the end). They are then processed by the order of rules within the rule group and according to the logical operators between them (AND/OR). Rules help you to process, parse, and restructure log data to prepare for monitoring and analysis. For more information, see Working with rule groups.

Types of rules

You can configure the following parsing rules:

• Parse : Convert unstructured text into JSON format.

• Extract : Extract values as JSON keys without parsing the entire log.

• Extract JSON : Extract the value of a JSON key into a metadata field.

• Replace : Replace the value of a field with a fixed string value to fix the log structure, change the log severity, or obfuscate information.

• Block : Drop incoming logs by using a regex expression.

• Timestamp Extract : Replace the log timestamp with a timestamp that is included in the log.

• Remove Fields : Remove JSON fields in a log to reduce the number of indexed fields.

• Stringify JSON Field : Transform a JSON field into a stringify JSON field to reduce the number of indexed fields.

• Parse JSON Field : Transform escaped or stringified fields into JSON format.

Rules that change the original log

These rules modify the original log lines.

• Parse
• Timestamp Extract
• Replace
• Parse JSON field

Rules that add more fields to the original log

These rules add fields to the log line while the original log data is maintained.

• Extract JSON
• Extract

Rules that drop data

These rules remove data from a log line.

• Block
• Remove
• Stringify JSON field (can keep original log or not)

Order of execution of rules

As logs are ingested, rules groups are applied in order, starting from the first configured rule.

Specify block rules first and use the rule matcher when possible to prevent unnecessary processing of logs and speed-up your data processing.

Within a rules group:

1. The rules group matcher query is applied first.

   Only logs that match the applications, subsystems, and severities that are configured continue applying the parsing rules.

2. Rules are applied in order from the first rule until a rule is followed by an OR logical operator.

   Rules are applied immediately so that the output of one rule becomes the input of the next one.

   The OR and AND operators in the group do not follow the mathematical order of operations.

The order of execution of rules groups and rules is important, and define the final log’s structure.