Managing data mapping exceptions

In IBM Cloud Logs, you can get a data mapping exception when data that is ingested through the Priority insights data pipeline detects the same field sent by different log records with different types.

For instance, in a log ingested from an application, you see a field with type string and in another log generated by a different application, the same field is insgested as a number or a different type. This causes a mapping exception.

A mapping exception can happen if new log fields are ingested with field names that were previously ingested by IBM Cloud Logs, but have a different value type. When IBM Cloud Logs indexes ingested data, it tries to map the fields to one of the following types:

String
Numeric
Object
Array

If, for example, the field user has been ingested and the first received value is “username”, IBM Cloud Logs will create an index of type string for the field user:

"user": “username”`

If later on a new log is ingested that also has a field named “user” that is formatted like this:

"user": {
  “email”: “username@yourcompany.com”,
  ”login”: “username”`
}

A mapping exception is created in IBM Cloud Logs. Any searches looking for a field named user would be expected to have a string value based on the first log ingested with a user field.

Any searches looking for a user with the array type (user.email or user.login) will not return any results due to the mapping exception.

Checking the number of mapping exceptions

To check the number of mapping exceptions, in the navigation bar, click the Usage icon Usage icon > Mapping Stats. You can get the daily total number of mapping exceptions in the Mapping Exceptions - Today's count section.

Identifying fields with mapping exceptions

To identify the fields that caused the mapping exceptions, you can run the following query:

_exists_:"coralogix.failed_reason"

Searching data that includes mapping exceptions

When you search in Priority insights, log records that include a mapping exception can only be searched by using a free text query.

If you have a data bucket associated to the instance, you can search logs through All Logs. You can search by using a free text query or by using key:value pairs.

Alerting

Alerting is not affected by data mapping exceptions and will continue to be triggered as normal.

Fixing fields with mapping exceptions

To fix a mapping exception, identify the field that reports a mapping exception and the applications that are sending the same field with different types. Then, try to fix the problem at source.

To identify mapping exceptions, complete the following steps:

Check the IBM Cloud Logs UI for an icon next to the logs count:

Mapping exceptions icon

Hovering over the icon will display the number of logs with mapping exceptions within the query results in the selected timeframe.
To identify the specific logs that were not properly indexed, enable the Show Mapping Exception option:

Selecting how to show mapping errors
After enabling this option, refresh your browser.

An exclamation mark will be displayed on the fields that have mapping exceptions and were not indexed.

Flagging of fields with mapping exceptions

Another way to quickly identify which fields have a mapping exception is to run the query: _exists_: coralogix.failed_reason

Results of a query to show mapping exceptions.

This query will return only the log lines that have a mapping exception for the selected timeframe.

There are two ways you can resolve mapping exceptions:

Option 1: Search without indexing.

If you do not find a field when querying due to mapping exceptions, you can search in All Logs instead of Priority Insights. When searching in All Logs, indexing and mapping exceptions are bypassed by the search.
Option 2: Fix log records at ingestion.

You can fix new log records being ingested with the same field names but with different data types.
1. Create a RegEx expression to replace the field name with the appropriate sufix. For example:
  field_obj
  If IBM Cloud Logs is expecting a string, but found an object:
  - Regular Expression: ("field"\s*:\s*{)
  - Replace with: "field_obj":{
  field_arr
  If IBM Cloud Logs is expecting a string, but found an array:
  - Regular Expression: ("field"\s*:\s*[)
  - Replace with: "field_arr":[
  field_str
  If IBM Cloud Logs is expecting an object, but found a string:
  - Regular Expression: ("field"\s*:\s*")
  - Replace with: "field_str":"
2. Create a parsing replace rule to fix new logs using the RegEx.