Why am I not seeing results from my searches?

When searching in IBM® Cloud Logs I am not seeing the expected results returned. I know the fields I am searching for in my logs exist, but IBM Cloud Logs doesn't seem to find them with my searches.

Searching for field values does not return expected results.

The problem is often related to a mapping exception.

A mapping exception can happen if new log fields are ingested with field names that were previously ingested by IBM Cloud Logs, but have a different value type. When IBM Cloud Logs indexes ingested data, it tries to map the fields to one of the following types:

• String
• Numeric
• Object
• Array

If, for example, the field user has been ingested and the first received value is “username”, IBM Cloud Logs will create an index of type string for the field user:

"user": “username”`

If later on a new log is ingested that also has a field named “user” that is formatted like this:

"user": {
  “email”: “username@yourcompany.com”,
  ”login”: “username”`
}

A mapping exception is created in IBM Cloud Logs. Any searches looking for a field named user would be expected to have a string value based on the first log ingested with a user field.

Any searches looking for a user with the array type (user.email or user.login) will not return any results due to the mapping exception.

You can determine if there are mapping exceptions using the IBM Cloud Logs UI.

To identify mapping exceptions, complete the following steps:

1. Check the IBM Cloud Logs UI for an icon next to the logs count:

   

   Hovering over the icon will display the number of logs with mapping exceptions within the query results in the selected timeframe.

2. To identify the specific logs that were not properly indexed, enable the Show Mapping Exception option:

   

3. After enabling this option, refresh your browser.

   An exclamation mark will be displayed on the fields that have mapping exceptions and were not indexed.

   

   Another way to quickly identify which fields have a mapping exception is to run the query: _exists_: coralogix.failed_reason

   

   This query will return only the log lines that have a mapping exception for the selected timeframe.

There are two ways you can resolve mapping exceptions:

• Option 1: Search without indexing.

  If you do not find a field when querying due to mapping exceptions, you can search in All Logs instead of Priority Insights. When searching in All Logs, indexing and mapping exceptions are bypassed by the search.

• Option 2: Fix log records at ingestion.

  You can fix new log records being ingested with the same field names but with different data types.

  1. Create a RegEx expression to replace the field name with the appropriate sufix. For example:

     field_obj

     If IBM Cloud Logs is expecting a string, but found an object:

     • Regular Expression: ("field"\s*:\s*{)
     • Replace with: "field_obj":{

     field_arr

     If IBM Cloud Logs is expecting a string, but found an array:

     • Regular Expression: ("field"\s*:\s*[)
     • Replace with: "field_arr":[

     field_str

     If IBM Cloud Logs is expecting an object, but found a string:

     • Regular Expression: ("field"\s*:\s*")
     • Replace with: "field_str":"

  2. Create a parsing replace rule to fix new logs using the RegEx.