--- name: cloud-logs-query-mapping-exceptions title: Managing data mapping exceptions description: In IBM Cloud Logs, you can get a data mapping exception when data that is ingested through the Priority insights data pipeline detects the same field sent by different log records with different types. last-updated: 2024-10-11 --- > ## Documentation Index > The table of contents for this documentation set is at https://cloud.ibm.com/docs/cloud-logs?format=markdown > The index for all IBM Cloud docs is at: https://cloud.ibm.com/docs/llms.txt > Use these files to discover more information as needed. # Managing data mapping exceptions {: #query-mapping-exceptions} In IBM Cloud Logs, you can get a data mapping exception when data that is ingested through the Priority insights data pipeline detects the same field sent by different log records with different types. {: shortdesc} For instance, in a log ingested from an application, you see a field with type string and in another log generated by a different application, the same field is insgested as a number or a different type. This causes a mapping exception. A mapping exception can happen if new log fields are ingested with field names that were previously ingested by IBM Cloud Logs, but have a different value type. When IBM Cloud Logs indexes ingested data, it tries to map the fields to one of the following types: * `String` * `Numeric` * `Object` * `Array` If, for example, the field `user` has been ingested and the first received value is “username”, IBM Cloud Logs will create an index of type `string` for the field `user`: ```json "user": “username”` ``` {: codeblock} If later on a new log is ingested that also has a field named “user” that is formatted like this: ```json "user": { “email”: “username@yourcompany.com”, ”login”: “username”` } ``` {: codeblock} A mapping exception is created in IBM Cloud Logs. Any Priority insights searches looking for a field named `user` would be expected to have a `string` value based on the first log ingested with a `user` field. Any Priority insights searches looking for a `user` with the `array` type (`user.email` or `user.login`) will not return any results due to the mapping exception. ## Checking the number of mapping exceptions {: #query-mapping-exceptions-1} To check the number of mapping exceptions, in the navigation bar, click the **Usage** icon ![Usage icon](/icons/usage.svg "Usage") > **Mapping Stats**. You can get the daily total number of mapping exceptions in the *Mapping Exceptions - Today's count* section. ## Identifying fields with mapping exceptions {: #query-mapping-exceptions-2} To identify the fields that caused the mapping exceptions, you can run the following query: ```text _exists_:"coralogix.failed_reason" ``` {: codeblock} ## Searching data that includes mapping exceptions {: #query-mapping-exceptions-3} When you search in Priority insights, log records that include a mapping exception can only be searched by using a free text query. If you have a data bucket associated to the instance, you can search logs through **All Logs**. You can search by using a free text query or by using key:value pairs. ## Alerting {: #query-mapping-exceptions_alert} Alerting is not affected by data mapping exceptions and will continue to be triggered as normal. ## Fixing fields with mapping exceptions {: #query-mapping-exceptions-4} To fix a mapping exception, identify the field that reports a mapping exception and the applications that are sending the same field with different types. Then, try to fix the problem at source. To identify mapping exceptions, complete the following steps: 1. Check the IBM Cloud Logs UI for an icon next to the logs count: ![Mapping exceptions icon](../images/me_01a.png){: caption="Mapping exceptions icon" caption-side="bottom"} Hovering over the icon will display the number of logs with mapping exceptions within the query results in the selected timeframe. 2. To identify the specific logs that were not properly indexed, enable the `Show Mapping errors` option: ![Selecting how to show mapping errors. Click Settings and then click Show mapping errors.](../images/me_02a.png){: caption="Selecting how to show mapping errors" caption-side="bottom"} 3. After enabling this option, refresh your browser. An exclamation mark will be displayed on the fields that have mapping exceptions and were not indexed. ![Flagging of fields with mapping exceptions](../images/me_03a.png){: caption="Flagging of fields with mapping exceptions" caption-side="bottom"} Another way to quickly identify which fields have a mapping exception is to run the query: `_exists_: coralogix.failed_reason` ![Query to show mapping exceptions](../images/me_04a.png){: caption="Query to show mapping exceptions." caption-side="bottom"} This query will return only the log lines that have a mapping exception for the selected timeframe. There are two ways you can resolve mapping exceptions: - Option 1: Search without indexing. If you do not find a field when querying due to mapping exceptions, you can search in `All Logs` instead of `Priority Insights`. When searching in `All Logs`, indexing and mapping exceptions are bypassed by the search. - Option 2: Fix log records at ingestion. You can fix new log records being ingested with the same field names but with different data types. 1. Create a [RegEx expression](https://cloud.ibm.com/docs/cloud-logs?topic=cloud-logs-parse-rules-regex&format=markdown) to replace the field name with the appropriate sufix. For example: `field_obj` : If IBM Cloud Logs is expecting a string, but found an object: * Regular Expression: `("field"\s*:\s*{)` * Replace with: `"field_obj":{` `field_arr` : If IBM Cloud Logs is expecting a string, but found an array: * Regular Expression: `("field"\s*:\s*[)` * Replace with: `"field_arr":[` `field_str` : If IBM Cloud Logs is expecting an object, but found a string: * Regular Expression: `("field"\s*:\s*")` * Replace with: `"field_str":"` 2. Create a parsing [replace rule](https://cloud.ibm.com/docs/cloud-logs?topic=cloud-logs-parse-replace-rule&interface=ui&format=markdown) to fix new logs using the RegEx.