How your data is processed

HTML fields

The html field in the document index stores structural information about the document.

• If you use the Smart Document Understanding tool to annotate a collection, the document representation is indexed in the html field.
• If you use the Smart Document Understanding tool to apply a pretrained model to a collection, the document representation is indexed both in the html field and text field.
• The html field has a size limit. For more information, see Field limits.

Note about enhancing data:

• If you want to apply an enrichment that can understand the tables in a document, the document must contain an html field.

Date format settings

If your documents have a root-level field with date information in it, you can set the field to be a Date data type field in the index.

Discovery recognizes the following date formats automatically:

yyyy-MM-dd'T'HH:mm:ssZ
yyyy-MM-dd'T'HH:mm:ssXXX
yyyy-MM-dd'T'HH:mm:ss.SSSZ
yyyy-MM-dd'T'HH:mm:ss.SSSX
yyyy-MM-dd
M/d/yy
yyyyMMdd
yyyy/MM/dd

If you store dates in other formats, you can add the format to the list of supported formats.

To add more date formats, complete the following steps:

1. From the Manage fields page for the collection, add a format as a new line in the Date formats field.

   Specify a date format that is supported by the Java SimpleDateFormat class.

   For example, if your records store only year values for dates, add yyyy to the supported date formats list. You can then set the data type for the field that contains a year value to Date, and reprocess your collection. As a result, an occurrence of 2019 in the date field is stored as 2019-01-01T05:00:00Z in the index.

   When you add a date format, you must specify an associated time zone for the date.

2. Specify a time zone.

3. Optionally, select a date locale.

   The locale that you choose is used to parse a string value that represents the date for the date-type data set fields. For example, by using the EEE, MM dd, yyyy format, the English (United States) locale can parse the string value of "Wednesday, 07 01, 2020", and the Japanese (Japan) locale can parse the same string value of "水曜日, 07 01, 2020".

4. If you already imported documents with dates in formats that were not recognized, reprocess the documents.

Discovery cannot store a date that is mentioned within a text field as a Date field in the index. However, you can use an enrichment such as the Entities enrichment to identify dates that are mentioned in text.

CSV files

Notes about adding data:

• Each line that is defined in the CSV file is added to the index as a separate document, each with the same parent_document_id.

  The child documents typically have a document ID with the syntax {parent-ID}_n where {parent-ID} is the document ID of the original file that was added and n is a sequential number. For example, if you upload a CSV file with 5 rows, then five documents are added to the collection with document IDs such as f5214225c1e03e25190ffcdfad8e84ff_0 through f5214225c1e03e25190ffcdfad8e84ff_4.

• You cannot enable the Optical Character Recognition (OCR) feature for CSV files.

• If the CSV file has headers, the header names are used to name the fields in which the content from the corresponding column is stored. Do not use names that have special meaning in Discovery. Be sure that the field names conform to the naming rules, such as having no spaces and no appended numbers. For example, you can rename the start date header to start_date and label1 to label-one before you add the file. For more information, see How fields are handled.

• When a CSV file header name contains restricted characters, the document converter automatically removes the restricted characters from the field name when it adds the resulting field to the index.

Note about enhancing data:

• You cannot apply prebuilt or user-trained Smart Document Understanding models to CSV files.

HTML files

If you upload an HTML file or crawl a data source with HTML files, such as a website, an html field is generated along with the text field. For more information, see HTML fields.

JSON files

Notes about adding data:

• Object names from the source JSON file are used to name the fields in which the content is stored. Do not use names that have special meaning in Discovery. Be sure that the names conform to the naming rules, such as having no spaces and no appended numbers. For example, you can rename the updated on object to updated_on and answer2 to answer-two before you add the file. For more information, see How fields are handled.

• If a root-level field is an array but contains no items, the field is omitted from the index.

• If a root-level field is an array and contains only one item, the array is indexed as the data type of the one item. For example, a string array with one string is indexed as a string.

• If a nested field contains an array, even if the array has only one value, it is indexed as an array.

• If a root-level field is an array and contains more than one item, the data is indexed as an array.

• If you copy JSON that is generated by Discovery and then upload it as a JSON file, remove these system-generated fields from the file first: document_id, parent_document_id, filename, and title.

• You cannot enable the Optical Character Recognition (OCR) feature for JSON files.

• If your source document has a field with the name document_id, the field is skipped and not added to the index in the collection.

  How the document_id field in a JSON file is handled changed with the 2023-03-31 version update of the API. Before the update, when you uploaded a JSON file from the product user interface or used the API to add it with the Add document method, the value in the document_id field from the file was shown as the document_id value in query results. However, a different document ID was assigned to it and stored in the parent_document_id field. The assigned document ID is what was returned when you called the List documents method and is what had to be used as the document_id in the endpoint URL for a Delete document method request. When you used the Update document method to assign a new document_id, the original ID continued to be returned in query results. However, the assigned ID had to be used to delete the document. If you have an application that relies on the previous behavior, you can specify a version number earlier than 2023-03-31, such as 2020-08-30, in your API calls.

Notes about enhancing data:

• You cannot apply prebuilt or user-trained Smart Document Understanding models to JSON files.

• When you apply an enrichment to a field from the JSON file, the field data type is converted to an array. The field is converted to an array even if it contains a single value. For example, "field1": "Discovery" becomes "field1": ["Discovery"].

• Only the first 50,000 characters of a custom field from a JSON file are enriched.

• In project types where the Part of Speech (POS) enrichment is applied automatically, the enrichment is applied to the field that contains the bulk of the file content in the first JSON file that is added to the collection. This field is determined by the following rules:

  • If a field is named text, the POS enrichment is applied to it.
  • The field with the longest string value and highest number of distinct values is chosen.
  • If more than one field meets the previous condition, one of the fields is chosen at random.

• If you want to apply an enrichment to a nested field, you must create a Content Mining project, and then apply the enrichment to the field. If you want to use a project type other than Content Mining, you can reuse the collection that you created with the Content Mining project type elsewhere. For more information, see Applying enrichments.

You can specify the normalizations and conversions objects in the Update a collection method of the API to move or merge JSON fields.