--- name: discovery-data-nlu title: Use built-in Watson NLP to find common terms description: Take advantage of award-winning Watson Natural Language Processing (NLP) capabilities by adding prebuilt enrichments to your documents. last-updated: 2024-08-28 --- > ## Documentation Index > The table of contents for this documentation set is at https://cloud.ibm.com/docs/discovery-data?format=markdown > The index for all IBM Cloud docs is at: https://cloud.ibm.com/docs/llms.txt > Use these files to discover more information as needed. # Use built-in Watson NLP to find common terms {: #nlu} Take advantage of award-winning Watson Natural Language Processing (NLP) capabilities by adding prebuilt enrichments to your documents. {: shortdesc} With Watson NLP, you can identify and tag meaningful information in your collections so you can understand what it all means and make more informed decisions. The following Watson NLP enrichments are available: - [Entities](#nlu-entities): Recognizes proper nouns such as people, cities, and organizations that are mentioned in the content. - [Keywords](#nlu-keywords): Recognizes significant terms in your content. - [Part of Speech](#nlu-pos): Identifies the parts of speech (nouns and verbs, for example) in the content. - [Sentiment](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-nlu-sentiment&format=markdown): Understands the overall sentiment of the content. The following other pretrained enrichments are available with Discovery: - [Contracts](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-contracts-schema&format=markdown) - [Document structure](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-sdu-pretrained&format=markdown) - [Table understanding](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-understanding_tables&format=markdown) ## Watson NLP enrichments {: #nlu-overview} For example, the following screen capture shows a transcript of the US Declaration of Independence that was added to a Discovery collection where the Entities and Keywords enrichments are enabled. The mentions that are recognized by the enrichments are highlighted in the document text. ![Shows an excerpt of the US Declaration of Independence with several terms highlighted.](images/nlu-rich-both.png){: caption="Excerpt of the US Declaration of Independence with highlighted terms" caption-side="bottom"} Some of the NLP enrichments are applied to projects automatically. You don't need to apply them yourself if you are using one of these project types. Some prebuilt enrichments are applied automatically to collections in a project based on the project type. The following table shows the default enrichments that are applied to each project type. | Enrichment | Document Retrieval | Document Retrieval for Contracts | Conversational Search | Content Mining | |------------|--------------------|----------------------------------|-----------------------|----------------| | Contracts | | ![checkmark icon](../icons/checkmark-icon.svg) | | | | Entities | ![checkmark icon](../icons/checkmark-icon.svg) | ![checkmark icon](../icons/checkmark-icon.svg) | | | | Keywords | | | | | | Part of Speech | | | | ![checkmark icon](../icons/checkmark-icon.svg) | | Sentiment of Document | | | | | | Table Understanding | | ![checkmark icon](../icons/checkmark-icon.svg) | | | {: row-headers} {: class="comparison-table"} {: caption="Default enrichments per project type" caption-side="top"} {: summary="This table has row and column headers. The row headers identify project types. The column headers identify different enrichments. To understand which enrichments are applied to a project type by default, go to the row that describes the enrichments, and find the columns for the project type that you are interested in."} For more information about the following prebuilt enrichments, see the following topics: - [Contracts](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-contracts-schema&format=markdown) - [Table Understanding](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-understanding_tables&format=markdown) For more information about how to create custom enrichments, see [Adding domain-specific resources](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain&format=markdown). For more information about how to get the most from enrichments, read the [Enriching your documents can make search more effective](https://community.ibm.com/community/user/watsonai/blogs/bill-murdock1/2022/01/14/enriching-your-documents-can-make-search-more-effe){: external} blog post. For more information about how to apply enrichments by using the API, see [Applying enrichments by using the API](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-manage-enrichments&format=markdown#enrichments-api-task). ## Add enrichments {: #nlu-task} To add an NLP enrichment, complete the following steps: 1. Open your project and go to the *Manage collections* page. 1. Click to open the collection that you want to enrich. 1. Open the **Enrichments** tab. 1. Scroll to find the NLP enrichment that you want to apply to your documents. Both built-in enrichments and custom enrichments are listed. Built-in enrichments have a type value of `System`. {: note} 1. Choose one or more fields to apply the enrichment to. You can apply enrichments to the `text` and `html` fields, and to custom fields that were added from uploaded JSON or CSV files or from the Smart Document Understanding (SDU) tool. 1. Click **Apply changes and reprocess**. Enrichments that you enable are applied to the documents in random order. For information about how to remove an enrichment, see [Managing enrichments](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-manage-enrichments&format=markdown). ## Entities {: #nlu-entities} Identifies entities. *Entities* are terms that typically represent proper nouns such as people, cities, and organizations that are mentioned in the data collection. Discovery can recognize entities that are part of an entity type system that is defined by the Watson Natural Language Processing (NLP) service. If you want to be able to identify uncommon terms that are significant to your business, you can train your own model to recognize custom entities. For more information, see [Entity extractor](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-entity-extractor&format=markdown). The Watson NLP entity extractor service that is used by Discovery is called the *NLU type system*. The name originates from the fact that the type system is used by the Watson Natural Language Understanding (NLU) service in addition to the Watson Discovery service. However, it is the Watson NLP implementation of the type system that is used directly by Discovery, not the Watson NLU implementation. As a result, the two implementations can produce different results. To get a general idea of the types of entities that are recognized by the service, see [Entities](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-entity-types-version-2&format=markdown){: external}. The following screen capture shows that the Entities enrichment recognizes the terms *Systems of Government* and *King of Great Britain* (among others) and tags them as entity mentions. ![Shows the declaration with the terms Governments and King of Great Britain highlighted.](images/nlu-rich-view.png){: caption="The recognized entities, Governments and King of Great Britain, are highlighted" caption-side="bottom"} From the JSON view of the document, you can see the underlying JSON structure of the entity mentions. ![Shows the JSON view of the Systems of Government and King of Great Britain entities that are identified in the document](images/nlu-json-view.png){: caption="JSON representation of recognized entity mentions" caption-side="bottom"} If you want to search for the Organization entity type, for example, you can copy all of the JSON content into a text editor and search for `Organization`. Click the *Copy* icon from the root of the JSON tree view. ### Example {: #nlu-entities-example} #### Input {: #nlu-entities-example-input} ```text "IBM is an American multinational technology company headquartered in Armonk." ``` {: codeblock} #### Response {: #nlu-entities-example-response} In the JSON output: - `text` = string. The entity text - `type` = string. The entity type, such as `Organization`, `Location`, `Person`, `Number`. - `mentions` = array. The entity mentions and locations - `model_name` = string. For custom models, this field contains the user-provided model name. Otherwise, this field contains the default name of the model, such as `watson_knowledge_studio`, `dictionary`, `character_pattern`, or `natural_language_understanding` ```json { "entities": [ { "model_name": "natural_language_understanding", "mentions": [ { "confidence": 0.8317045, "location": { "end": 3, "begin": 0 }, "text": "IBM" } ], "text": "IBM", "type": "Organization" }, { "model_name": "natural_language_understanding", "mentions": [ { "confidence": 0.6114863, "location": { "end": 75, "begin": 69 }, "text": "Armonk" } ], "text": "Armonk", "type": "Location" } ] } ``` {: codeblock} ## Keywords {: #nlu-keywords} Returns important keywords in the content. For example, the following screen capture shows highlighted terms from the US Declaration of Independence that are recognized by the Keywords enrichment. ![Shows the keywords that are recognized in the document text](images/nlu-rich-keywords.png){: caption="Terms recognized by the Keywords enrichment" caption-side="bottom"} From the JSON view of the document, you can see the underlying JSON structure of the `Declaration` keyword mention. ![Shows the JSON view of keywords that are identified in the document](images/nlu-json-keywords.png){: caption="JSON representation of Keywords enrichment mentions" caption-side="bottom"} ### Example {: #nlu-keywords-example} #### Input {: #nlu-keywords-example-input} ```text "Watson Discovery is an award-winning AI search technology." ``` {: codeblock} #### Response {: #nlu-keywords-example-response} In the JSON output: - `text` = The keyword text - `mentions` = The entity mentions and locations ```json { "keywords": [ { "mentions": [ { "location": { "end": 157, "begin": 141 }, "text": "Watson Discovery" } ], "text": "Watson Discovery", "relevance": 0.503613 }, { "mentions": [ { "location": { "end": 177, "begin": 164 }, "text": "award-winning" } ], "text": "award-winning", "relevance": 0.728722 }, { "mentions": [ { "location": { "end": 198, "begin": 181 }, "text": "search technology" } ], "text": "search technology", "relevance": 0.779356 } ] } ``` {: codeblock} ### Keywords limits {: #nlu-keywords-limits} The Keywords enrichment can identify up to 50 keywords, each with one or many mentions, per document. ## Part of speech {: #nlu-pos} Recognizes and tags parts of speech, including nouns, verbs, adjectives, adverbs, conjunctions, interjections, and numerals.