Define custom entities

Labeling tips

Review these tips before you begin:

• The document collection that you label must contain a representative set of documents. The documents must have many and varied examples of the entity types that you want the entity extractor to recognize. If the collection you selected when you started to create the entity extractor does not meet the requirement, stop now and start over with a different document collection.
• Define entity types that are clearly distinct from one another.
• Aim to label at least 40 examples of each entity type.
• Label every valid example of an entity type. Do not skip any occurrences. To speed up the process, use the bulk label feature.

Labeling entity examples

Label terms in the document that represent examples of the entity types you defined. When you are done with one document, switch the document status from In progress to Complete, and then move on to the next document.

To label entity examples, complete the following steps:

1. Review the text of the document. Look for entity examples to label.

   The following table shows some examples.

   Entity types and examples

   Entity type	Examples to label in the document
   color	white, green, purple
   car	convertible, SUV, sedan
   auto_model	Explorer, Civic, Sorrento
   auto_manufacturer	Ford, Honda, Kia
   clothing	shirt, blouse, skort
   instruments	bonds, stocks, ETFs, munis

   If an entity type that you want to identify is not created yet, add the entity type. From the Entity types panel, click Create new. For more information about adding entity types, see Defining entity types.

2. First, click the entity type from the Entity types panel.

3. In the document body, select the word or phrase that represents the entity example.

   The term is selected and a color label is applied to the term. The first two characters of the entity type name are shown in uppercase superscript within the label boundary. Both the 2-character ID and the label color help you to associate the example with the entity type it represents.

   

   The example text is also added to the Entity types panel. If you click the chevron to view details, you can see that the example is listed. The example text is saved in lowercase, regardless of the capitalization that is used in the original text.

4. If bulk labeling is enabled, a notification is displayed to show the number of occurrences of the term that were found and labeled in the current document.

5. If you want to label occurrences of the term in all of the documents in the collection, click Apply to all documents.

   When you enable this option, occurrences of the term are labeled in all of the documents in the collection, including documents that you already reviewed and marked complete.

   You are asked to confirm the action because it cannot be undone. If you don't want to have to confirm the action every time you choose to apply bulk labeling to all documents, select Do not ask for confirmation again. Click Run.

   

   For more information, see Labeling examples in bulk.

6. Scroll through the document to label every valid example of every entity type that you want your extractor to recognize.

   You can search for terms that you want to label as entity examples. For more information, see Searching for examples by using keywords.

   The machine learning model learns as much from the terms that you don't label as the terms that you do.

   If you miss labeling a valid example, the model learns that when the term is used in that context, it is not a valid mention of the entity type. In some cases, an omission is appropriate. For example, some terms have different meanings in different contexts. You don't want to label the term when it is used in the wrong context. However, if the term is used in the right context and you don't label it, you are teaching the model to ignore it. You decrease the model's effectiveness when your training data is inconsistent.

   After you label many examples, entity example suggestions are displayed. You can accept or reject entity example suggestions.

   

   Accepting example suggestions is another way to speed up the labeling process. For more information, see Entity example suggestions. After you accept a suggestion, you can bulk label the term.

7. If you make a mistake and label the wrong word or a word was labeled incorrectly by the bulk labeling process, you can delete the label.

   Hover over the labeled word until the Delete this example option is displayed, and then click it. You can choose to delete only this mention or all of the mentions in the document. Make a choice, and then click Delete.

8. After you label all of the entity examples in the current document, change the document status from In progress to Complete.

   Another document from the collection is displayed.

9. Label examples of your entity types in each document in the collection.

   At any time during the labeling process, you can click Save entity extractor to save your work.

10. If you don't have enough examples in the current set of documents, you can add more documents.

    From the Document list panel, click Add documents. The option is available only when more documents are available in the collection. You can add up to 20 documents. If bulk labeling for all documents is enabled, labels are applied to the newly-added documents automatically.

11. After you label examples in as many documents in the collection as you want, click Save entity extractor, and then open the Train extractor page.

Searching for examples by using keywords

Using the search feature, you can find entity examples in a document and label them easily. You can also use search to find labeled examples and unlabeled examples and correct any labeling inconsistencies.

To search by using keywords, complete the following steps:

1. On the Label documents view, click the Find icon.

2. In the Find field, specify a keyword to search in the document.

   The search results from the document are displayed when you enter the keyword.

   To browse the search results, you can click the Next result and Previous result icons. To choose a label for an unlabeled example in the result, click the Edit label icon and select a label. You can also remove a label from an already labeled example in the result by clicking the Edit label icon.

3. To filter the search results, click the Show filter options icon.

   The following table describes the filter options.

   Filter options in find

   Option	Description
   All	To find all examples in a document that match the keyword.
   Labeled text	To find existing labeled examples in a document that match the keyword.
   Unlabeled text	To find unlabeled examples in a document that match the keyword.
   Match case	To find examples that match both the keyword and its case.
   Whole words	To find examples that match the word boundaries of the keyword. For example, if you specify york as the keyword, yorktown is not matched when this option is selected.

For the unlabeled examples in the results, you can accept or reject a label suggestion.

To resolve any overlapping examples, click Review suggestions and choose an entity example suggestion from the Overlapping entity example suggestions dialog box.

Labeling examples in bulk

For most entity examples, enabling the bulk label feature is helpful. You might want to skip it if a term has more than one meaning in different contexts. In that case, evaluate each occurrence individually. Remember, if you enable the bulk label feature, you can check the accuracy of the labels that were added automatically and make corrections when necessary as you review the document.

After you enable the bulk label feature, a notification is displayed that indicates how many occurrences of an entity example were found in the current document. From the current page, the labeling tool cannot access other documents to report how many occurrences exist in other documents from the collection. However, the mention count is shown in the Entity types panel. When you first open other documents, you can check the mention counts to see how many mentions were labeled automatically.

Did the bulk label feature miss an occurrence?

Occurrences of the term are not labeled if they occur in the same phrase in which the term is already labeled. For example, the first occurrence of the term husband is not labeled when the bulk label feature is switched on for the second occurrence of the term in the following sentence.

Entity example suggestions

After you label enough examples, suggested entity type examples are displayed. The system learns from the types of examples you label, and applies what it learns to identify potential new examples. For example, after you label red, orange, yellow, green, and blue as examples of the color entity type, the Example suggestions panel might show indigo and violet as suggested examples for you to label. Suggestions are not displayed until after you label many examples of an entity type.

The following example shows suggestions that are made for family member mentions.

You might notice that a term that you chose to bulk label is not labeled, but is displayed as a suggestion instead. A term is skipped in the following situations:

• The term might occur in different noun phrases in different sections of the document. For example, the term father might occur in the noun phrases the kindest *father* and to her *father*. When a word is included in a noun phrase with adjectives, the meaning can change. Therefore, such terms sometimes are suggested rather than labeled automatically.
• A word might be a valid example on its own and as part of a multiple-word mention. For example, a mention of IBM might refer to the company International Business Machines, Corp. or might be used as part of a product name, such as IBM Cloud Pak for Data. However, a word or phrase can be part of only one example. Example labels cannot overlap one another. Therefore, you must choose which example suggestion is the most accurate. In this example, where the term IBM is used as part of a product name, it is more accurate to label the full phrase as an example of the Product entity type.
• The service might recognize that a term is a possible example of more than one entity type. For example, the word top might mean the best or might mean shirt.

To investigate a suggestion further, click it to see the word in context within the document. Seeing the term in context helps you to decide whether the occurrence is a valid entity example for you to label.

Exporting labeled data for an entity extractor

You can export the labeled data for an entity extractor from Discovery. You can use the exported labeled data for training or building large language models (LLMs) on a service such as Watson Studio and Natural Language Processing (NLP).

To export the labeled data, complete the following steps:

1. From the Improvement tools panel of the Improve and customize page, expand Teach domain concepts, and then click Extract entities.

2. For the entity extractor from which you want to export labeled data, click the Actions icon, and then select Download labeled data.

   A compressed file is downloaded with labeled data. The compressed file contains the following JSON files.

   • labeled_data.json: Includes the text and labels. The data format is based on the input data format for entity extraction in Watson Natural Language Processing. For more information, see Input data format.
   • metadata.json: Includes metadata for the workspace and labeled data.

Troubleshoot training issues

Learn about possible error messages and how to address them.

The training data is too large

Your training data contains large text document or many entity types and resources that are needed to process the data is greater than the resources that are available to your service instance. This error can occur even when your workspace doesn't exceed the documented entity extractor limits. To resolve the issue, you can try one of the following approaches:

• Remove one or more entity types to decrease the size of your training data.
• Remove extra large documents from the training data. For example, if one of the labeled documents is extra large, change its status from Completed to In progress to omit it from the training data.
• Reduce the number of documents that are included in the training set. The default split ratio (70%-23%-7%) for the training data uses 70% of the documents in the training set. You can change the percentage of documents that are used in the training set to a smaller number. For example, you might change the split ratio to 60%-33%-7%.
• IBM Cloud Pak for Data Increase the capacity of your deployed service instance by scaling up service pods.

Improving the extractor

The following table shows suggested fixes for common problems.

Adding documents to the training data

To add more documents, complete the following steps:

1. Open the Label documents tab.

2. From the Document list panel, choose Add documents.

   This button is disabled if no other documents are available to add to the entity extractor from the current collection. To add more documents to the collection, go to the Activity page for the collection, and then click the Upload data tile to browse for and add more files.

You cannot choose the documents from the collection to show in the Document list for labeling purposes. If there are specific types of documents that you want to label, consider adding representative documents to a collection that you can use to create the entity extractor.

There are limits to the number of documents that can be included in the training data. If your training data includes documents with a combination of sections that are labeled and others that are not, the system might sample some examples from unlabeled sentences. Subsampling helps to balance the number of positive and negative examples that are used for training. Balancing the examples in the training set improves the training performance.

Entity extractor output

When the enrichment recognizes one of your custom entities in a document, an entry is added to the enriched_text.entities section of the JSON representation of the document. The section contains occurrences of entities that are recognized by your custom model along with entities that are recognized by the built-in Entities enrichment. The built-in enrichment uses the Watson NLP service to identity entites that are part of what it calls the Natural Language Understanding type system. For more information about the built-in Entities enrichment, see Entities.

The following JSON output is produced by a custom model that is named literature that recognizes family member mentions.

Monitoring performance over time

You can retrain your entity extractor model at any time. Each time that you train the model, review the performance metric scores to determine whether your most recent changes increase or decrease the model's scores.

1. To compare one test run against another, click View score history.

   The history view shows the last 5 training runs.

   To retain the score information for more than the most recent 5 training runs, you can export the metrics in comma-separated value format, and track the scores in a separate application. Click the tabular representation icon , and then click Download as CSV.

If a subsequent training run results in lower scores, don't publish that version of the model.