Creating custom annotators

Dictionary configuration

You can import an existing dictionary by uploading it or you can create a dictionary by adding terms one at a time.

If you plan to import a dictionary, the dictionary terms must be defined in a CSV file. Specify each term and its synonyms in a separate line. Use the following syntax to specify each term:

{term},{synonym},{synonym},...

To add a dictionary, complete the following steps:

1. Do one of the following things:

   • To import the dictionary terms:

     1. Click Import, and then browse for the file with your dictionary terms.
     2. Click Import.

   • To define the dictionary terms:

     1. Click Add.
     2. Click Word list to add the dictionary terms.
     3. Click Add, and then add the term in the Base word field and any synonyms that you want to define for the term in the Other words field. Separate multiple synonyms with commas. Click OK.
     4. Repeat the previous step to add more dictionary terms.
     5. After you finish adding dictionary terms, click Basic settings.

2. Name the dictionary.

3. If you plan to define terms with a part of speech other than a noun, specify the part of speech.

   If the selected language is Chinese, Japanese, Korean, or Hebrew, you can only specify Noun for the part of speech.

4. Decide how you want to handle case.

   When case is ignored, the terms Sat, SAT, and sat are all labeled as occurrences of the Sat dictionary term.

   When you deselect the Ignore case checkbox to create a case-sensitive dictionary, the surface form of the term with uppercase match is used. Annotations are added for the term exactly as written and for variations of the term in which the letters are uppercase.

   For example, a sat entry in the dictionary results in annotations for sat, Sat, or SAT mentions when they occur in text. For a Sat entry in the dictionary, annotations are added for occurrences of Sat and SAT, but not for sat.

5. Identify the facet name to use for this dictionary.

   The facet name that you specify for the annotator is the facet name that is displayed from the collection search view.

   You can create a hierarchical facet by including a period (.) in the facet name. For example, you might create one dictionary with the facet path Food.Vegetables and others with the facet paths Food.Fruits and Food.Proteins. Add more facet groups with more periods. For example, you can add Food.Proteins.Nuts and Food.Proteins.Meats to categorize proteins even further.

   

6. If you want documents that are returned for a subfacet to be included when a user filters on the root facet, select Lift up words.

   For example, you might enable Lift up words for Food.Fruits and Food.Proteins but not Food.Vegetables. As a result, when a user clicks the Food facet, the returned documents include documents that mention terms included in the Fruits and Meats dictionaries, such as apples and beef.

   

   However, a user must click the Food>Vegetables facet explicitly to get documents that mention terms in the Vegetables dictionary, such as lettuce, to be returned.

   

7. Repeat previous steps to add more dictionaries.

8. Click Save.

From the custom annotator page, you can see dictionaries that were created in other projects, including non-Content Mining projects. Dictionaries from other project types show the enrichment name as the annotator name. The Ignore case and Lift up words settings are disabled and the dictionary is named custom dict.

Machine learning configuration

You can import an existing machine learning model.

To use Discovery to create a model, see Entity extractor.

To import a model, complete the following steps:

1. Click Select file, and then browse for the machine learning model file.

2. In the Facet path field, specify the root facet name to use for the model.

   The facet name that you specify for the annotator is the facet name that is displayed from the collection search view.

3. Click Save.

Regular expressions configuration

You can import existing patterns by uploading them in a JSON file or you can add patterns.

To add patterns, complete the following steps:

1. Add the regular expression pattern to the New pattern field, and then click Add.

2. Specify a name for the pattern, and then identify the facet name to use for this pattern.

   The facet name that you specify for the annotator is the facet name that is displayed from the collection search view.

3. Optional: Specify a facet value. You can specify a value from the options that are described in the table.

   Regular expression facet value options

   Facet value	Description
   $0	Displays the matched text as-is.
   $n	If your regular expression pattern contains groups, you can specify a group number to return the matched text from the pattern group only. For example, if your regular expression consists of 3 groups that define a US phone number pattern, such as (\d{3})-(\d{3})-(\d{4}), and you want to return only the area code portion of the phone number, you can specify $1. If the matched text is 212-555-1234, then the facet value is displayed as 212. Only specify a group as the facet value for patterns that you know will return matches.
   {prefix-text}:$0	Adds hardcoded text in front of the facet name. You might want to use this option if you want to distinguish facets that are generated by this regular expression from facets that are similar but generated in some other way. For example, MyRegex:$0 results in a facet named MyRegex:212-555-1234.

4. Click Save.

To import patterns, complete the following steps:

1. Define the patterns that you want to add in a JSON file.

   The pattern definition must use the following syntax:

   [
     {
       "name": "US Phone number",
       "description": "US mobile phone number",
       "pattern": "(\\d{3})-(\\d{3})-(\\d{4})",
       "facetPath": ".regex.usphonenumber",
       "facetValue": "$0"
     }
   ]
   

   Keep the following notes in mind:

   • The patterns must be defined in an array, even if you plan to define only one pattern.
   • Escape any backslash (\) characters with a backslash.
   • For more information about the facet value options, see the Regular expression facet value options table.

2. Click Import, and then choose the JSON file where the patterns are defined.

3. Click Save.

Applying the annotator

After the annotator is created, you must apply it to your collection.

1. From the Create a custom annotator for your analytics solutions page of the Content Mining application, click custom annotator, and then select collection from the list.

2. In the tile for your collection, click the options icon, and choose Edit collection.

3. Click the Enrichment tab, and then select the annotator that you created.

   You might need to scroll to find it.

4. Click Save, and then confirm the action.

Give the index time to rebuild.

Filtering documents with your facet

1. Click the collection tile to open your collection in the data analysis page.

2. Do one of the following things:

   • Your custom facets are listed in the Facets view. Scroll and click Load more repeatedly until your facets are displayed.

   • Submit an empty search to return all documents. In the Facet analysis pane, select the facet that you created.

   • To access your custom facets more quickly, add them to a custom view. Select Custom as the view, and then click Edit. Select one or more facets to add to the view, and then click Save.