Classify sentences

Before you begin

Find or create a collection with documents having various sentence examples that you want Discovery to learn about. To teach the sentence classifier, you must label examples of sentence classes. You can only label examples if your collection contains valid examples. Try to find documents that have many and varying sentences that function as examples of every sentence class that you want to define.

Adding a sentence classifier

To add a sentence classifier, complete the following steps:

1. Open the project where you want to create the sentence classifier.

   The project must have at least one collection with documents that are representative of data that you want to classify.

2. From the Improvement tools panel of the Improve and customize page, expand Teach domain concepts, and then click Sentence classifiers.

3. Click New.

4. Add a sentence classifier name and optionally a description.

   This name is used as the model name and as the name of the enrichment that is created when you publish the model. The name is displayed as the enrichment name in the Enrichments page where you and others can apply it to collections. It is also displayed as the model name in the JSON representation of documents where sentence classes are found. The name is stored with the capitalization and spacing that you specify.

5. Choose a collection with documents that are representative of data that you want to classify.

6. Choose fields from the document to show in the document view where you will label documents from the collection.

   • Document title is shown in the page header as the document name. Choose a field that has a unique value per document, such as the file name, which is stored in the extracted_metadata.filename field.
   • Document body is where you label sentence examples in the content. Choose a field that contains the bulk of the document content, such as the text field.

7. Click Create.

A document from the collection that you selected is displayed in the Label documents view. You will label occurrences of the sentence classes that you want Discovery to recognize from this and other documents in the collection.

If no text is displayed in the body of the page, start over now by creating a new sentence classifier. This time, when you select a value for the Document body field, be sure to choose a field from your processed documents that contains text.

Defining sentence classes

Define sentence classes by completing the following steps:

1. Click Add a sentence class.

2. Add the sentence class name and an optional description.

3. Optional: Pick the color to use for the sentence class in the document.

   You can click a color from the Label color palette, click the Renew color icon to tab from one color to the next. To use a custom color, specify its hexadecimal color code (#fff0f7).

4. Click Create.

5. Repeat this process to add all of the sentence classes that you want the classifier to classify.

   If you aren't sure what to add for sentence classes, it might help to review the documents in the collection first. By reviewing the content, you can get a feel for which sentences have significant meaning and look for logical ways to classify such sentences.

Labeling sentences

From the Label documents view, find sentences in the documents from your collection and label them to indicate their sentence classes. While you are labeling, the sentence classifier automatically trains a model in the background to present labeling suggestions.

The labeling suggestions speed up the process of labeling your documents. Instead of going through documents and spending time reading text, the labeling suggestions enable you to easily find relevant examples and examine them in context.

You can label sentences in the following ways:

• Manual labeling. For more information, see Manual labeling.
• Find-in-document labeling. For more information, see Find-in-document labeling.
• Smart labeling. For more information, see Smart labeling.

Unlike entity extractor, there is no concept of document status such as Not started, In progress, and Complete for sentence classifier. In the case of entity extractor, you mark documents as Complete to indicate that they are used for training. In the case of sentence classifier, only labeled sentences are used for training and unlabeled data in each document is ignored.

Adding documents to the training data

To add more documents, complete the following steps:

1. Navigate to the Label documents view.
2. In the Document list panel, click Add documents.

This option is not available when there are no other documents in the collection to add to the sentence classifier workspace. To add more documents to the collection, navigate to the Activity page for the collection, and then click the Upload data tile to browse and add more documents.

Even if you add more documents from a collection, all the documents may or may not be used for training the model. Unlike entity extractor where all the completed documents are used for training, sentence classifier uses only labeled sentences for training and unlabeled data is ignored.

You cannot choose the documents from the collection to show in the Document list panel for labeling. If there are specific types of documents that you want to label, consider creating a new collection that contains only those documents.

Training the classifier

After you label documents, you can review the training data in the Train classifier view. The training data is used to train the sentence classifier model.

To train the classifier, complete the following steps:

1. Navigate to the Train classifier view.

2. Review your labeling summary to check if you labeled enough to train the classifier.

   To train the classifier, each sentence class must have a minimum of 20 positive labels and two negative labels. Otherwise, the Train classifier button is disabled and you cannot start training. Sentence classes that have no positive labels or negative labels are ignored.

3. Review whether you want to apply advanced options for training. Most models do not require changes to the advanced options.

   Your sentences are split at random into sets. The training set is used to train the classifier. The test set is used to test the model after it is trained. The blind set has reserved sentences that you don't see during training. They are used to generate an unbiased evaluation of the model periodically. The default split uses the standard ratio for training. For more information, see Document sets for training.

4. Click Train classifier.

   When you train the classifier, Discovery uses sentences from the training set to build a machine learning model. The results of the test are displayed for you to review in the Evaluate classifier view.

Evaluating the classifier

To review metrics from the test run of the sentence classifier model that you created, click the Evaluate classifier tab.

The following table describes the available evaluation metrics.

1. Review the metrics that are provided about the classifier model test run to determine whether more training is needed.

2. Explore the test results in more detail by clicking Review training results in test set.

   

   Sentences from the test set are displayed with the actual and predicted labels shown in the list on the left pane. If you click on a sentence in the list, it is shown in the document view on the right pane.

   • Actual labels are the examples that a person labeled manually. They are considered as the correct labels.
   • Predicted labels are the examples that the sentence classifier identified and labeled as sentence classes.

   The performance of the model is rated based on how closely the predicted labels match the actual labels.

3. To filter the list, click the Filter icon and choose the Sentence class and Prediction, and then click Apply.

Reviewing training results from performance breakdown

To review training results from the performance breakdown, complete the following steps:

1. Click a number in the Performance breakdown by sentence class table.

   The Review training results dialog box is displayed.

2. Review the relevant sentences, which are based on the number that you clicked.

Publishing the sentence classifier as an enrichment

When you think the sentence classifier is ready, publish the sentence classifier model. You can consider that the model is ready when the score doesn't change after several test runs in which you make improvements. You can return to update and retrain the model after you publish it.

1. Navigate to the Evaluate classifier view, and click Publish classifier.
2. Click Publish.
3. Click Apply to data.
4. Choose a collection, and then select the text field where you want the sentence classifier enrichment to be applied.
5. Click Apply.

Downloading the sentence classifier model

A sentence classifier model that you create and deploy in one project is available as an enrichment that can be applied to a collection from any project in the same service instance.

If you want to use the sentence classifier model in a project from another service instance, you can export or download the sentence classifier model. To use it elsewhere, follow the steps to create a machine learning model in Use imported ML models to find custom terms. You cannot continue to edit a sentence classifier that you import into another project.

The sentence classifier that you want to export must be fully trained.

To export a sentence classifier, complete the following steps:

1. Open the project with the sentence classifier that you want to export.

2. From the Improvement tools panel of the Improve and customize page, expand Teach domain concepts, and then click Sentence classifiers.

3. From the Sentence classifiers list, find the sentence classifier that you want to export.

4. Click the Actions icon for your classifier, and then select Download model to save the model to your system.

   The Download model option is not available unless the model is trained.

The sentence classifier model is saved as a .sc file. You can import it into a project in another service instance as a machine learning model, and then apply it to your collections.

Downloading labeled data for a sentence classifier

You can download or export the labeled data for a sentence classifier from Discovery. You can use the exported labeled data for training or building large language models (LLMs) on a service such as Watson Studio and Natural Language Processing (NLP).

To export the labeled data, complete the following steps:

1. From the Improvement tools panel of the Improve and customize page, expand Teach domain concepts, and then click Sentence classifiers.

2. For the sentence classifier from which you want to export labeled data, click the Actions icon, and then select Download labeled data.

   A compressed file is downloaded with labeled data. The compressed file contains the following JSON files.

   • labeled_data.json: Includes the text and labels. The data format is based on the input data format for text classification in Watson Natural Language Processing. For more information, see Input data format.
   • metadata.json: Includes metadata for the workspace and labeled data.

$ cat labeled_data.json | jq -r '.[] | [.text, .labels[]] | @csv'

    "sentence1", class-label1, class-label2
    "sentence2", class-label3
    "sentence3", ...
    ...

Applying a sentence classifier enrichment

When you publish the sentence classifier, you specify the field where you want the sentence classifier to be applied. If you decide to apply the enrichment to different or more fields later, you can follow these steps to do so.

1. From the navigation panel, click Manage collections.

2. Click to open the collection where you want to apply the enrichment.

3. Click Enrichments.

4. Find the sentence classifier name in the list, and then choose a field to apply the enrichment to.

   You can choose a field that contains either text or html.

5. Click Apply changes and reprocess.

For more information about how to apply a sentence classifier enrichment to a collection, see Managing enrichments

Sentence classifier output

When the enrichment classifies one of your sentences in a document, an entry is added to the enriched_text.element_classes section of the JSON representation of the document. The section contains sentences classified by your classifier model along with their sentence classes.

A sentence classifier does not classify sentences with confidence scores that are lower than 0.5.

The following JSON output is an example result of sentence classification.

Deleting a sentence classifier

You can delete a sentence classifier if it is not in use, such as when the enrichment that is published from the sentence classifier is not applied to a collection.

You might want to delete a sentence classifier if you reach the limit for the maximum number of sentence classifiers that are allowed for your plan, for example.

There are two different limits, the maximum number of sentence classifier workspaces and the maximum number of sentence classifier enrichments. You create a sentence classifier workspace when you navigate to the Improvement tools panel, expand Teach domain concepts, click Sentence classifiers, and then click the New button. You create a sentence classifier enrichment when you publish your trained sentence classifier or upload a sentence classifier model. For information about the limits, see Sentence classifier limits.

Remember, limits are defined per service instance, not per project. If you do not have the maximum number of sentence classifiers in the current project, but you cannot create new sentence classifier workspaces or publish your trained sentence classifier, then check other projects in the same service instance. There might be sentence classifier workspaces or enrichments that aren't being used in other projects which can be deleted.

Using the APIs for sentence classifier

The sentence classifier API is beta functionality.

You can use the APIs for applying sentence classifier enrichment to your documents. Using the APIs, you can create a sentence classifier enrichment, and also manage the enrichment such as updating and deleting it.

For using the sentence classifier APIs, do the following things:

1. Create a sentence classifier enrichment using the create an enrichment method in the API.

   For more information about creating the enrichment, see Create an enrichment in the API reference.

   You must specify the labeled data to train the sentence classifier model when you create the sentence classifier enrichment. The labeled data must be in the following CSV format:

   "sentence1", class-label1, class-label2
   "sentence2", class-label3
   "sentence3", ...
   ...
   

   Each row is a sentence followed by a comma-separated list of zero or more sentence class labels that are associated with the sentence.

   As a best practice, each sentence class label in the CSV file should be represented by at least 100 sentences to achieve sentence classification of reasonable quality. Sentences that are associated with a sentence class label are considered as positive examples of that sentence class. Sentences that are not associated with a sentence class label are considered as negative examples of that sentence class.

   After the sentence classifier enrichment is successfully created, navigate to the Manage collections page, choose the collection, and then open the Enrichments tab. You can find the sentence classifier enrichment in the list of available enrichments.

   After the sentence classifier enrichment Status is ready, you can apply the sentence classifier enrichment to the documents in your collection.

2. Apply the sentence classifier enrichment that you have created to the fields (text or html) in your documents to classify sentences. For more information about applying an enrichment and managing the enrichment using APIs, see Using the API to manage enrichments.

Sentence classifier limits

The number of sentence classifiers that you can create per service instance depends on your Discovery plan type.