Improving result relevance with training

When to use relevancy training

Relevancy training is optional. Test the quality of your search results. If the results of your queries meet your needs, no further training is necessary.

The training improves the relevancy of the documents that are returned in query responses. It does not improve the passages or answers that are returned per document. If you're using passage retrieval and your test results are returning good documents, but the wrong passages from the documents, relevancy training will not help.

For more information about when to use relevancy training, read the Relevancy training for time-sensitive users blog post on Medium.

How fields are handled

When you train a project from the product user interface, the results are always taken from the text field of the documents. If your documents don't have a text field, use the API to train your project instead. Your documents might not have a text field if you uploaded a CSV file that doesn't have a column named text, or uploaded a JSON file that doesn't have an object named text, or if you used the Smart Document Understanding tool to define fields with other names in which the bulk of the content of your documents now are stored.

When you train a project from the API, results are taken from all of the root-level fields and they are all considered to have equal significance. Unlike Discovery Query Language queries, with natural language queries you cannot specify which fields from the document you care about or how much significance to give to each one. When you teach Discovery with examples, the service figures out for you how much weight to give to each field.

Discovery builds a model that assigns different weights to term, bigram, and skip-gram matches for each of the root-level fields and balances them against matches from all of the other document fields. With enough examples, Discovery can return better answers because it knows where the best answers are typically stored.

Relevancy training cannot be used to give more weight to nested fields. Nested fields are grouped and assigned one overall score. No matter how much you train, Discovery never gives a nested field more weight than it gives to a root-level field. For more information about nested fields, see the FAQ.

Training a project

The training data that is used to train the relevancy model includes these parts:

• A natural language query that is representative of a query that your users might submit
• Results of the query which are returned by the service
• The rating that you apply to the result that indicates whether the result is relevant or not relevant

To apply relevancy training to a project, complete the following steps:

1. Go to the Improve and customize page. On the Improvement tools panel, select Improve relevance, then Relevancy training.

2. Enter a natural language query in the Enter a question to train field.

   Do not include a question mark in your query. Use the same wording as your users. For example, IBM Watson in healthcare. Write queries that include some of the terms that are mentioned in the target answer. Term overlap improves the initial results when the natural language query is evaluated.

3. Click Add+.

4. Click Rate results.

5. After the results are displayed, assess each result, and then select Relevant or Not relevant, whichever option applies given the quality of the result.

   In the Discovery UI, when you mark a document as Relevant, the service applies a relevancy score of 10 to the result and a score of 0 when you mark it as Not relevant. The only two score values that you can assign when you apply the relevancy training through the Discovery UI are 0 and 10.

   In the Discovery API, you can assign relevancy score values between 0 and every integer between up to 100. In the same project, you cannot mix training examples that are created in the Discovery UI (scores of 0 and 10) with examples created that use confidence score levels other than 0 and 10. If you use any custom score scale through the API, you must continue to apply all your training through the API. You cannot edit the training examples that are applied through the API in the Discovery UI unless they use only the two scores of 0 and 10.

   If the result shows the message, “No content preview available for this document”, it means that the document that was returned does not contain a text field or that its text field is empty. If none of the documents in your collection have a text field, use the API to train the project instead of training it from the product user interface.

6. When you are finished, click Back to queries.

7. Continue adding queries and rating them.

   As you rate results, your progress is shown. Check your progress to see when enough rating information is available to meet the training threshold needs. Your progress is broken into the following tasks:

   • Add more queries
   • Rate more results
   • Add more variety to your ratings

   You must evaluate at least 50 unique queries, maybe more, depending on the complexity of your data. You cannot add more than 10,000 training queries.

8. You can continue adding queries and rating results after you reach the threshold. Enter all of the queries that you think your users will ask.

   To delete a training query, click the Delete icon. To delete all of the training queries in your collection at one time, use the API. For more information, see Delete training queries.

If two or more users attempt to train identical queries at the same time, the ratings that are submitted by one of the users overwrites the others.

Testing and iterating on the relevancy of results

When you are done rating results, and training is completed, test to see whether your query results are better. To do so, run test natural language queries that are related (but not identical) to your training queries. Review the results.

If you want to continue to improve the results after testing, you can:

• Add more documents to your collection.
• Add more training queries.
• Rate more results, making sure to use both the Relevant and Not relevant ratings.

Confidence scores

Discovery returns a confidence score for natural language queries of trained collections. This confidence score is not interchangeable with confidence scores that are returned by untrained collections.

The confidence score can range from 0.0 to 1.0. The higher the number, the more relevant the result.

The confidence score can be found in the query results, under the result_metadata for each document. This number is calculated based on how relevant the result is estimated to be, compared to the trained model.

{
  "matching_results": 4,
  "retrieval_details": {
    "document_retrieval_strategy": "trained"
  },
  "results": [
    {
	  "id": "eea16dfd5fe6139a25324e7481a32f89_13",
	  "result_metadata": {
	    "confidence": 0.08793
	  }
    }
  ]
}

The document_retrieval_strategy can be found under the retrieval_details. If you query a trained collection by using the Discovery Query Language, or the trained model is temporarily disabled, the document_retrieval_strategy is untrained.

For more information on querying a project, see the Query overview.

Relevancy training limits

The following limits apply to relevancy training models:

• One model per project
• 10,000 queries per model
• 40 models per service instance for Enterprise and Premium plans; 20 models for Plus plan instances

Running optimal training sessions

The following example describes how to optimally run sessions for relevancy training in projects.

Consider that you added 100 training queries to a new project, and Discovery runs 100 queries in one training session to create a ranker model. Later, if you add another 20 queries, the model starts to retrain and Discovery runs a total of 120 queries. However, for adding the last 20 queries, if you add 10 queries first, wait for an hour or so, and then add the next 10 queries, Discovery trains the model twice. In this case, the first training session runs 110 queries, and the second training session runs 120 queries. This way of adding queries, interspersed between time gaps, results in an increased number of training sessions and total queries run by Discovery.

Instead, to minimize the number of training sessions, you can use the Update a training query API. The API method updates the training data for multiple collections under one project at once. For more information, see Update a training query in the API reference.

To reduce the processing load on Discovery, you should create or update a ranker model through relevancy training after all the collections in a project have completed processing documents. Also, during the ranker model training process, you should have low query activities so that the limit of concurrent requests does not exceed.

Other ways to improve relevancy

If you prefer to use the Discovery API to train Discovery, see the API reference.

You also can use the API to add curations. Curations is a beta feature that you can use to teach Discovery to return a specific document every time a certain query is submitted. For more information, see Curations.

Adding a custom stopwords list can also improve the relevance of results for natural language queries. For more information, see Identifying words to ignore.

Understanding relevancy training

Answers to common questions about training a project.

Interpreting relevancy training errors and warnings

The following list has explanations for some common error and warning messages.