Migrating to Discovery v2

Plan how to adapt your application

One of the main changes between versions is that Discovery v2 introduces projects. A project consists of one or more collections. The advantage of using projects is that one query can run against many collections at the same time. Each collection can contain documents that you upload or that you crawl from a single data source, such as a website, Microsoft SharePoint, and more.

Things to consider when you adapt your application to use projects:

• Although the concept of an environment does not exist in v2, data is still organized into collections. In v2, collections are grouped into projects. In most cases, you want to migrate a single v1 collection to a single v2 collection.

  If you want to keep relevancy training information that is applied to a v1 collection, add the collection documents to a single collection in your v2 project.

• Decide how many collections you want to add to each v2 project. All project types, except Content Mining projects, can contain up to 5 collections. Choose the right type of project for your data.

  To optimize search results, different enrichments and configuration options are applied automatically to collections that are added to different project types. For more information, see the following topics:

  • Project descriptions
  • Default project settings
  • Default query settings

• The Discovery v2 API changed to account for projects and collections, among other enhancements. Some API calls changed to support actions at the project level instead of the collection level, such as submitting a query and running relevancy training. Many other API methods changed and some are not available in v2. For a detailed comparison of the v1 and v2 API methods, see API version comparison.

Picking a service plan

Choose among the Plus, Enterprise, and Premium managed plans or opt for an on-premises installation by purchasing the Discovery Cartridge for IBM Cloud Pak for Data. Review the benefits and limits of each type of plan before you choose one.

• For more information about the plans, see Discovery pricing plans.
• For more information about artifact limits, see Limit details.

The following table shows plan types for managed deployments that are generally similar between v1 and v2.

To get information about the current storage, documents, and collections used, click the Environment details icon from the product user interface header.

You cannot do an in-place upgrade from a v1 plan, such as Lite or Advanced, to a v2 plan. You must create a new v2 plan, and then move your data to the new service instance. While you migrate your data from v1 to v2, you will likely have both a v1 and v2 instance deployed at the same time. Consider using the 30-day no charge trial that is available with your first Plus plan instance during this time.

Uploaded data

If you used the API to upload documents into Discovery v1, a similar API is available in v2 to upload documents into collections. You must update any workflows that you use to automate the process to account for the new arrangement of projects and collections.

If the original documents that you ingested into Discovery v1 are no longer available, you can use the query API to extract the document text from Discovery v1. You can then add the text to a collection in Discovery v2. For more information, see Recovering documents.

Crawled data

If you crawled data from an external data source in v1, you can continue to crawl data from the same external data source in v2. All of the same data sources are supported.

To use data from an external data source, you must re-create the collections within a v2 project, and configure how the data source is crawled. For more information, see Overview of data sources.

The service needs time and resources to crawl and ingest documents from external data sources. Re-create the connectors one at a time. Factor the time it takes to recrawl the data into your migration plan schedule.

Prebuilt data collections

The following built-in data source collections are not available in v2:

Watson Discovery News

This pre-enriched data source is not offered in v2. For more information about an alternative way to get news data, see Using a news service with v2.

COVID-19 kit

This pre-built collection was designed to help you fuel a dynamic chatbot that is built with IBM® watsonx™ Assistant and Discovery to answer your customers' questions about COVID-19. In v2, you can build a similar solution. Create a Conversational Search project type with collections that crawl trusted websites for answers to COVID-19 questions.

Retaining document IDs

Document IDs are assigned to the documents that you add to a v2 collection when you upload them from the product user interface or add them by using the Add a document API method.

You might want to retain the IDs of your v1 documents in v2 if you are using processes that depend on these unique identifiers. For example, regression testing for the application might verify that specific documents are returned by checking the document IDs. Relevancy training uses the document IDs to track documents between training runs. These processes are easier to adapt if the document IDs are the same between your v1 and v2 instances. Otherwise, the processes that are used with the Discovery v1 instance must be remapped to the IDs that are assigned to the documents after they are added to the Discovery v2 instance.

If you specified your own documents IDs when you added documents to the v1 service instance, you can retain the IDs by using the Update a document method instead of the Add a document method. With the update method, you can assign a document ID to the document as you add it to the v2 collection. For more information, see Update a document.

If your data is stored in a JSON file, an array in the original document generates a document ID with a number appended to it. For example, original_id_n. To retain the original document ID without the number suffix, remove the array in the JSON file. Change [ {"name": "value"} ] to {"name": "value"}, for example.

If your v1 documents have system-generated IDs, you can submit an empty search query to retrieve a list of the documents and their IDs. You can then assign the same ID to each document as you add it to your new collection in v2.

Recovering documents

In some cases, the original documents that were ingested into Discovery V1 are no longer available. You can use the Discovery v1 instance to retrieve information from the document. Discovery creates a text copy of each document that it ingests. The copy is text only, so any documents in HTML, PDF, or other nontext formats are converted to a text-only version.

You can recover only the first 10,000 documents in a collection by using this method. For more information about a way to recover more than 10,000 documents, see Recovering more than 10,000 documents from a collection.

To transfer document information from v1 to v2, complete the following steps:

1. Extract the documents from v1 by using the API to submit an empty query.

   For example, GET {url}/v1/environments/{environment_id}/collections/{collection_id}/query?q=.

   The API returns the results. The matching_results field specifies the total number of results. The results object returns the matching documents. Each document is returned as a separate JSON object. It returns a maximum of 10 documents by default.

   {
     "matching_results": 34,
     "session_token": "nnn",
     "results": [
       {"{result objects}":"{maximum of 10 by default}"}
     ]
   }
   

2. You can use the count and offset parameters to page through the query results and save all of the documents.

   For example, to get 100 documents at a time, you can set the count to 100 and offset to 0 and submit the query.

   GET {url}/v1/environments/{environment_id}/collections/{collection_id}/query?q=&count=100&offset=0
   

   Next, you can again set the count to 100, but this time set the offset to 100 to get the next 100 documents.

   GET {url}/v1/environments/{environment_id}/collections/{collection_id}/query?q=&count=100&offset=100`
   

   Repeat this process, incrementing the offset by 100 until you retrieve all of the documents.

3. Prepare the exported documents to be ingested into v2.

   Each resulting JSON file that you get from Discovery v1 contains data that is extracted from the original document, such as text, html, and other fields. If custom metadata was associated with the document when it was uploaded to v1, it is also present in the JSON file. In addition, the file contains several fields that were generated by the v1 analysis. Retain only a subset of this data as part of the document that you add to Discovery v2.

   The following tips can help you decide which fields to keep:

   • Include the text field or any other field with textual content that you want to be able to enrich or search in Discovery v2.
   • Include any custom metadata that is stored in the document. This metatdata is typically specific to the application that uses Discovery and is used to filter documents in a search. For example, metadata.customer_id.
   • Do not include enrichments from Discovery v1. For example, enriched_text.entities. Discovery v2 generates its own enrichments.
   • Exclude fields that are generated by Discovery unless they are used by your application and contain information that is unique to the v1 version of the document. In that case, rename the field so that it does not get replaced when the document is ingested into Discovery v2. For example, extracted_metadata.publicationdate is a field that is generated by Discovery when a document is ingested. Maybe you want to retain the metadata.parent_document_id information from v1 to understand how subdocuments were originally generated from a single source document.
   • Avoid fields that have reserved field names. For more information, see How fields are handled.

4. Ingest each edited v1 JSON document into the Discovery v2 instance. The Discovery v1 document ID can be maintained in Discovery v2. For more information about how to retain the document ID, see Retaining document IDs.

Recovering more than 10,000 documents from a collection

A query can only return up to 10,000 documents. However, if you want to recover more than 10,000 documents from your collection, you need a way to separate the documents into non-overlapping subgroups. Each subgroup should contain fewer than 10,000 documents that can be returned by a query. Then, you can paginate through the results to retrieve the documents.

Pagination for results is restricted to the maximum of 10,000 documents that are returned by the query. Specifically, the combined use of the count and offset pagination parameters cannot exceed 10,000 documents.

One way to separate the documents into non-overlapping subgroups is to leverage a field that exists in every document and contains a unique value. For example, the SHA-1 field contains a hash of the original source file and is formatted as a hexadecimal string value. You can use the first character of the field as a way of dividing the collection into subgroups. Because SHA-1 contains a hexadecimal value, the first character can have up of 16 possible values (0-9 or a-f). If you filter by the first_char_of (SHA-1) == 0, it might return approximately 1/16 of the entire collection. You can then loop through each of the possible 16 values to get the rest of the documents. If optimum number of documents are not returned in one of the subgroups, you can use the first 2 characters of the SHA-1 field to divide the collection into 256 subgroups instead.

Update how your application handles query results

The way that your application shows query results might need to be updated due to the following differences between the query results document syntax between the v1 and v2 queries:

• At the entity enrichment level, the following information is not supported in v2:

  • Disambiguation
  • Emotion
  • Sentiment

  The Part of Speech enrichment is applied automatically to documents in most project types in v2, but the index fields that are generated by the enrichment are not displayed in the JSON representation of the document.

  

• Instead of the count and relevance in v1, v2 includes the mentions.

  Each entry in the mention corresponds to an occurrence of the entity in the document text. In the following example, seven occurrences are found. For each occurrence, a confidence score and the offsets of the mention text are displayed. You can use the offsets to highlight the mention in the document text when the result is displayed in a user interface.

  

• The JSON structure of query responses is rearranged slightly in v2.

• Deduplication information is not included in the v2 query response.

• In v2, enriched_text is an array instead of an object.

• In Discovery v2, the Entities v2 enrichment is used. Entity type names in v2 are specified in headline case, instead of all uppercase letters. If you use a query or aggregation that specifies an entity name, you must change the capitalization. For example, change PERSON to Person.

• Fields from JSON files that are added to a collection are converted differently during ingestion between v1 and v2. If your application manipulates these results, you might need to make adjustments.

  You can specify the normalizations and conversions objects in the Update a collection method of the API to move or merge JSON fields.

  How JSON source fields are handled

  Original JSON field content	v1 representation	v2 representation	Notes
  "field": null	"field": null	N/A	v1 retains the null value. v2 skips the null field altogether.
  "field": ""	"field": ""	N/A	v1 retains the empty text value. v2 skips the empty text field altogether.
  "field": "value2"	"field": "value2"	"field": "value2"	No difference.
  "field": []	"field": []	N/A	v1 retains the empty array. v2 skips the field with the empty array altogether.
  "field": [ "value4" ]	"field": [ "value4" ]	"field": "value4"	v1 retains the singleton array. v2 converts the singleton array into the value only; it is not stored as part of an array.
  "field": [ 1, 2, 3 ]	"field": [ 1, 2, 3 ]	"field": [ 1, 2, 3 ]	No difference.
  "field": [ "v6", "v7", "v8" ]	"field": [ "v6", "v7", "v8"]	"field": [ "v6", "v7", "v8"]	No difference.