Creating a custom language model

Create a custom language model

You use the POST /v1/customizations method to create a new custom language model. The method accepts a JSON object that defines the attributes of the new custom model as the body of the request. The new custom model is owned by the instance of the service whose credentials are used to create it. For more information, see Ownership of custom models.

You can create a maximum of 1024 custom language models per owning credentials. For more information, see Maximum number of custom models.

A new custom language model has the following attributes:

name (required string)

A user-defined name for the new custom model. Use a localized name that matches the language of the custom model and describes the domain of the model, such as Medical custom model or Legal custom model.

• Include a maximum of 256 characters in the name.
• Do not use backslashes, slashes, colons, equal signs, ampersands, or question marks in the name.
• Use a name that is unique among all custom language models that you own.

base_model_name (required string)

The name of the base language model that is to be customized by the new custom model. You must use the name of a model that is returned by the GET /v1/models method. The new custom model can be used only with the base model that it customizes.

dialect (optional string)

The dialect of the specified language that is to be used with the new custom model. For all languages, it is always safe to omit this field. The service automatically uses the language identifier from the name of the base model. For example, the service automatically uses en-US for all US English models.

If you specify the dialect for a new custom model, follow these guidelines:

• For non-Spanish previous-generation models and for next-generation models, you must specify a value that matches the five-character language identifier from the name of the base model.
• For Spanish previous-generation models, you must specify one of the following values:
  • es-ES for Castilian Spanish (es-ES models)
  • es-LA for Latin American Spanish (es-AR, es-CL, es-CO, and es-PE models)
  • es-US for Mexican (North American) Spanish (es-MX models)

All values that you pass for the dialect field are case-insensitive.

description (optional string)

A recommended description of the new custom model.

• Use a localized description that matches the language of the custom model.
• Include a maximum of 128 characters in the description.

The following example creates a new custom language model named Example model. The model is created for the base model en-US-BroadbandModel and has the description Example custom language model. The required Content-Type header specifies that JSON data is being passed to the method.

IBM Cloud

curl -X POST -u "apikey:{apikey}" \
--header "Content-Type: application/json" \
--data "{\"name\": \"Example model\", \
  \"base_model_name\": \"en-US_BroadbandModel\", \
  \"description\": \"Example custom language model\"}" \
"{url}/v1/customizations"

IBM Cloud Pak for Data

curl -X POST \
--header "Authorization: Bearer {token}" \
--header "Content-Type: application/json" \
--data "{\"name\": \"Example model\", \
  \"base_model_name\": \"en-US_BroadbandModel\", \
  \"description\": \"Example custom language model\"}" \
"{url}/v1/customizations"

The example returns the customization ID of the new model. Each custom model is identified by a unique customization ID, which is a Globally Unique Identifier (GUID). You specify a custom model's GUID with the customization_id parameter of calls that are associated with the model.

{
  "customization_id": "74f4807e-b5ff-4866-824e-6bba1a84fe96"
}

Add a corpus to the custom language model

Once you create your custom language model, the next step is to add domain-specific data to the model. The recommended means of populating a custom model is to add one or more corpora. A corpus is a plain text file that ideally contains sample sentences from your domain.

• For custom models that are based on large speech models, the service parses and extracts word sequences from one or multiple corpora files. The characters help the service learn and predict character sequences from audio. For more information about using corpora with custom models that are based on large speech models, see Working with corpora for large speech models and next-generation models.

• For custom models that are based on previous-generation models, the service parses a corpus file and extracts any words that are not in its base vocabulary. Such words are referred to out-of-vocabulary (OOV) words. For more information about using corpora with custom models that are based on previous-generation models, see Working with corpora for previous-generation models.

• For custom models that are based on next-generation models, the service parses and extracts character sequences from a corpus file. The characters help the service learn and predict character sequences from audio. For more information about using corpora with custom models that are based on next-generation models, see Working with corpora for large speech models and next-generation models.

By providing sentences that include domain-specific words, corpora allow the service to learn the words and character sequences in context. You can also augment and modify a model's words individually. Training a model only on individual words as opposed to words added from corpora is more time-consuming and can produce less effective results.

You use the POST /v1/customizations/{customization_id}/corpora/{corpus_name} method to add a corpus to a custom model:

customization_id (required string)

Specify the customization ID of the custom model to which the corpus is to be added.

corpus_name (required string)

Specify a name for the corpus. Use a localized name that matches the language of the custom model and reflects the contents of the corpus.

• Include a maximum of 128 characters in the name.
• Do not use characters that need to be URL-encoded. For example, do not use spaces, slashes, backslashes, colons, ampersands, double quotes, plus signs, equals signs, questions marks, and so on in the name. (The service does not prevent the use of these characters. But because they must be URL-encoded wherever used, their use is strongly discouraged.)
• Do not use the name of a corpus or grammar that has already been added to the custom model.
• Do not use the name user, which is reserved by the service to denote custom words that are added or modified by the user.
• Do not use the name base_lm or default_lm. Both names are reserved for future use by the service.

Pass the corpus text file as the required body of the request. The following example adds the corpus text file healthcare.txt to the custom model with the specified ID. The example names the corpus healthcare.

IBM Cloud

curl -X POST -u "apikey:{apikey}" \
--data-binary @healthcare.txt \
"{url}/v1/customizations/{customization_id}/corpora/healthcare"

IBM Cloud Pak for Data

curl -X POST \
--header "Authorization: Bearer {token}" \
--data-binary @healthcare.txt \
"{url}/v1/customizations/{customization_id}/corpora/healthcare"

The method also accepts an optional allow_overwrite query parameter that overwrites an existing corpus for a custom model. Use the parameter if you need to update a corpus file after you add it to a model.

The method is asynchronous. It can take on the order of minutes to complete. The duration of the operation depends on the total number of words in the corpus and the current load on the service. For custom models that are based on previous-generation models, the duration also depends on the number of new words that the service finds in the corpus. For more information about checking the status of a corpus, see Monitoring the add corpus request.

You can add any number of corpora to a custom model by calling the method once for each corpus text file. The addition of one corpus must be fully complete before you can add another. A corpus has the status being_processed when you first add it to a model. Its status becomes analyzed when the service finishes processing it.

For custom models that are based on previous-generation models, after the addition of a corpus is complete, examine the new custom words that were extracted from it to check for typographical and other errors. For more information, see Validating a words resource for previous-generation models.

curl -X GET -u "apikey:{apikey}" \
"{url}/v1/customizations/{customization_id}/corpora/corpus1"

curl -X GET \
--header "Authorization: Bearer {token}" \
"{url}/v1/customizations/{customization_id}/corpora/corpus1"

{
  "name": "corpus1",
  "total_words": 5037,
  "out_of_vocabulary_words": 401,
  "status": "analyzed"
}

Add words to the custom language model

Although adding corpora is the recommended means of adding words to a custom language model, you can also add individual custom words to the model directly. The service parses custom words for the custom model just as it does the contents of words from corpora.

If you have only one or a few words to add to a model, using corpora to add the words might not be practical or even viable. The simplest approach is to add a word with only its spelling. But you can also indicate how the word is to be displayed and one or more pronunciations for the word.

• For more information about adding words to a custom model that is based on a previous-generation model, see Working with custom words for previous-generation models.
• For more information about adding words to a custom model that is based on large speech models and next-generation models, see Working with custom words for large speech models and next-generation models.

After you add words to a custom model, examine the new custom words to check for typographical and other errors. This check is especially important when you add multiple words at one time.

• For custom models that are based on previous-generation models, see Validating a words resource for previous-generation models.
• For custom models that are based on large speech models and next-generation models, see Validating a words resource for large speech models and next-generation models.

curl -X PUT -u "apikey:{apikey}" \
--header "Content-Type: application/json" \
--data "{\"sounds_like\": [\"N. C. A. A.\", \"N. C. double A.\"]}" \
"{url}/v1/customizations/{customization_id}/words/NCAA"

curl -X PUT \
--header "Authorization: Bearer {token}" \
--header "Content-Type: application/json" \
--data "{\"sounds_like\": [\"N. C. A. A.\", \"N. C. double A.\"]}" \
"{url}/v1/customizations/{customization_id}/words/NCAA"

curl -X GET -u "apikey:{apikey}" \
"{url}/v1/customizations/{customization_id}"

curl -X GET \
--header "Authorization: Bearer {token}" \
"{url}/v1/customizations/{customization_id}"

{
  "customization_id": "74f4807e-b5ff-4866-824e-6bba1a84fe96",
  "created": "2016-06-01T18:42:25.324Z",
  "updated": "2016-06-01T18:45:11.737Z",
  "language": "en-US",
  "dialect": "en-US",
  "owner": "297cfd08-330a-22ba-93ce-1a73f454dd98",
  "name": "Example model",
  "description": "Example custom language model",
  "base_model_name": "en-US_BroadbandModel",
  "status": "pending",
  "progress": 0
}

Train the custom language model

Once you populate a custom language model with new words (by adding corpora, by adding words directly, or by adding grammars), you must train the model on the new data. Training prepares the custom model to use the data in speech recognition. The model does not use words that you add via any means until you train it on the data.

You use the POST /v1/customizations/{customization_id}/train method to train a custom model. You pass the method the customization ID of the model that you want to train, as in the following example:

IBM Cloud

curl -X POST -u "apikey:{apikey}" \
"{url}/v1/customizations/{customization_id}/train"

IBM Cloud Pak for Data

curl -X POST \
--header "Authorization: Bearer {token}" \
"{url}/v1/customizations/{customization_id}/train"

The method is asynchronous. Training can take on the order of minutes to complete depending on the number of new words on which the model is being trained and the current load on the service. For more information about checking the status of a training operation, see Monitoring the train model request.

The method includes the following optional query parameters:

• The word_type_to_add parameter specifies the words on which the custom model is to be trained:

  • Specify all or omit the parameter to train the model on all of its words, regardless of their origin.
  • Specify user to train the model only on words that were added or modified by the user, ignoring words that were extracted only from corpora or grammars.

  For custom models that are based on previous-generation models, this option is useful if you add corpora with noisy data, such as words that contain typographical errors. Before training the model on such data, you can use the word_type query parameter of the GET /v1/customizations/{customization_id}/words method to review words that are extracted from corpora and grammars. For more information, see Listing custom words from a custom language model.

  For custom models that are based on large speech models and next-generation models, the service ignores the word_type_to_add parameter. The words resource contains only custom words that the user adds or modifies directly, so the parameter is unnecessary.

• The customization_weight parameter specifies the relative weight that is given to words from the custom model as opposed to words from the base vocabulary when the custom model is used for speech recognition. You can also specify a customization weight with any recognition request that uses the custom model. For more information, see Using customization weight.

• The strict parameter indicates whether training is to proceed if the custom model contains a mix of valid and invalid resources (corpora, words, and grammars). By default, training fails if the model contains one or more invalid resources. Set the parameter to false to allow training to proceed as long as the model contains at least one valid resource. The service excludes invalid resources from the training. For more information, see Training failures for custom language models.

curl -X GET -u "apikey:{apikey}" \
"{url}/v1/customizations/{customization_id}"

curl -X GET \
--header "Authorization: Bearer {token}" \
"{url}/v1/customizations/{customization_id}"

{
  "customization_id": "74f4807e-b5ff-4866-824e-6bba1a84fe96",
  "created": "2016-06-01T18:42:25.324Z",
  "updated": "2016-06-01T18:45:11.737Z",
  "language": "en-US",
  "dialect": "en-US",
  "owner": "297cfd08-330a-22ba-93ce-1a73f454dd98",
  "name": "Example model",
  "description": "Example custom language model",
  "base_model_name": "en-US_BroadbandModel",
  "status": "training",
  "progress": 0
}