Working with corpora and custom words for next-generation models

Guidelines for adding words to custom models based on improved next-generation models

For custom models that are based on next-generation language models that use improved customization, limit the number of custom words that you add directly to the model with the following methods:

• POST /v1/customizations/{customization_id}/words
• PUT /v1/customizations/{customization_id}/words/{word_name}

Using these methods to add custom words to a custom model can significantly increase the training time of the model. Training time increases linearly with the total number of words that you add. However, the training time increases only when the model is first trained on the new custom words. The time required for subsequent training requests, without new custom words, returns to normal.

Training a custom model with words added only via corpora with the following method is typically quick:

• POST /v1/customizations/{customization_id}/corpora/{corpus_name}

For more information about improved customization for next-generation models, see Improved language model customization for next-generation models.

The non-Japanese models use 'sounds_like' for mapping ('sounds_like' -> 'display_as').

Guidelines for adding words to Japanese models based on improved next-generation models

Do not add custom words for known words, words that are commonly recognized and have a general correspondence between the word and how it is pronounced. Use custom words to create a mapping between how less common words are spelled and how they are pronounced. Also use custom words to add unknown words, those that lack a correspondence between the word and its pronunciation or that are not commonly recognized as words. Adding such words to a corpus is equally effective.

The service enforces a maximum limit of 25 total characters, not including leading or trailing spaces, for custom words and sounds-likes. If you add a custom word or sounds-like that exceeds this limit, the service automatically treats the word as if it were added by a corpus. The word does not appear as a custom word for the model. For the most effective training, it is recommended that Japanese custom words and sounds-likes contain no more than 20 characters. Add long words such as ＩＢＭクラウド音声認識サービス to a corpus.

For example, the pronunciation アイビーエム for the word ＩＢＭ is recognized without customization, so there is no need to add it as a custom word. Because ＩＢＭ, クラウド, 音声認識, and サービス are common words, adding them as custom words has no effect.

For a custom word, enter a commonly used notation that reflects the word's usage and pronunciation. This allows for more efficient customization. For instance, the following example does not reliably produce the string Artificial_Intelligence in response to the utterance エーアイ because Artificial_Intelligence and 人工知能 are not generally uttered as AI:

{\"word\": \"Artificial_Intelligence\", \"sounds_like\": [\"エーアイ\"], \"display_as\": \"Artificial_Intelligence\"},
{\"word\": \"人工知能\", \"sounds_like\": [\"エーアイ\"], \"display_as\": \"Artificial_Intelligence\"}

Because the context is typically something like これからＡＩはますます発展してきます, ＡＩ is the most appropriate notation for the sounds-like エーアイ. The following example is therefore likely to yield better results:

{\"word\": \"ＡＩ\", \"sounds_like\": [\"エーアイ\"], \"display_as\": \"Artificial_Intelligence\"}

Finally, in custom words, half-width alphabetic characters are converted to full-width characters. English uppercase and lowercase characters are treated as different characters.

Preparing a corpus text file

Follow these guidelines to prepare a corpus text file:

• Provide a plain text file that is encoded in UTF-8 if it contains non-ASCII characters. The service assumes UTF-8 encoding if it encounters such characters.

  Make sure that you know the character encoding of your corpus text files. The service preserves the encoding that it finds in the text files. You must use that same encoding when working with custom words in the custom model. For more information, see Character encoding for custom words.

• Use consistent capitalization for words in the corpus. Mix upper- and lowercase letters and use capitalization only when intended.

• Include each sentence of the corpus on its own line, and terminate each line with a carriage return. Including multiple sentences on the same line can degrade accuracy.

• Add personal names as discrete units on separate lines. Do not add the individual elements of a name on separate lines or as individual custom words, and do not include multiple names on the same line of a corpus. The following example shows the correct way to improve recognition accuracy for three names:

  Gakuto Kutara
  Sebastian Leifson
  Malcolm Ingersol
  

  Include additional contextual information where appropriate, for example, Doctor Sebastian Leifson or President Malcolm Ingersol. As with all words, duplicating the names multiple times can improve recognition accuracy.

• Beware of typographical errors. The service assumes that typographical errors are new words. Remember the adage Garbage in, garbage out!

• More sentences result in better accuracy. But the service does limit a model to a maximum of 10 million total words from all sources combined.

What happens when I add a corpus file?

When you add a corpus file, the service analyzes the file's contents. To distill the most meaning from the content, the service tokenizes and parses the data that it reads from a corpus file. The following topics describe how the service parses a corpus file for each supported language.

Information for the following languages is not yet available: Arabic, Chinese, Czech, Hindi, and Swedish. If you need this information for your custom language model, contact your IBM Support representative.

Using the sounds_like field

The sounds_like field specifies how a word is pronounced by speakers in audio. By default, the service does not automatically attempt to generate a sounds-like pronunciation for a word for which you do not provide one. You can add sounds-likes for any words that do not have them. After adding or modifying words, you must validate the words resource to ensure that each word's definition is complete and valid. For more information, see Validating a words resource for next-generation models.

You can provide as many as five alternative pronunciations for a word that is difficult to pronounce or that can be pronounced in different ways. Some possible uses of the field follow:

• Provide different pronunciations for acronyms. For example, the acronym NCAA can be pronounced as it is spelled or colloquially as N C double A The following example adds both of these sounds-like pronunciations for the word NCAA:

  IBM Cloud

  curl -X PUT -u "apikey:{apikey}" \
  --header "Content-Type: application/json" \
  --data "{\"sounds_like\": [\"N C A A\", \"N C double A\"]}" \
  "{url}/v1/customizations/{customization_id}/words/NCAA"
  

  IBM Cloud Pak for Data

  curl -X PUT \
  --header "Authorization: Bearer {token}" \
  --header "Content-Type: application/json" \
  --data "{\"sounds_like\": [\"N C A A\", \"N C double A\"]}" \
  "{url}/v1/customizations/{customization_id}/words/NCAA"
  

  For more information about how the service recognizes acronyms, see Additional transcription efforts.

• Handle foreign words. For example, the French word garçon contains a character that is not found in the English language. You can specify a sounds-like of gaarson, replacing ç with s, to tell the service how English speakers would pronounce the word.

The following topics provide guidelines for specifying a sounds-like pronunciation. Speech recognition uses statistical algorithms to analyze audio, so adding a word does not guarantee that the service transcodes it with complete accuracy. When you add a word, consider how it might be pronounced. Use the sounds_like field to provide various pronunciations that reflect how a word can be spoken.

Information for the following languages is not yet available: Arabic, Chinese, Czech, Hindi, and Swedish. If you need this information for your custom language model, contact your IBM Support representative.

Using the display_as field

The display_as field specifies how a word is displayed in a transcript. By default, the service sets the field to match the spelling of the custom word. The field is intended for cases where you want the service to display a string that is different from the word's spelling. For example, you can indicate that the word hhonors is to be displayed as HHonors.

IBM Cloud

curl -X PUT -u "apikey:{apikey}" \
--header "Content-Type: application/json" \
--data "{\"display_as\": \"HHonors\"}" \
"{url}/v1/customizations/{customization_id}/words/hhonors"

IBM Cloud Pak for Data

curl -X PUT \
--header "Authorization: Bearer {token}" \
--header "Content-Type: application/json" \
--data "{\"display_as\": \"HHonors\"}" \
"{url}/v1/customizations/{customization_id}/words/hhonors"

As another example, you can indicate that the word IBM is to be displayed as IBM™.

IBM Cloud

curl -X PUT -u "apikey:{apikey}" \
--header "Content-Type: application/json" \
--data "{\"display_as\":\"IBM™\"}" \
"{url}/v1/customizations/{customization_id}/words/IBM"

IBM Cloud Pak for Data

curl -X PUT \
--header "Authorization: Bearer {token}" \
--header "Content-Type: application/json" \
--data "{\"display_as\":\"IBM™\"}" \
"{url}/v1/customizations/{customization_id}/words/IBM"

What happens when I add or modify a custom word?

How the service responds to a request to add or modify a custom word depends on the fields and values that you specify. It also depends on whether the character sequences of the word exist in the base model's character sequences.

• Omit both the sounds_like and display_as fields:

  • The service sets the display_as field to the value of the word field. The service does not attempt to set the sounds_like field to a pronunciation of the word.

• Specify only the sounds_like field:

  • If the sounds_like field is valid, the service sets the value of the sounds_like field to the specified value. The service also sets the display_as field to the value of the word field.
  • If the sounds_like field is invalid:
    • The POST /v1/customizations/{customization_id}/words method adds an error field to the word in the model's words resource.
    • The PUT /v1/customizations/{customization_id}/words/{word_name} method fails with a 400 response code and an error message. The service does not add the word to the words resource.

• Specify only the display_as field:

  • The service sets the display_as field to the specified value. The service does not attempt to set the sounds_like field to a pronunciation of the word.

• Specify both the sounds_like and display_as fields:

  • If the sounds_like field is valid, the service sets the sounds_like and display_as fields to the specified values.
  • If the sounds_like field is invalid, the service responds as it does in the case where the sounds_like field is specified but the display_as field is not.

Additional transcription efforts

For custom language models that are based on next-generation models, the service makes additional efforts to ensure the most effective transcription:

• For sounds-like pronunciations for custom words, the service uses the reverse of the sounds-like as well as its definition in a custom word. For example, given a sounds_like field of I triple E for the word IEEE, the service also effectively uses a reverse sounds-like of IEEE for the "word" I triple E. This enhances the application of sounds-like pronunciations for speech recognition. (Note that it is not possible for users to create custom words that contain spaces.)

• For acronyms that are parsed from corpora or defined as custom words, the service makes additional speech recognition efforts. An acronym is any word that consists of two or more consecutive uppercase letters. If the acronym contains one or more vowels, the service attempts to recognize the acronym as a series of individual characters and as a pronounced word.

  For example, the acronym NASA can be read as four individual letters or pronounced as a word that sounds like nassa. The service checks for both during speech recognition. This enhances its ability to represent acronyms correctly in a transcript.