Keyword spotting and word alternatives

Keyword spotting results

The service returns the results in a keywords_result field that is an element of the results array. The keywords_result field is a dictionary, or associative array, of enumerable properties. Each property is identified by a specified keyword and includes an array of objects. The service returns one element of the array for each match that it finds for the keyword. The object for each match includes the following fields:

• normalized_text is the specified keyword that is normalized to the spoken phrase that matched in the input audio.
• start_time is the start time in seconds of the match.
• end_time is the end time in seconds of the match.
• confidence is the service's confidence that the match represents the specified keyword. The confidence must be at least as great as the specified threshold to be included in the results.

A keyword for which the service finds no matches is omitted from the array. A keyword might not be found if

• The audio simply does not include the keyword. Absence of the keyword is the most obvious explanation.

• The threshold is set too high. The service might identify the keyword but with a lower level of confidence, in which case it omits the match from the results.

• A keyword string that contains multiple tokens (for example, Speech to Text) is spoken with too much silence between its tokens. When the service transcribes audio, it chops the stream into a series of blocks. Each block represents a continuous chunk of audio that does not have an interval of silence that exceeds half of a second. It constructs an array of result objects that consists of these blocks.

  The service matches a multi-token keyword only if

  • The keyword's tokens are in the same block.
  • The tokens are either adjacent or separated by a gap of no more than 0.1 seconds.

  In the latter case, a brief filler or non-lexical utterance, such as "uhm" or "well," can lie between two tokens of the keyword. In this case, previous-generation models can insert a hesitation marker at this position of the transcript. Next-generation models include the actual hesitation in the transcription results. For more information, see Speech hesitations and hesitation markers.

Keyword spotting example

The following example request sets the keywords parameter to a URL-encoded array of three strings (colorado, tornado, and tornadoes) and the keywords_threshold parameter to 0.5:

IBM Cloud

curl -X POST -u "apikey:{apikey}" \
--header "Content-Type: audio/flac" \
--data-binary @{path}audio-file.flac \
"{url}/v1/recognize?keywords=colorado%2Ctornado%2Ctornadoes&keywords_threshold=0.5"

IBM Cloud Pak for Data

curl -X POST \
--header "Authorization: Bearer {token}" \
--header "Content-Type: audio/flac" \
--data-binary @{path}audio-file.flac \
"{url}/v1/recognize?keywords=colorado%2Ctornado%2Ctornadoes&keywords_threshold=0.5"

The service finds qualifying occurrences of colorado and tornadoes:

{
  "result_index": 0,
  "results": [
    {
      "keywords_result": {
        "colorado": [
          {
            "normalized_text": "Colorado",
            "start_time": 4.94,
            "confidence": 0.91,
            "end_time": 5.62
          }
        ],
        "tornadoes": [
          {
            "normalized_text": "tornadoes",
            "start_time": 1.52,
            "confidence": 1.0,
            "end_time": 2.15
          }
        ]
      },
      "alternatives": [
        {
          "confidence": 0.96,
          "transcript": "several tornadoes touch down as a line of
severe thunderstorms swept through Colorado on Sunday "
        }
      ],
      "final": true
    }
  ]
}

Word alternatives results

The service returns the results in a word_alternatives field that is an element of the results array. The word_alternatives field is an array of objects, each of which provides the following fields for an alternative word:

• start_time indicates the time in seconds at which the word for which hypotheses are returned starts in the input audio.
• end_time indicates the time in seconds at which the word for which hypotheses are returned ends in the input audio.
• alternatives is an array of hypothesis objects. Each object includes a confidence that indicates the service's confidence score for the hypothesis and a word that identifies the hypothesis. The confidence must be at least as great as the specified threshold to be included in the results. The service orders the alternatives by confidence.

Word alternatives example

The following example request specifies a rather low word_alternatives_threshold of 0.10. The simple input audio includes words with common homonyms and different possible acoustic interpretations.

IBM Cloud

curl -X POST -u "apikey:{apikey}" \
--header "Content-Type: audio/wav" \
--data-binary @{path}audio-file.wav \
"{url}/v1/recognize?word_alternatives_threshold=0.10"

IBM Cloud Pak for Data

curl -X POST \
--header "Authorization: Bearer {token}" \
--header "Content-Type: audio/wav" \
--data-binary @{path}audio-file.wav \
"{url}/v1/recognize?word_alternatives_threshold=0.10"

Because of the low threshold, the service returns multiple hypotheses and confidence scores for some of the words, along with the start and end times of all words. The final transcript correctly recognizes the input audio.

{
   "result_index": 0,
   "results": [
      {
         "final": true,
         "alternatives": [
            {
               "transcript": "yes I ate that tuna ",
               "confidence": 0.82
            }
         ],
         "word_alternatives": [
            {
               "start_time": 0.0,
               "end_time": 0.31,
               "alternatives": [
                  {
                     "word": "yes",
                     "confidence": 1.0
                  }
               ]
            },
            {
               "start_time": 0.31,
               "end_time": 0.46,
               "alternatives": [
                  {
                     "word": "I",
                     "confidence": 1.0
                  }
               ]
            },
            {
               "start_time": 0.46,
               "end_time": 0.63,
               "alternatives": [
                  {
                     "word": "ate",
                     "confidence": 0.89
                  },
                  {
                     "word": "eat",
                     "confidence": 0.11
                  }
               ]
            },
            {
               "start_time": 0.63,
               "end_time": 0.77,
               "alternatives": [
                  {
                     "word": "that",
                     "confidence": 0.72
                  },
                  {
                     "word": "the",
                     "confidence": 0.27
                  }
               ]
            },
            {
               "start_time": 0.77,
               "end_time": 1.2,
               "alternatives": [
                  {
                     "word": "tuna",
                     "confidence": 0.94
                  }
               ]
            }
         ]
      }
   ]
}