Understanding speech recognition results

The alternatives field

The alternatives field provides an array of transcription results. For this request, the array includes just one element.

• The transcript field provides the results of the transcription.
• The confidence field is a score that indicates the service's confidence in the transcript, which for this example exceeds 90 percent.

The final and result_index fields qualify the meaning of these fields.

Internal changes and improvements to the service can affect transcripts and confidence scores. For example, speech recognition may be improved to return more precise transcription results. Similarly, transcript and word confidence scores might change slightly as a result of improved speech recognition. Such changes are expected to be modest, but do not expect transcripts and confidence scores to remain unchanged over time.

The final field

The final field indicates whether the transcript shows final transcription results:

• The field is true for final results, which are guaranteed not to change. The service sends no further updates for final results.
• The field is false for interim results, which are subject to change. If you use the interim_results parameter with the WebSocket interface, the service returns evolving hypotheses in the form of multiple results fields as it transcribes the audio. For interim results, the final field is always false and the confidence field is always omitted.

For more information about using the WebSocket interface to obtain interim results with large speech models, previous- and next-generation models, see the following topics:

• Interim results
• Requesting interim results and low latency
• How the service sends recognition results

The result_index field

The result_index field provides an identifier for the results that is unique for that request. If you request interim results, the service sends multiple results fields for evolving hypotheses of the input audio. The indexes for interim results for the same audio always have the same value, as do the final results for the same audio.

The same index can also be used for multiple final results of a single request. Regardless of whether you request interim results, the service can return multiple final results with the same index if your audio includes pauses or extended periods of silence. For more information, see Pauses and silence.

Once you receive final results for any audio, the service sends no further results with that index for the remainder of the request. The index for any further results is incremented by one.

If your audio produces multiple final results, concatenate the transcript elements of the final results to assemble the complete transcription of the audio. Assemble the results in the order in which you receive them. When you assemble a complete final transcript, you can ignore interim results for which the final field is false.

Additional response content

Many speech recognition parameters impact the contents of the service's response. Some parameters can cause the service to return multiple transcription results:

• end_of_phrase_silence_time
• interim_results
• split_transcript_at_phrase_end

Some parameters can modify the contents of a transcript:

• profanity_filter
• redaction
• smart_formatting

Other parameters can add more information to the results:

• audio_metrics
• keywords and keywords_threshold
• max_alternatives
• processing_metrics and processing_metrics_interval
• speaker_labels
• timestamps
• word_alternatives_threshold
• word_confidence

For more information about the available parameters, see Using speech recognition parameters and the Parameter summary.

Hesitations for previous-generation models

For previous-generation models, the service includes hesitation markers in transcription results for most languages. Different languages can use different hesitation markers or not indicate hesitations at all:

• For US English, hesitation markers are indicated by the token %HESITATION. Words that generate hesitation markers are aah, ah, hm, hmm, huh, huh-uh, hum, ohh, ugh, uh, uh-huh, uh-oh, uh-uh, um, and um-hum.
• For Japanese, hesitation markers typically begin with D_.
• For Spanish, the service does not produce hesitation markers.

The following example shows the token %HESITATION for a US English transcript:

{
  "result_index": 0,
  "results": [
    {
      "alternatives": [
        {
          "confidence": 0.99,
          "transcript": ". . . that %HESITATION that's a . . ."
        }
      ],
      "final": true
    }
  ]
}

Hesitation markers can appear in both interim and final results. Enabling smart formatting prevents hesitation markers from appearing in final results for US English. For more information, see Smart formatting.

Hesitation markers can also appear in other fields of a transcript. For example, if you request Word timestamps for the individual words of a transcript, the service reports the start and end time of each hesitation marker.

{
  "result_index": 0,
  "results": [
    {
      "alternatives": [
        {
          "timestamps": [
            . . .
            [
              "that",
              7.31,
              7.69
            ],
            [
              "%HESITATION",
              7.69,
              7.98
            ],
            [
              "that's",
              7.98,
              8.41
            ],
            [
              "a",
              8.41,
              8.48
            ],
            . . .
          ],
          "confidence": 0.99,
          "transcript": ". . . that %HESITATION that's a . . ."
        }
      ],
      "final": true
    }
  ]
}

Unless you need to use them for your application, you can safely filter hesitation markers from a transcript.

Hesitations for next-generation models

For next-generation models, the service includes the actual hesitations words in all transcription results. Next-generation models treat hesitations as words, so hesitations can appear in interim results, final results, and other fields such as the results for word timestamps. Next-generation models do not produce hesitation markers, and enabling smart formatting does not cause hesitations to be removed from final results. Different languages can identify different hesitation words:

• For US English, common hesitation words, as for previous-generation models, are aah, ah, hm, hmm, huh, huh-uh, hum, ohh, ugh, uh, uh-huh, uh-oh, uh-uh, um, and um-hum. Not all of these hesitation words might appear in transcripts.
• For Japanese, hesitation words typically consist of half-width characters such as ｱ, ｱﾉｰ, ｳｰﾝ, ｴｰﾄ, ﾏ, ﾃ, ﾏ, and ﾝｰﾄ. Some hesitations might be recognized as full-width characters.

To increase the likelihood of seeing hesitations in your response, you can use a custom language model. In the custom model, add corpora that include the hesitations or create custom words whose sounds-likes capture the way users say the disfluencies. For more information about custom language models, see Creating a custom language model.

The following example shows the hesitation "uhm" in a US English transcript:

{
  "result_index": 0,
  "results": [
    {
      "alternatives": [
        {
          "confidence": 0.99,
          "transcript": ". . . that uhm that's a . . ."
        }
      ],
      "final": true
    }
  ]
}