Handling phone interactions

Adding phone-specific responses to your dialog or actions

To initiate a voice-specific interaction from a dialog node or a step in an action, add a response within the output.generic array using the appropriate response type.

Although many response types can be specified using the Watson Assistant user interface, phone-specific response types must currently be added using the JSON editor.

For more information about using the JSON editor to add responses, see Defining responses using the JSON editor.

Applying advanced settings to the Speech to Text service

Use the speech_to_text response type to send configuration commands to the Speech to Text service instance used by the phone integration. By sending a speech_to_text response from a dialog node or action step, you can dynamically change the Speech to Text configuration during a conversation.

By default, any Speech to Text configuration changes you make persist for the remainder of the conversation, or until you update them again. You can change this behavior by specifying the update_strategy property of the parameters object.

The format of the speech_to_text response type is as follows:

{
  "output": {
    "generic": [
      {
        "response_type": "speech_to_text",
        "command_info": {
          "type": "<command type>",
          "parameters": {
            "parameter name1": "parameter value",
            "parameter name2": "parameter value"
          }
        }
      }
    ]
  }
}

Each command type along with its related parameters are described in the following sections.

{
  "output": {
    "generic": [
      {
        "response_type": "speech_to_text",
        "command_info": {
          "type": "configure",
          "parameters": {
            "narrowband_recognize": {
              "x-watson-learning-opt-out": true,
              "model": "en-US_NarrowbandModel",
              "profanity_filter": true,
              "smart_formatting": true,
              "language_customization_id": "81d3630-ba58-11e7-aa4b-41bcd3f6f24d",
              "acoustic_customization_id": "e4766090-ba51-11e7-be33-99bd3ac8fa93"
            }
          }
        }
      }
    ]
  }
}

{
  "output": {
    "generic": [
      {
        "response_type": "speech_to_text",
        "command_info": {
          "type": "configure",
          "parameters": {
            "update_strategy": "merge_once",
            "narrowband_recognize": {
              "x-watson-learning-opt-out": true,
              "grammar_name": "names-abnf",
              "language_customization_id": "81d3630-ba58-11e7-aa4b-41bcd3f6f24d"
            }
          }
        }
      }
    ]
  }
}

{
  "output": {
    "generic": [
      {
        "response_type": "speech_to_text",
        "command_info": {
          "type": "configure",
          "parameters": {
            "narrowband_recognize": {
              "model": "es-ES_NarrowbandModel",
              "smart_formatting": true
            }
          }
        }
      }
    ]
  }
}

{
  "output": {
    "generic": [
      {
        "response_type": "speech_to_text",
        "command_info": {
          "type": "configure",
          "parameters": {
            "update_strategy": "merge_once",
            "narrowband_recognize": {
              "language_customization_id": "ao45vohgFuxyOQRgztu-02I10ut7aJcM-AdInT-VWgj3V" 
            }
          }
        }
      }
    ]
  }
}

Applying advanced settings to the Text to Speech service

Use the text_to_speech response type to send configuration commands to the Text to Speech service instance used by the phone integration. By sending a text_to_speech response from a dialog node or action step, you can dynamically change the Text to Speech configuration during a conversation.

By default, any Text to Speech configuration changes you make persist for the remainder of the conversation, or until you update them again. You can change this behavior by specifying the update_strategy property of the parameters object.

The format of the speech_to_text response type is as follows:

{
  "output": {
    "generic": [
      {
        "response_type": "text_to_speech",
        "command_info": {
          "type": "<command type>",
          "parameters": {
            "parameter name": "parameter value",
            "parameter name": "parameter value"
          }
        }
      }
    ]
  }
}

Each command type along with its related parameters are described in the following sections.

{
  "output": {
    "generic": [
      {
        "response_type": "text_to_speech",
        "command_info": {
          "type": "configure",
          "parameters": {
            "synthesize": {
              "voice": "en-GB_KateV3Voice"
            }
          }
        }
      }
    ]
  }
}

Transferring a call to a human agent

When you configure the phone integration, you can optionally set up backup call center support, which makes it possible for the assistant to transfer a call to a human agent. You can use the Connect to human agent response type in a dialog node or action step to initiate a transfer to a human agent at a specific point in the conversation. When a Connect to human agent response is sent to the phone integration, a SIP transfer is initiated using the SIP REFER message, as defined by RFC 5589.

For more information about initiating a transfer to a human agent during the conversation, see the following documentation:

• Adding Connect to human agent response type (dialog node)
• Deciding what to do next (action step)

The phone integration supports additional parameters for the Connect to human agent response type. You can add these phone-specific parameters to the connect_to_agent response type using the JSON editor.

The connect_to_agent response type supports the ability to specify the target transfer information under the transfer_info parameter.

The following example shows a transfer that uses all of the configurable parameters:

{
  "output": {
    "generic": [
      {
        "response_type": "connect_to_agent",
        "transfer_info": {
          "target": {
            "service_desk": {
              "sip": {
                "uri": "sip:user\\@domain.com",
                "transfer_headers": [
                  {
                    "name": "Customer-Header1",
                    "value": "Some-Custom-Info"
                  },
                  {
                    "name": "User-to-User",
                    "value": "XXXXXX"
                  }
                ],
                "transfer_headers_send_method": "refer_to_header"
              }
            }
          }
        },
        "agent_available": {
          "message": "I'll transfer you to an agent"
        },
        "agent_unavailable": {
          "message": "Sorry, I could not find an agent."
        },
        "message_to_human_agent": "The caller needs help resetting their password"
      }
    ]
  }
}

The connect_to_agent response type supports the following phone-specific properties.

If you define a SIP URI as the transfer target, escape the at sign (@) in the URI by adding two backslashes (\\) in front of it. This is to prevent the string from being recognized as part of the entity shorthand syntax.

    "uri": "sip:12345556789\\@myhost.com"

REFER sip:b@atlanta.example.com SIP/2.0
Via: SIP/2.0/UDP agenta.atlanta.example.com;branch=z9hG4bK2293940223
To: <sip:b@atlanta.example.com>
From: <sip:a@atlanta.example.com>;tag=193402342
Call-ID: 898234234@agenta.atlanta.example.com
CSeq: 23 REFER
Max-Forwards: 7
Refer-To: sip:user@domain.com
X-Watson-Assistant-Token: 8f817472-8c57-4117-850d-fdf4fd23ba7
User-to-User: dev::latest::212033::0a64c30d-c558-4055-85ad-ef75ad6cc29d::978f1fd7-4e24-47d8-adb0-24a8a6eff69e::b5ffd6c2-902f-4658-b586-e3fc170a6cf3::7ad616a350cc48078f17e3ee3df551de;encoding=ascii
Contact: sip:a@atlanta.example.com
Content-Length: 0

REFER sip:b@atlanta.example.com SIP/2.0
Via: SIP/2.0/UDP agenta.atlanta.example.com;branch=z9hG4bK2293940223
To: <sip:b@atlanta.example.com>
From: <sip:a@atlanta.example.com>;tag=193402342
Call-ID: 898234234@agenta.atlanta.example.com
CSeq: 93809823 REFER
Max-Forwards: 70
Refer-To: sip:user@domain.com
User-to-User: 637573746f6d2d757365722d746f2d75736572;encoding=hex;
X-Watson-Assistant-Session-History-Key: dev::latest::212033::0a64c30d-c558-4055-85ad-ef75ad6cc29d::978f1fd7-4e24-47d8-adb0-24a8a6eff69e::b5ffd6c2-902f-4658-b586-e3fc170a6cf3::7ad616a350cc48078f17e3ee3df551de
Contact: sip:a@atlanta.example.com
Content-Length: 0

REFER sip:b@atlanta.example.com SIP/2.0
Via: SIP/2.0/UDP agenta.atlanta.example.com;branch=z9hG4bK2293940223
To: <sip:b@atlanta.example.com>
From: <sip:a@atlanta.example.com>;tag=193402342
Call-ID: 898234234@agenta.atlanta.example.com
CSeq: 23 REFER
Max-Forwards: 70
Refer-To: sip:user@domain.com?User-to-User=dev::latest::893499::dff9c274-adc4-4f63-93de-781166760bf8::978f1fd7-4e24-47d8-adb0-24a8a6eff69e::b5ffd6c2-902f-4658-b586-e3fc170a6cf3::7ad616a350cc48078f17e3ee3df551de%3Bencoding%3Dascii
Contact: sip:a@atlanta.example.com
Content-Length: 0

REFER sip:b@atlanta.example.com SIP/2.0
Via: SIP/2.0/UDP agenta.atlanta.example.com;branch=z9hG4bK2293940223
To: <sip:b@atlanta.example.com>
From: <sip:a@atlanta.example.com>;tag=193402342
Call-ID: 898234234@agenta.atlanta.example.com
CSeq: 93809823 REFER
Max-Forwards: 70
Refer-To: sip:user@domain.com?User-to-User=637573746f6d2d757365722d746f2d75736572%3Bencoding%3Dhe&X-Watson-Assistant-Session-History-Key=dev::latest::893499::dff9c274-adc4-4f63-93de-781166760bf8::978f1fd7-4e24-47d8-adb0-24a8a6eff69e::b5ffd6c2-902f-4658-b586-e3fc170a6cf3::7ad616a350cc48078f17e3ee3df551de
Contact: sip:a@atlanta.example.com
Content-Length: 0

Playing hold music or a voice recording

To play hold music or to play a recorded message, use the audio response type. This response type can be specified in a dialog node or action step using the JSON editor.

You cannot play hold music during a call transfer. However, you might want to play hold music if your assistant needs time to perform processing of some kind, such as calling a client-side action or making a call to a webhook.

The phone integration supports the following properties for the audio response type:

If you set channel_options.voice_telephony.loop to true, add a user-defined response with the vgwActForceNoInputTurn command. This command instructs the phone integration to initiate a turn with a vgwNoInputTurn text without waiting for an input from the caller. In the vgwNoInputTurn turn you can initiate a transaction while the caller is on hold. When the vgwNoInputTurn turn completes, the looped audio stops.

The following example shows an audio response with loop=true, and a user_defined response specifying the vgwActForceNoInputTurn command.

{
  "output": {
    "generic": [
      {
        "user_defined": {
          "vgwAction": {
            "command": "vgwActForceNoInputTurn"
          }
        },
        "response_type": "user_defined"
      },
      {
        "source": "https://upload.wikimedia.org/wikipedia/commons/d/d8/Random_composition3.wav",
        "response_type": "audio",
        "channel_options": {
          "voice_telephony": {
            "loop": true
          }
        }
      }
    ]
  }
}

Enabling keypad entry

If you want customers to be able to send information by typing it on their phone keypad instead of speaking, you can add support for phone keypad entry. The best way to implement this type of support is to enable dual-tone multifrequency (DTMF) signaling. DTMF is a protocol used to transmit tones that are generated when a user presses keys on a push-button phone. The tones have a specific frequency and duration that can be interpreted by the phone network.

To start listening for tones as the user presses phone keys, use the dtmf response type in a dialog node or action step. This response type can be added using the JSON editor.

{
  "output": {
    "generic": [
      {
        "response_type": "dtmf",
        "command_info": {
          "type": "<command type>",
          "parameters": {
            "parameter name": "parameter value",
            "parameter name": "parameter value"
          }
        },
        "channels": [
          {
            "channel": "voice_telephony"
          }
        ]
      }
    ]
  }
}

The command_info property specifies a DTMF command for the phone integration. The supported commands and their related parameters are as follows.

{
  "output": {
    "generic": [
      {
        "response_type": "dtmf",
        "command_info": {
          "type": "collect",
          "parameters": {
            "termination_key": "#",
            "count": 16,
            "ignore_speech": true
          }
        },
        "channels": [
          {
            "channel": "voice_telephony"
          }
        ]
      }
    ]
  }
}

{
  "output": {
    "generic": [
      {
        "response_type": "dtmf",
        "command_info": {
          "type": "send",
          "parameters": {
            "digits": [
              {
                "code": "9",
                "volume": -8
              },
              {
                "code": "11"
              }
            ],
            "send_interval": 100
          }
        },
        "channels": [
          {
            "channel": "voice_telephony"
          }
        ]
      }
    ]
  }
}

Transferring the conversation to the web chat integration

You can transfer the caller from the current phone call to a web chat session by using the channel_transfer response type. The assistant sends an SMS message to the caller that includes a URL that the caller can tap to load the web chat widget in the phone's browser. The web chat session displays the history of the phone call and can start the process of collecting information needed to complete the transaction. This can be useful in situations where the customer can provide information more easily in writing than by speaking (for example, changing an address).

After the transfer has successfully completed, the caller can hang up the phone and continue the copnversation using the web chat. The channel_transfer response type can be used with the phone integration only if the SMS with Twilio integration is also configured for the assistant.

{
  "output": {
    "generic": [
      {
        "response_type": "text",
        "values": [
          {
            "text": "I will send you a text message now with a link to our website."
          }
        ],
        "selection_policy": "sequential"
      },
      {
        "response_type": "channel_transfer",
        "message_to_user": "Click the link to connect with an agent using our website.",
        "transfer_info": {
          "target": {
            "chat": {
              "url": "https://example.com/webchat"
            }
          }
        }
      }
    ]
  }
}

Ending the call

You can instruct your assistant end a phone call by using the end_session response type, as shown in this example.

{
  "output": {
    "generic": [
      {
        "response_type": "end_session"
      }
    ]
  }
}

You can optionally include custom headers to include with the SIP BYE request that is generated when the phone integration receives this response type.

This example shows the end_session response type with custom SIP headers:

{
  "output": {
    "generic": [
      {
        "response_type": "end_session",
        "channel_options": {
          "voice_telephony": {
            "sip": {
              "headers": [
                {
                  "name": "Customer-Header1",
                  "value": "Some-Custom-Info"
                },
                {
                  "name": "User-to-User",
                  "value": "XXXXXX"
                }
              ]
            }
          }
        }
      }
    ]
  }
}

Sending a text message during a phone conversation

There are some situations when it is useful to be able to send a text message during an ongoing voice. For example, you might want the customer to specify a street address, which is easier to communicate accurately in writing than by transcribing voice input.

Before you can send SMS messages during a phone call, you must set up the SMS with Twilio integration. For more information, see Integrating with SMS with Twilio.

When you exchange a text with a customer during a conversation, the dialog initiates the SMS message exchange. It sends a text message to the user and asks for the user to respond to it.

To send a specific message from a dialog node or action step, use the user_defined response type with the vgwActSendSMS command:

{
  "output": {
    "generic": [
      {
        "response_type": "text",
        "values": [
          {
            "text": "I will send you a text message now."
          }
        ],
        "selection_policy": "sequential"
      },
      {
        "response_type": "user_defined",
        "user_defined": {
          "vgwAction": {
            "command": "vgwActSendSMS",
            "parameters": {
              "message": "Hey, this is Watson Assistant. To send me your street address, respond to this text message with your address."
            }
          }
        }
      }
    ]
  }
}

You can specify any of the following parameters in the parameters object:

If your SMS with Twilio integration supports more than one SMS phone number, or you are using a non-Twilio SIP trunk, be sure to specify the phone number that you want to use to send the text message. Otherwise, the text is sent using the same phone number that was called.

After the assistant receives an SMS message, a new conversation turn is initiated with the text input vgwSMSMessage. This input indicates that a message was received from the caller. The text of the customer's message is included as the value of the vgwSMSMessagecontext variable.

If the assistant is unable to send an SMS message to the caller, a new turn is initiated with the text input vgwSMSFailed. This input indicates that an SMS message could not be sent the caller. You can design your dialog or actions to handle such a failure by creating intents or actions that are triggered by the input text vgwSMSFailed.

{
  "input": {
    "message_type": "text",
    "text": "vgwSMSMessage"
  },
  "context": {
    "skills": {
      "main skill": {
        "user_defined": {
          "vgwSMSMessage": "1545 Lexington Ave."
        }
      }
    }
}

Defining a sequence of phone commands

If you want to run more than one command in succession, include multiple responses in the output.generic array. These commands are processed in the order in which they are specified in the array.

This example shows two responses: first a text response, followed by an end_session response to end the call.

{
  "output": {
    "generic": [
      {
        "values": [
          {
            "text": "Goodbye."
          }
        ],
        "response_type": "text",
        "selection_policy": "sequential"
      },
      {
        "response_type": "end_session"
      }
    ]
  }
}