Inferring switching conditions for switching between modalities in a speech application environment extended for interactive text exchanges

• US 8,000,969 B2
• Filed: 12/19/2006
• Issued: 08/16/2011
• Est. Priority Date: 12/19/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Mandalia, Baiju D., Moore, Victor S., Nusbickel, Wendi L., Da Palma, William V.
• Primary Examiner(s): Rider; Justin W
• Application Number:
  US11/613,176
• Publication Number:
  US 20080147407A1
• Time in Patent Office: 1,701 Days
• Field of Search: 704/270
• US Class Current: 704/270
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

The disclosed solution includes a method for dynamically switching modalities based upon inferred conditions in a dialogue session involving a speech application. The method establishes a dialogue session between a user and the speech application. During the dialogue session, the user interacts using an original modality and a second modality. The speech application interacts using a speech modality only. A set of conditions indicative of interaction problems using the original modality can be inferred. Responsive to the inferring step, the original modality can be changed to the second modality. A modality transition to the second modality can be transparent the speech application and can occur without interrupting the dialogue session. The original modality and the second modality can be different modalities; one including a text exchange modality and another including a speech modality.

Claims

1. A method for inferential switching between voice and text exchange modalities during communication sessions involving a voice server comprising:

detecting at least one condition indicative of interaction problems with a current modality of a communication session between a text exchange client and a speech application, wherein, in the current modality, automatic text-to-voice and voice-to-text conversions between text exchange client and the speech application occur in a manner transparent to the speech application;

inferring a need to switch modalities based upon the detecting of the at least one condition; and

performing at least one programmatic action relating to switching modalities,

wherein the at least one condition includes at least three conditions selected from a group of conditions consisting of inappropriate text being entered by the user that indicates user frustration, excessively long textual input being entered by the user, long delays between consecutive user inputs, and detecting received input is out of context.

2. The method of claim 1, wherein the programmatic action results in a voice connection being established between the speech application and a voice client, wherein the voice connection permits a user to input voice messages and to receive voice messages, wherein a dialogue state for the communication session remains unchanged for the voice client from a state existing for the text exchange client before the voice connection was established.

3. The method of claim 2, wherein the programmatic action automatically closes a communication link between the speech application and the text exchange client.

4. The method of claim 2, wherein the voice connection is a separate communication channel from that used for the text exchange client, and wherein the voice client is a telephone.

5. The method of claim 1, wherein the programmatic action sends a hyperlink to the text exchange client, wherein the hyperlink, when clicked, opens a voice connection between a voice client and the speech application, wherein a dialogue state for the communication session remains unchanged for the voice client from a state existing for the text exchange client before the voice connection was established.

6. The method of claim 1, wherein the text exchange client is an off-the-shelf unmodified client, and wherein the speech application is an unmodified VoiceXML based application that lacks an inherent text exchange capability.

7. A method for dynamically switching modalities based upon inferred conditions in a dialogue session involving a voice server comprising:

establishing a dialogue session between a user and a speech application, wherein the dialogue session, when established, allows the user to interact with the speech application using an original modality;

automatically inferring a set of conditions indicative of interaction problems using the original modality; and

responsive to the inferring step, changing the original modality to a second modality that is different from the original modality, wherein a modality transition to the second modality is transparent to the speech application, wherein the original modality and the second modality are selected from a group of modalities consisting of a speech modality, a text exchange modality, and a multi mode modality that includes a speech modality and a text exchange modality, and wherein one of the original modality and the second modality is the text exchange modality,

wherein communications during the dialogue session involve a Chatbot server, a text exchange client associated with the text modality, and a voice client associated with the voice modality, wherein, for portions of the dialogue session involving the text exchange client and the speech application, the voice server uses a text interface that permits a voice server to directly consume text input, wherein the Chatbot server dynamically converts voice markup into text that is provided to the text exchange client.

8. The method of claim 7, wherein the original modality is a text exchange modality, and wherein the set of conditions include at least one condition selected from a group of conditions consisting of inappropriate text being entered by the user that indicates user frustration, excessively long textual input being entered by the user, long delays between consecutive user inputs, and detecting received input is out of context.

9. The method of claim 7, wherein the second modality is a voice modality involving a voice device associated with a phone number registered within a user profile;

automatically searching the profile for the phone number; and

automatically dialing the phone number to establish a new voice channel with the voice device during the changing step.

10. The method of claim 7, wherein the original modality is a text exchange modality established between the speech application and a text exchange client over a text exchange channel, and wherein the second modality is a speech modality established between the speech application and a voice device over a voice channel, said voice device being a different device than the text exchange client, and wherein the voice device is associated with a phone number, said changing step further comprising:

automatically calling the phone number to establish the voice channel.

11. The method of claim, 7 further comprising:

when the set of conditions are inferred, automatically prompting the user to switch from the original modality to the second modality, wherein the changing step is performed responsive to a positive user response to the prompting step.

12. The method of claim 11, the prompting step further comprising:

prompting the user to selectively decide to continue interactions involving the original modality when switching to the second modality, wherein depending upon the user response, the original modality is selectively retained after the changing step, wherein when the original modality is retained, interactions between the user and the speech application occur using either the second modality or the original modality.

13. The method of claim 7, wherein a text exchange client is used by the user when communicating via the text exchange modality, wherein the text exchange client is an off-the-shelf unmodified client, and wherein the speech application is a VoiceXML based application that lacks an inherent text exchange capability.

14. The method of claim 7, wherein said steps of claim 7 are steps performed automatically be at least one machine in accordance with at least one computer program having a plurality of code sections that are executable by the at least one machine.

15. A system for providing text exchange services comprising:

a text exchange client configured to send and receive real-time text exchanges over a text exchange channel;

a voice client configured to send and receive speech over a voice channel;

a speech application environment configured to execute an automated speech response application that permits users to interact in real-time;

a Chatbot server configured to establish a communication session involving that text exchange client and the speech automated speech response application, wherein the voice server uses a text interface that permits the voice server to directly consume text input, wherein the Chatbot server dynamically converts voice markup to text that is provided to the text exchange client during the communication session in a manner transparent to the text exchange client and to the automated speech response application;

a modality switching engine configured to dynamically switch a modality through with a user interacts with the automated speech response application, wherein a dynamic modality switch switches an activation state of at least one of the text exchange client and the voice client; and

a switching inference module configured to automatically determine conditions indicative of interaction difficulties with a current modality and to take at least one programmatic action responsive to determining the indicative conditions, wherein the programmatic action is directed towards having a user involved in the communication session utilize the modality switching capability of the modality switching engine.

16. The system of claim 15, wherein the text exchange interface is an off-the-shelf unmodified interface, wherein the voice client is at least one of an off-the-shelf software application and a telephony device, and wherein the automated speech response application is an unmodified VoiceXML based application that lacks an inherent text exchange capability.