SWITCHING BETWEEN MODALITIES IN A SPEECH APPLICATION ENVIRONMENT EXTENDED FOR INTERACTIVE TEXT EXCHANGES

• US 20080147406A1
• Filed: 12/19/2006
• Published: 06/19/2008
• Est. Priority Date: 12/19/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: MOORE, VICTOR S., NUSBICKEL, WENDI L., MANDALIA, BAIJU D., DA PALMA, WILLIAM V.
• Granted Patent: US 7,921,214 B2
• Time in Patent Office: Days
• US Class Current: 704/260
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: H04L 51/02 using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages

Abstract

The present solution includes a method for dynamically switching modalities in a dialogue session involving a voice server. In the method, a dialogue session can be established between a user and a speech application. During the dialogue session, the user can interact using an original modality, which is either a speech modality, a text exchange modality, or a multi mode modality that includes a text exchange modality. The speech application can interact using a speech modality. A modality switch trigger can be detected that changes the original modality to a different modality. The modality transition to the second modality can be transparent to the speech application. The speech application can be a standard VoiceXML based speech application that lacks an inherent text exchange capability.

Claims

1. A method for dynamically switching modalities in a dialogue session involving a voice server comprising:

establishing a dialogue session between a user and a speech application, wherein during the dialogue session, the user interacts using an original modality;

detecting a modality switch trigger; and

changing the original modality to a second modality that is different from the original modality, wherein a modality transition to the second modality is transparent to a voice server used by the speech application and is transparent to the speech application, wherein the original modality and the second modality are selected from a group of modalities consisting of a speech modality, a text exchange modality, and a multi mode modality that includes a speech modality and a text exchange modality, and wherein one of the original modality and the second modality is the text exchange modality.

2. The method of claim 1, wherein the text exchange modality includes at least one of an online-chat modality, an instant messaging modality, and a text messaging modality.

3. The method of claim 1, wherein the text exchange client is an off-the-shelf unmodified client, and wherein the speech application is a VoiceXML based application that lacks an inherent text exchange capability.

4. The method of claim 1, wherein communications during the dialogue session involve a Chatbot server, a text exchange client, and a voice client.

5. The method of claim 4, wherein during the dialogue session the text exchange client exchanges text exchange messages over a first communication channel with the Chatbot server, the voice server exchanges messages over a second communication channel with the Chatbot server, and the voice client exchanges voice-based messages over a third communication channel with the Chatbot server, wherein the changing step changes an activation state of at least one of the first communication channel and the third communication channel.

6. The method of claim 5, wherein during the dialogue session only one of the first communication channel and the third communication channel is concurrently active.

7. The method of claim 1, wherein the user inputs a switching code provided within a message written in the original modality, wherein the switching codes causes the modality switch trigger to fire.

8. A method for dynamically switching modalities in a dialogue session involving a voice server comprising:

identifying an automated dialogue session between a text exchange client and a speech application, wherein communications during the dialogue session involve a Chatbot server and a voice server, wherein during the dialogue session the text exchange client exchanges text exchange messages over a first communication channel with the Chatbot server, the voice server exchanges messages over a second communication channel with the Chatbot server, wherein the voice server uses a text interface that permits the voice server to directly consume text input, wherein the Chatbot server dynamically converts voice markup generated by the voice server to text that is provided to the text exchange client;

the Chatbot server detecting a modality switch trigger; and

automatically establishing a third communication channel between the Chatbot server and a voice device, wherein bidirectional voice messages for the automated dialogue session are thereafter conveyed between the voice device and the speech application.

9. The method of claim 8, wherein the speech application remains unaware of whether user entered messages originate from the first communication channel or the second communication channel and remains unaware of whether automatically generated output is conveyed to the text exchange client or to the voice device.

10. The method of claim 8, wherein the text exchange client is an off-the-shelf unmodified client, and wherein the speech application is an unmodified speech enabled application that lacks an inherent text exchange capability.

11. The method of claim 8, further comprising:

responsive to detecting the modality switch trigger, automatically closing the first communication channel.

12. The method of claim 8, wherein the modality switch trigger is associated with a phone number for the voice device, said method further comprising:

the Chatbot server automatically calling the phone number to establish the third communication channel.

13. The method of claim 12, wherein the phone number is conveyed to the Chatbot server within a text exchange message entered through the text exchange client.

14. The method of claim 12, wherein the phone number is automatically looked-up in a data store maintained by the Chatbot server, wherein within the data store, said phone number relates to at least one of a user of the text exchange client, a unique identifier associated with the text exchange client, a unique identifier associated with a computing device upon which the text exchange client resides, and a unique identifier associated with the dialogue session.

15. The method of claim 8, wherein a programmatic action of the speech application initiates the modality switch trigger.

16. The method of claim 8, further comprising:

simultaneously maintaining the first communication channel, the second communication channel, and the third communication channel, wherein text exchange input from the first communication channel and voice message input from the third communication channel are conveyed to the voice server over the second communication channel.

17. The method of claim 16, wherein the first communication channel and the third communication channel are the same channel over which multi modal information is conveyed to a multi mode device that includes a text modality and a speech modality.

18. The method of claim 16, wherein the first communication channel and the third communication channel are different channels, and wherein the text exchange client is associated with a different communication channel than the voice device.

19. A system for providing text exchange services comprising:

a text exchange client configured to send and receive real-time text exchanges over a text exchange channel;

a voice client configured to send and receive speech over a voice channel;

a speech application environment configured to execute an automated speech response application that permits users to interact in real-time over a voice channel;

a Chatbot server configured to establish a communication session involving the text exchange client and the automated speech response application, wherein the Chatbot server can dynamically convert messages between the text exchange client and the speech response application during the communication session in a manner transparent to the text exchange client and to the automated speech response application; and

a modality switching engine configured to dynamically switch a modality through which a user interacts with the automated speech response application, wherein a dynamic modality switch switches an activation state of at least one of the text exchange client and the voice client.

20. The system of claim 19, wherein the text exchange interface is an off-the-shelf unmodified interface, wherein the voice client is at least one of an off-the-shelf software application and a telephony device, and wherein the automated speech response application is a VoiceXML based application that lacks an inherent text exchange capability.