Providing Expressive User Interaction With A Multimodal Application

• US 20080255850A1
• Filed: 04/12/2007
• Published: 10/16/2008
• Est. Priority Date: 04/12/2007
• Status: Active Grant

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Jablokov, Igor R., Eide, Ellen M., Cross, Charles W.
• Granted Patent: US 8,725,513 B2
• Time in Patent Office: Days
• US Class Current: 704/275
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

Methods, apparatus, and products are disclosed for providing expressive user interaction with a multimodal application, the multimodal application operating in a multimodal browser on a multimodal device supporting multiple modes of user interaction including a voice mode and one or more non-voice modes, the multimodal application operatively coupled to a speech engine through a VoiceXML interpreter, including: receiving, by the multimodal browser, user input from a user through a particular mode of user interaction; determining, by the multimodal browser, user output for the user in dependence upon the user input; determining, by the multimodal browser, a style for the user output in dependence upon the user input, the style specifying expressive output characteristics for at least one other mode of user interaction; and rendering, by the multimodal browser, the user output in dependence upon the style.

Claims

1. A computer-implemented method of providing expressive user interaction with a multimodal application, the multimodal application operating in a multimodal browser on a multimodal device supporting multiple modes of user interaction including a voice mode and one or more non-voice modes, the multimodal application operatively coupled to a speech engine through a VoiceXML interpreter, the method comprising:

receiving, by the multimodal browser, user input from a user through a particular mode of user interaction;

determining, by the multimodal browser, user output for the user in dependence upon the user input;

determining, by the multimodal browser, a style for the user output in dependence upon the user input, the style specifying expressive output characteristics for at least one other mode of user interaction; and

rendering, by the multimodal browser, the user output in dependence upon the style.

2. The method of claim 1 further comprising prompting, by the multimodal browser, the user for the user input.

3. The method of claim 1 wherein:

receiving, by the multimodal browser, user input from a user through a particular mode of user interaction further comprises receiving graphical user input from the user through a graphical user interface;

determining, by the multimodal browser, user output for the user in dependence upon the user input further comprises determining the user output for the user in dependence upon the graphical user input; and

determining, by the multimodal browser, a style for the user output further comprises determining the style for the user output in dependence upon the graphical user input.

4. The method of claim 1 wherein:

receiving, by the multimodal browser, user input from a user through a particular mode of user interaction further comprises;

receiving a voice utterance from the user, and determining recognition results using the speech engine and a grammar;

determining, by the multimodal browser, user output for the user in dependence upon the user input further comprises determining the user output for the user in dependence upon the recognition results; and

determining, by the multimodal browser, a style for the user output further comprises determining the style for the user output in dependence upon the recognition results.

5. The method of claim 1 wherein:

the style specifies prosody for the voice mode of user interaction; and

rendering, by the multimodal browser, the user output in dependence upon the style further comprises;

synthesizing, through the VoiceXML interpreter using the speech engine, the user output into synthesized speech in dependence upon the style, and

playing the synthesized speech for the user.

6. The method of claim 5 further comprising providing, by the speech engine to the multimodal browser through the VoiceXML interpreter, a prosody event in dependence upon the synthesization of the user output into synthesized speech.

7. The method of claim 1 wherein:

the style specifies visual characteristics for a visual mode of user interaction; and

rendering, by the multimodal browser, the user output in dependence upon the style further comprises displaying the user output to the user in dependence upon the style.

8. The method of claim 7 further comprising receiving, by the multimodal browser through the VoiceXML interpreter, a prosody event from the speech engine, and wherein rendering, by the multimodal browser, the user output in dependence upon the style further comprises rendering the user output in dependence upon the prosody event.

9. Apparatus for providing expressive user interaction with a multimodal application, the multimodal application operating in a multimodal browser on a multimodal device supporting multiple modes of user interaction including a voice mode and one or more non-voice modes, the multimodal application operatively coupled to a speech engine through a VoiceXML interpreter, the apparatus comprising a computer processor and a computer memory operatively coupled to the computer processor, the computer memory having disposed within it computer program instructions capable of:

receiving, by the multimodal browser, user input from a user through a particular mode of user interaction;

determining, by the multimodal browser, user output for the user in dependence upon the user input;

determining, by the multimodal browser, a style for the user output in dependence upon the user input, the style specifying expressive output characteristics for at least one other mode of user interaction; and

rendering, by the multimodal browser, the user output in dependence upon the style.

10. The apparatus of claim 9 further comprising computer program instructions capable of prompting, by the multimodal browser, the user for the user input.

11. The apparatus of claim 9 wherein:

receiving, by the multimodal browser, user input from a user through a particular mode of user interaction further comprises receiving graphical user input from the user through a graphical user interface;

determining, by the multimodal browser, user output for the user in dependence upon the user input further comprises determining the user output for the user in dependence upon the graphical user input; and

determining, by the multimodal browser, a style for the user output further comprises determining the style for the user output in dependence upon the graphical user input.

12. The apparatus of claim 9 wherein:

receiving, by the multimodal browser, user input from a user through a particular mode of user interaction further comprises;

receiving a voice utterance from the user, and determining recognition results using the speech engine and a grammar;

determining, by the multimodal browser, user output for the user in dependence upon the user input further comprises determining the user output for the user in dependence upon the recognition results; and

determining, by the multimodal browser, a style for the user output further comprises determining the style for the user output in dependence upon the recognition results.

13. The apparatus of claim 9 wherein:

the style specifies prosody for the voice mode of user interaction;

rendering, by the multimodal browser, the user output in dependence upon the style further comprises;

synthesizing, through the VoiceXML interpreter using the speech engine, the user output into synthesized speech in dependence upon the style, and

playing the synthesized speech for the user; and

the computer memory also has disposed within it computer program instructions capable of providing, by the speech engine to the multimodal browser through the VoiceXML interpreter, a prosody event in dependence upon the synthesization of the user output into synthesized speech.

14. The apparatus of claim 9 wherein:

the style specifies visual characteristics for a visual mode of user interaction;

the computer memory also has disposed within it computer program instructions capable of receiving, by the multimodal browser through the VoiceXML interpreter, a prosody event from the speech engine; and

rendering, by the multimodal browser, the user output in dependence upon the style further comprises displaying the user output to the user in dependence upon the style and rendering the user output in dependence upon the prosody event.

15. A computer program product for providing expressive user interaction with a multimodal application, the multimodal application operating in a multimodal browser on a multimodal device supporting multiple modes of user interaction including a voice mode and one or more non-voice modes, the multimodal application operatively coupled to a speech engine through a VoiceXML interpreter, the computer program product disposed upon a computer-readable medium, the computer program product comprising computer program instructions capable of:

receiving, by the multimodal browser, user input from a user through a particular mode of user interaction;

determining, by the multimodal browser, user output for the user in dependence upon the user input;

determining, by the multimodal browser, a style for the user output in dependence upon the user input, the style specifying expressive output characteristics for at least one other mode of user interaction; and

rendering, by the multimodal browser, the user output in dependence upon the style.

16. The computer program product of claim 15 further comprising computer program instructions capable of prompting, by the multimodal browser, the user for the user input.

17. The computer program product of claim 15 wherein:

receiving, by the multimodal browser, user input from a user through a particular mode of user interaction further comprises receiving graphical user input from the user through a graphical user interface;

determining, by the multimodal browser, user output for the user in dependence upon the user input further comprises determining the user output for the user in dependence upon the graphical user input; and

determining, by the multimodal browser, a style for the user output further comprises determining the style for the user output in dependence upon the graphical user input.

18. The computer program product of claim 15 wherein:

receiving, by the multimodal browser, user input from a user through a particular mode of user interaction further comprises;

receiving a voice utterance from the user, and

determining recognition results using the speech engine and a grammar;

determining, by the multimodal browser, user output for the user in dependence upon the user input further comprises determining the user output for the user in dependence upon the recognition results; and

determining, by the multimodal browser, a style for the user output further comprises determining the style for the user output in dependence upon the recognition results.

19. The computer program product of claim 15 wherein:

the style specifies prosody for the voice mode of user interaction;

rendering, by the multimodal browser, the user output in dependence upon the style further comprises;

synthesizing, through the VoiceXML interpreter using the speech engine, the user output into synthesized speech in dependence upon the style, and

playing the synthesized speech for the user; and

the computer program product further comprises computer program instructions capable of providing, by the speech engine to the multimodal browser through the VoiceXML interpreter, a prosody event in dependence upon the synthesization of the user output into synthesized speech.

20. The computer program product of claim 15 wherein:

the style specifies visual characteristics for a visual mode of user interaction;

the computer program product further comprises computer program instructions capable of receiving, by the multimodal browser through the VoiceXML interpreter, a prosody event from the speech engine; and

rendering, by the multimodal browser, the user output in dependence upon the style further comprises displaying the user output to the user in dependence upon the style and rendering the user output in dependence upon the prosody event.