Providing expressive user interaction with a multimodal application

• US 8,725,513 B2
• Filed: 04/12/2007
• Issued: 05/13/2014
• Est. Priority Date: 04/12/2007
• Status: Expired due to Fees

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Cross, Charles W. Jr., Eide, Ellen M., Jablokov, Igor R.
• Primary Examiner(s): Shah, Paras D
• Application Number:
  US11/734,422
• Publication Number:
  US 20080255850A1
• Time in Patent Office: 2,588 Days
• Field of Search: 704/258, 704/261, 704/270.1, 704/275, 704/231, 704/235, 704/251, 704/252, 704/257, 704/270
• US Class Current: 704/270
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

Methods, apparatus, and products are disclosed for providing expressive user interaction with a multimodal application, the multimodal application operating in a multimodal browser on a multimodal device supporting multiple modes of user interaction including a voice mode and one or more non-voice modes, the multimodal application operatively coupled to a speech engine through a VoiceXML interpreter, including: receiving, by the multimodal browser, user input from a user through a particular mode of user interaction; determining, by the multimodal browser, user output for the user in dependence upon the user input; determining, by the multimodal browser, a style for the user output in dependence upon the user input, the style specifying expressive output characteristics for at least one other mode of user interaction; and rendering, by the multimodal browser, the user output in dependence upon the style.

Claims

1. A computer-implemented method of providing expressive user interaction with a multimodal application, the multimodal application operating in a multimodal browser on a multimodal device supporting multiple modes of user interaction including a voice mode and one or more non-voice modes, the multimodal application operatively coupled to a speech engine through a VoiceXML interpreter, the method comprising:

receiving, by the multimodal browser, user input from a user through a particular mode of user interaction;

determining, by the multimodal browser, user output for the user in dependence upon the user input;

determining, by the multimodal browser, a style for the user output in dependence upon the user input, the style specifying expressive output characteristics for at least one other mode of user interaction; and

rendering, by the multimodal browser, the user output in dependence upon the style,

wherein determining the style for the user output in dependence upon the user input comprises performing a determination distinct from determining the user output,

wherein determining, by the multimodal browser, a style for the user output in dependence upon the user input comprises determining, by the multimodal browser, a style for the user output in dependence upon meaning of the user input.

2. The method of claim 1 further comprising prompting, by the multimodal browser, the user for the user input.

3. The method of claim 1 wherein:

receiving, by the multimodal browser, user input from the user through the particular mode of user interaction further comprises receiving graphical user input from the user through a graphical user interface;

determining, by the multimodal browser, user output for the user in dependence upon the user input further comprises determining the user output for the user in dependence upon the graphical user input; and

determining, by the multimodal browser, the style for the user output further comprises determining the style for the user output in dependence upon the graphical user input.

4. The method of claim 1 wherein:

receiving, by the multimodal browser, user input from the user through the particular mode of user interaction further comprises;

receiving a voice utterance from the user, and

determining recognition results using the speech engine and a grammar;

determining, by the multimodal browser, user output for the user in dependence upon the user input further comprises determining the user output for the user in dependence upon the recognition results; and

determining, by the multimodal browser, the style for the user output further comprises determining the style for the user output in dependence upon the recognition results.

5. The method of claim 1 wherein:

the style specifies prosody for the voice mode of user interaction; and

rendering, by the multimodal browser, the user output in dependence upon the style further comprises;

synthesizing, through the VoiceXML interpreter using the speech engine, the user output into synthesized speech in dependence upon the style, and

playing the synthesized speech for the user.

6. The method of claim 5 further comprising providing, by the speech engine to the multimodal browser through the VoiceXML interpreter, a prosody event in dependence upon the synthesizing of the user output into synthesized speech.

7. The method of claim 1 wherein:

the style specifies visual characteristics for a visual mode of user interaction; and

rendering, by the multimodal browser, the user output in dependence upon the style further comprises displaying the user output to the user in dependence upon the style.

8. The method of claim 7 further comprising receiving, by the multimodal browser through the VoiceXML interpreter, a prosody event from the speech engine, and wherein rendering, by the multimodal browser, the user output in dependence upon the style further comprises rendering the user output in dependence upon the prosody event.

9. Apparatus for providing expressive user interaction with a multimodal application, the multimodal application operating in a multimodal browser on a multimodal device supporting multiple modes of user interaction including a voice mode and one or more non-voice modes, the multimodal application operatively coupled to a speech engine through a VoiceXML interpreter, the apparatus comprising a computer processor and a computer memory operatively coupled to the computer processor, the computer memory having stored within it computer program instructions which, when executed by the computer processor, cause performance of a method comprising:

receiving, by the multimodal browser, user input from a user through a particular mode of user interaction;

determining, by the multimodal browser, user output for the user in dependence upon the user input;

determining, by the multimodal browser, a style for the user output in dependence upon the user input, the style specifying expressive output characteristics for at least one other mode of user interaction; and

rendering, by the multimodal browser, the user output in dependence upon the style,

wherein determining the style for the user output in dependence upon the user input comprises performing a determination distinct from determining the user output,

wherein determining, by the multimodal browser, a style for the user output in dependence upon the user input comprises determining, by the multimodal browser, a style for the user output in dependence upon meaning of the user input.

10. The apparatus of claim 9, wherein the computer memory also has stored within it computer program instructions which, when executed by the computer processor, cause prompting, by the multimodal browser, the user for the user input.

11. The apparatus of claim 9 wherein:

receiving, by the multimodal browser, user input from the user through the particular mode of user interaction further comprises receiving graphical user input from the user through a graphical user interface;

determining, by the multimodal browser, user output for the user in dependence upon the user input further comprises determining the user output for the user in dependence upon the graphical user input; and

determining, by the multimodal browser, the style for the user output further comprises determining the style for the user output in dependence upon the graphical user input.

12. The apparatus of claim 9 wherein:

receiving, by the multimodal browser, user input from the user through the particular mode of user interaction further comprises;

receiving a voice utterance from the user, and

determining recognition results using the speech engine and a grammar;

determining, by the multimodal browser, user output for the user in dependence upon the user input further comprises determining the user output for the user in dependence upon the recognition results; and

determining, by the multimodal browser, the style for the user output further comprises determining the style for the user output in dependence upon the recognition results.

13. The apparatus of claim 9 wherein:

the style specifies prosody for the voice mode of user interaction;

rendering, by the multimodal browser, the user output in dependence upon the style further comprises;

synthesizing, through the VoiceXML interpreter using the speech engine, the user output into synthesized speech in dependence upon the style, and

playing the synthesized speech for the user; and

the computer memory also has stored within it computer program instructions which, when executed by the computer processor, cause providing, by the speech engine to the multimodal browser through the VoiceXML interpreter, a prosody event in dependence upon the synthesizing of the user output into synthesized speech.

14. The apparatus of claim 9 wherein:

the style specifies visual characteristics for a visual mode of user interaction;

the computer memory also has stored within it computer program instructions which, when executed by the computer processor, cause receiving, by the multimodal browser through the VoiceXML interpreter, a prosody event from the speech engine; and

rendering, by the multimodal browser, the user output in dependence upon the style further comprises displaying the user output to the user in dependence upon the style and rendering the user output in dependence upon the prosody event.

15. A computer-recordable device storing computer program instructions for providing expressive user interaction with a multimodal application, the multimodal application operating in a multimodal browser on a multimodal device supporting multiple modes of user interaction including a voice mode and one or more non-voice modes, the multimodal application operatively coupled to a speech engine through a VoiceXML interpreter, wherein the computer program instructions, when executed, cause performance of a method comprising:

receiving, by the multimodal browser, user input from a user through a particular mode of user interaction;

determining, by the multimodal browser, user output for the user in dependence upon the user input;

determining, by the multimodal browser, a style for the user output in dependence upon the user input, the style specifying expressive output characteristics for at least one other mode of user interaction; and

rendering, by the multimodal browser, the user output in dependence upon the style,

wherein determining the style for the user output in dependence upon the user input comprises performing a determination distinct from determining the user output,

wherein determining, by the multimodal browser, a style for the user output in dependence upon the user input comprises determining, by the multimodal browser, a style for the user output in dependence upon meaning of the user input.

16. The computer-recordable device of claim 15, wherein the method further comprises prompting, by the multimodal browser, the user for the user input.

17. The computer-recordable device of claim 15 wherein:

receiving, by the multimodal browser, user input from the user through the particular mode of user interaction further comprises receiving graphical user input from the user through a graphical user interface;

determining, by the multimodal browser, user output for the user in dependence upon the user input further comprises determining the user output for the user in dependence upon the graphical user input; and

determining, by the multimodal browser, the style for the user output further comprises determining the style for the user output in dependence upon the graphical user input.

18. The computer-recordable device of claim 15 wherein:

receiving, by the multimodal browser, user input from the user through the particular mode of user interaction further comprises;

receiving a voice utterance from the user, and

determining recognition results using the speech engine and a grammar;

determining, by the multimodal browser, user output for the user in dependence upon the user input further comprises determining the user output for the user in dependence upon the recognition results; and

determining, by the multimodal browser, the style for the user output further comprises determining the style for the user output in dependence upon the recognition results.

19. The computer-recordable device of claim 15 wherein:

the style specifies prosody for the voice mode of user interaction;

rendering, by the multimodal browser, the user output in dependence upon the style further comprises;

synthesizing, through the VoiceXML interpreter using the speech engine, the user output into synthesized speech in dependence upon the style, and

playing the synthesized speech for the user; and

the method further comprises providing, by the speech engine to the multimodal browser through the VoiceXML interpreter, a prosody event in dependence upon the synthesizing of the user output into synthesized speech.

20. The computer-recordable device of claim 15 wherein:

the style specifies visual characteristics for a visual mode of user interaction;

the method further comprising receiving, by the multimodal browser through the VoiceXML interpreter, a prosody event from the speech engine; and

rendering, by the multimodal browser, the user output in dependence upon the style further comprises displaying the user output to the user in dependence upon the style and rendering the user output in dependence upon the prosody event.