Synchronizing visual and speech events in a multimodal application

• US 20060287845A1
• Filed: 06/16/2005
• Published: 12/21/2006
• Est. Priority Date: 06/16/2005
• Status: Active Grant

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Pike, Hilary A., Wintermute, David W., Lewis, Benjamin D., Jablokov, Igor R., Smith, Daniel M., Hollinger, Michael C., Zaitzeff, Michael A., Cross, Charles W. Jr.
• Granted Patent: US 7,917,365 B2
• Time in Patent Office: Days
• US Class Current: 704/4
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads

Abstract

Exemplary methods, systems, and products are disclosed for synchronizing visual and speech events in a multimodal application, including receiving from a user speech; determining a semantic interpretation of the speech; calling a global application update handler; identifying, by the global application update handler, an additional processing function in dependence upon the semantic interpretation; and executing the additional function. Typical embodiments may include updating a visual element after executing the additional function. Typical embodiments may include updating a voice form after executing the additional function. Typical embodiments also may include updating a state table after updating the voice form. Typical embodiments also may include restarting the voice form after executing the additional function.

Claims

1. A method for synchronizing visual and speech events in a multimodal application, the method comprising:

receiving from a user speech;

determining a semantic interpretation of the speech;

calling a global application update handler;

identifying, by the global application update handler, an additional processing function in dependence upon the semantic interpretation; and

executing the additional processing function.

2. The method of claim 1 further comprising updating a visual element after executing the additional processing function.

3. The method of claim 1 further comprising updating a voice form after executing the additional processing function.

4. The method of claim 3 further comprising updating a state table after updating the voice form.

5. The method of claim 1 wherein calling a global application update handler further comprises exiting a voice form.

6. The method of claim 5 further comprising restarting the voice form after executing the additional processing function.

7. The method of claim 1 wherein calling a global application update handler further comprises exiting a voice menu.

8. A system for synchronizing visual and speech events in a multimodal application, the system comprising at least one computer processor, at least one computer memory operatively coupled to the computer processor, and computer program instructions disposed within the computer memory capable of:

receiving from a user speech;

determining a semantic interpretation of the speech;

calling a global application update handler;

identifying, by the global application update handler, an additional processing function in dependence upon the semantic interpretation; and

executing the additional processing function.

9. The system of claim 8 further comprising computer program instructions disposed within the computer memory capable of updating a visual element after executing the additional processing function.

10. The system of claim 8 further comprising computer program instructions disposed within the computer memory capable of updating a voice form after executing the additional processing function.

11. The system of claim 10 further comprising computer program instructions disposed within the computer memory capable of updating a state table after updating the voice form.

12. The system of claim 8 wherein the computer program instructions disposed within the computer memory capable of exiting a voice form.

13. The system of claim 12 further comprising computer program instructions disposed within the computer memory capable of restarting the voice form after executing the additional processing function.

14. The system of claim 8 wherein the computer program instructions disposed within the computer memory capable of exiting a voice menu.

15. A computer program product for synchronizing visual and speech events in a multimodal application, the computer program product embodied on a computer-readable recording medium, the computer program product comprising:

computer program instructions that receive from a user speech;

computer program instructions that determine a semantic interpretation of the speech;

computer program instructions that call a global application update handler;

computer program instructions that identify, by the global application update handler, an additional processing function in dependence upon the semantic interpretation; and

computer program instructions that execute the additional processing function.

16. The computer program product of claim 15 further comprising computer program instructions that update a visual element after executing the additional processing function.

17. The computer program product of claim 15 further comprising computer program instructions that update a voice form after executing the additional processing function.

18. The computer program product of claim 15 wherein computer program instructions that call a global application update handler further comprises exiting a voice form.

19. The computer program product of claim 18 further comprising computer program instructions that restart the voice form after executing the additional processing function.

20. The computer program product of claim 15 wherein computer program instructions that call a global application update handler further comprises exiting a voice menu.