SYNCHRONIZING VISUAL AND SPEECH EVENTS IN A MULTIMODAL APPLICATION

• US 20120022875A1
• Filed: 09/30/2011
• Published: 01/26/2012
• Est. Priority Date: 06/16/2005
• Status: Active Grant

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Cross, Charles W. Jr., Hollinger, Michael C., Lewis, Benjamin D., Pike, Hilary A., Smith, Daniel M., Wintermute, David W., Zaitzeff, Michael A., Jablokov, Igor R.
• Granted Patent: US 8,571,872 B2
• Time in Patent Office: Days
• US Class Current: 704/275
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads

Abstract

Exemplary methods, systems, and products are disclosed for synchronizing visual and speech events in a multimodal application, including receiving from a user speech; determining a semantic interpretation of the speech; calling a global application update handler; identifying, by the global application update handler, an additional processing function in dependence upon the semantic interpretation; and executing the additional function. Typical embodiments may include updating a visual element after executing the additional function. Typical embodiments may include updating a voice form after executing the additional function. Typical embodiments also may include updating a state table after updating the voice form. Typical embodiments also may include restarting the voice form after executing the additional function.

Claims

1-20. -20. (canceled)

21. A method, comprising:

calling a voice form;

receiving speech from a user;

determining a semantic interpretation of at least a portion of the speech using the voice form;

calling a global application update handler;

identifying, by the global application update handler, an additional processing function based at least in part upon the semantic interpretation, wherein the additional processing function is independent of the voice form; and

executing the additional processing function.

22. The method of claim 21, wherein executing the additional processing function comprises providing content to the user.

23. The method of claim 22, wherein the content comprises an advertisement.

24. The method of claim 21, further comprising updating the voice form based at least in part upon the semantic interpretation after executing the additional processing function.

25. The method of claim 21, further comprising:

exiting the voice form; and

restarting the voice form after executing the additional processing function.

26. The method of claim 21, further comprising updating a visual element based at least in part upon the semantic interpretation after executing the additional processing function.

27. The method of claim 21, wherein determining a semantic interpretation of at least a portion of the speech comprises determining a plurality of semantic interpretations of the at least a portion of the speech.

28. A system, comprising:

at least one computer processor;

at least one computer memory operatively coupled to the computer processor; and

computer program instructions disposed within the computer memory that, when executed, cause the at least one computer processor to;

call a voice form;

receive speech from a user;

determine a semantic interpretation of at least a portion of the speech using the voice form;

call a global application update handler;

identify, by the global application update handler, an additional processing function based at least in part upon the semantic interpretation, wherein the additional processing function is independent of the voice form; and

execute the additional processing function.

29. The system of claim 28, further comprising computer program instructions disposed within the computer memory that, when executed, cause the at least one computer processor to provide content to the user based at least in part upon the additional processing function.

30. The system of claim 29, wherein the content comprises an advertisement.

31. The system of claim 28, further comprising computer program instructions disposed within the computer memory that, when executed, cause the at least one computer processor to update the voice form based at least in part upon the semantic interpretation after executing the additional processing function.

32. The system of claim 28, further comprising computer program instructions disposed within the computer memory that, when executed, cause the at least one computer processor to:

exit the voice form; and

restart the voice form after executing the additional processing function.

33. The system of claim 28, further comprising computer program instructions disposed within the computer memory that, when executed, cause the at least one computer processor to update a visual element based at least in part upon the semantic interpretation after executing the additional processing function.

34. The system of claim 28, further comprising computer program instructions disposed within the computer memory that, when executed, cause the at least one computer processor to determine a plurality of semantic interpretations of the at least a portion of the speech.

35. A computer-readable storage medium comprising instructions that, when executed on at least one computer processor, perform a method, comprising:

calling a voice form;

receiving speech from a user;

determining a semantic interpretation of at least a portion of the speech using the voice form;

calling a global application update handler;

identifying, by the global application update handler, an additional processing function based at least in part upon the semantic interpretation, wherein the additional processing function is independent of the voice form; and

executing the additional processing function.

36. The computer-readable storage medium of claim 35, further comprising instructions that, when executed, provide content to the user based at least in part upon the additional processing function.

37. The computer-readable storage medium of claim 36, wherein the content comprises an advertisement.

38. The computer-readable storage medium of claim 35, further comprising instructions that, when executed, update the voice form based at least in part upon the semantic interpretation after executing the additional processing function.

39. The computer-readable storage medium of claim 35, further comprising instructions that, when executed:

exit the voice form; and

restart the voice form after executing the additional processing function.

40. The computer-readable storage medium of claim 35, further comprising instructions that, when executed, update a visual element based at least in part upon the semantic interpretation after executing the additional processing function.

41. The computer-readable storage medium of claim 35, further comprising instructions that, when executed, determine a plurality of semantic interpretations of the at least a portion of the speech.

42. A computer-readable storage medium comprising instructions that, when executed on at least one computer processor, perform a method, comprising:

receiving speech from a user;

determining a semantic interpretation of at least a portion of the speech; and

providing an advertisement based at least in part upon the semantic interpretation.

43. The computer-readable storage medium of claim 42, wherein the advertisement is provided based at least in part upon location information.

44. The computer-readable storage medium of claim 42, further comprising instructions that, when executed, call a global application update handler.

45. The computer-readable storage medium of claim 44, wherein the speech is received using a voice form, and wherein the global application handler is independent of the voice form.