Semantic object synchronous understanding for highly interactive interface

• US 8,301,436 B2
• Filed: 05/29/2003
• Issued: 10/30/2012
• Est. Priority Date: 05/29/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wang, Kuansan
• Primary Examiner(s): Armstrong, Angela A
• Application Number:
  US10/447,399
• Publication Number:
  US 20040243419A1
• Time in Patent Office: 3,442 Days
• Field of Search: 704/9, 704/257, 704/270, 704/275, 704/231, 704/251
• US Class Current: 704/9
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/193 Formal grammars, e.g. finite state automata, context free grammars or word networks
• CPC Class Codes: G10L 15/197 Probabilistic grammars, e.g. word n-grams
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

A method and system provide a speech input mode which dynamically reports partial semantic parses, while audio captioning is still in progress. The semantic parses can be evaluated with an outcome immediately reported back to the user. The net effect is that task conventionally performed in the system turn are now carried out in the midst of the user turn thereby presenting a significant departure from the turn-taking nature of a spoken dialogue.

Claims

1. A computer-implemented method for interacting with a computer system, the method comprising:

receiving input comprising at least one command executable by an application from a user and capturing the input for processing;

performing recognition on the input to ascertain semantic information pertaining to a first portion of the input and outputting a semantic object comprising data including data for executing the at least one command in a format to be processed by a computer application and being in accordance with the input that has been recognized and semantic information for the first portion, wherein performing recognition and outputting the semantic object are performed using a language model comprising a combination of an N-gram language model and a context free grammar and while capturing continues for subsequent portions of the input, the language model storing information related to words and semantic information to be recognized; and

rendering information to the user while the user is providing the input, at least some of the information being responsive to the data regarding the at least one command within said semantic object.

2. The computer-implemented method of claim 1 wherein outputting the semantic object comprises outputting the data in a textual form.

3. The computer-implemented method of claim 2 wherein receiving and capturing input comprises receiving and capturing audible input from the user.

4. The computer-implemented method of claim 3 wherein receiving and capturing input comprises receiving and capturing speech input from the user.

5. The computer-implemented method of claim 1 wherein receiving and capturing input comprises receiving and capturing handwriting input from the user.

6. The computer-implemented method of claim 1 wherein receiving and capturing input comprises receiving and capturing visual input from the user.

7. The computer-implemented method of claim 1 and further comprising rendering information to the user while the user is providing the input, the information being a function of the semantic object received.

8. The computer-implemented method of claim 7 wherein rendering information comprises acknowledging what has been received by rendering information as a function of the semantic information for the semantic object.

9. The computer-implemented method of claim 8 wherein rendering information comprises rendering information as a function of the data in accordance with the input that has been recognized.

10. The computer-implemented method of claim 9 wherein rendering information comprising rendering an option to the user as a function of the data in accordance with the input that has been received.

11. The computer-implemented method of claim 10 wherein rendering information comprises rendering a plurality of options to the user as a function of the data in accordance with the input that has been received.

12. The computer-implemented method of claim 10 wherein rendering information comprises rendering an audible prompt to the user.

13. The computer-implemented method of claim 11 wherein rendering information comprises rendering visual indications to the user.

14. The computer-implemented method of claim 13 wherein rendering information comprises rendering synchronized audible and visual indications to the user.

15. A hardware computer readable storage media including instructions readable by a computing device which, when implemented, cause the computing device to handle information by performing steps comprising:

establishing a language model to perform recognition and understanding, the language model comprising a combination of an N-gram language model and a context free grammar and storing information related to words and semantic information to be recognized, the language model providing data in a format to be processed by a computer application and being in accordance with an input received and to provide semantic information for the input received;

receiving input comprising at least one command executable by an application from a user and capturing the input for processing;

processing the input with the language model to perform recognition on the input to ascertain semantic information pertaining to a first portion of the input and outputting a semantic object comprising data including data for executing the at least one command in a format to be processed by a computer application and being in accordance with the input that has been recognized and semantic information for the first portion, wherein performing recognition and outputting the semantic object are performed while capturing continues for subsequent portions of the input; and

rendering information to the user while the user is providing the input, the information being a function of said semantic object wherein at least some of the information includes information regarding the at least one command.

16. The hardware computer readable storage media of claim 15 wherein establishing a language model comprises defining multiple grammars to be used for recognition.

17. The hardware computer readable storage media of claim 16 wherein establishing a language model comprises using application program interfaces to define the multiple grammars to be used for recognition.

18. The hardware computer readable storage media of claim 17 wherein establishing a language model comprises using SALT.

19. The hardware computer readable storage media of claim 18 wherein processing the input comprise implementing a listen object of SALT in multiple mode.

20. The hardware computer readable storage media of claim 15 wherein outputting the semantic object comprises outputting the data in a textual form.

21. The hardware computer readable storage media of claim 20 wherein receiving and capturing input comprises receiving and capturing audible input from the user.

22. The hardware computer readable storage media of claim 21 wherein receiving and capturing input comprises receiving and capturing speech input from the user.

23. The hardware computer readable storage media of claim 15 wherein receiving and capturing input comprises receiving and capturing handwriting input from the user.

24. The hardware computer readable storage media of claim 15 wherein receiving and capturing input comprises receiving and capturing visual input from the user.

25. The hardware computer readable storage media of claim 15 and further comprising rendering information to the user while the user is providing the input, the information being a function of the semantic object received.

26. The hardware computer readable storage media of claim 25 wherein rendering information comprises acknowledging what has been received by rendering information as a function of the semantic information for the semantic object.

27. The hardware computer readable storage media of claim 26 wherein rendering information comprises rendering information as a function of the data in accordance with the input that has been recognized.

28. The hardware computer readable storage media of claim 27 wherein rendering information comprising rendering an option to the user as a function of the data in accordance with the input that has been received.

29. The hardware computer readable storage media of claim 28 wherein rendering information comprises rendering a plurality of options to the user as a function of the data in accordance with the input that has been received.

30. The hardware computer readable storage media of claim 25 wherein rendering information comprises rendering an audible prompt to the user.

31. The hardware computer readable storage media of claim 30 wherein rendering information comprises rendering visual indications to the user.

32. The hardware computer readable storage media of claim 31 wherein rendering information comprises rendering synchronized audible and visual indications to the user.

33. A hardware computer readable storage media including instructions readable by a computing device comprising:

a language model comprising a combination of an N-gram language model and a context-free grammar language model and storing information related to words and semantic information to be recognized;

a module receiving input including commands executable by an application from a user and capturing the input for processing, the module performing recognition on the input by accessing the language model and ascertaining semantic information pertaining to a first portion of the input and outputting a semantic object comprising data including data for executing commands in a format to be processed by a computer application and being in accordance with the input that has been recognized and semantic information for the first portion, wherein performing recognition and outputting the semantic object are performed while capturing continues for subsequent portions of the input; and

a second module performing a selected task, the second module receiving semantic objects synchronously in accordance with the user'"'"'s input, each semantic object comprising data including commands in a format to be processed by a computer application and in accordance with a portion of input from the user and semantic information pertaining to the portion of input from the user, the module taking action as a function of processing the semantic objects synchronously including providing information or utilizing other applications based on at least the data for executing the commands.

34. The hardware computer readable storage media of claim 33 wherein the module provides a semantic object that includes the data in a textual form.

35. The hardware computer readable storage media of claim 34 wherein the module captures audible input from the user.

36. The hardware computer readable storage media of claim 35 wherein the module captures speech input from the user.

37. The hardware computer readable storage media of claim 33 wherein the module captures handwriting input from the user.

38. The hardware computer readable storage media of claim 33 wherein the module captures visual input from the user.

39. The hardware computer readable storage media of claim 33 wherein the second module processes the data in a textual form.

40. The hardware computer readable storage media of claim 39 wherein the information rendered is a function of the data in accordance with the input.

41. The hardware computer readable storage media of claim claim 33 wherein the information rendered is a function of the semantic information for the semantic object.

42. The hardware computer readable storage media of claim 33 wherein the second module takes action comprising rendering information to the user while semantic objects related to subsequent portions of the user'"'"'s input are further received.

43. The hardware computer readable storage media of claim 42 wherein the information rendered comprises an option to the user as a function of the semantic objects received.

44. The hardware computer readable storage media of claim 43 wherein rendering information comprises rendering a plurality of options to the user as a function of the semantic objects received.

45. The hardware computer readable storage media of claim 42 wherein information rendered comprises an audible prompt to the user.

46. The hardware computer readable storage media of claim 45 wherein the information comprises rendering visual indications to the user.

47. The hardware computer readable storage media of claim 46 wherein rendering information comprises rendering synchronized audible and visual indications to the user.

48. The hardware computer readable storage media of claim 33 wherein utilizing other applications based upon at least the data regarding commands comprises accessing an e-mail server and scheduling a meeting with a third party.