Semantic object synchronous understanding for highly interactive interface

• US 20040243419A1
• Filed: 05/29/2003
• Published: 12/02/2004
• Est. Priority Date: 05/29/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wang, Kuansan
• Granted Patent: US 8,301,436 B2
• Time in Patent Office: Days
• US Class Current: 704/277
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/193 Formal grammars, e.g. finite state automata, context free grammars or word networks
• CPC Class Codes: G10L 15/197 Probabilistic grammars, e.g. word n-grams
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

A method and system provide a speech input mode which dynamically reports partial semantic parses, while audio captioning is still in progress. The semantic parses can be evaluated with an outcome immediately reported back to the user. The net effect is that task conventionally performed in the system turn are now carried out in the midst of the user turn thereby presenting a significant departure from the turn-taking nature of a spoken dialogue.

Claims

1. A computer-implemented method for interacting with a computer system, the method comprising:

receiving input from a user and capturing the input for processing; and

performing recognition on the input to ascertain semantic information pertaining to a first portion of the input and outputting a semantic object comprising data in a format to be processed by a computer application and being in accordance with the input that has been recognized and semantic information for the first portion, wherein performing recognition and outputting the semantic object are performed while capturing continues for subsequent portions of the input.

2. The computer-implemented method of claim 1 wherein outputting the semantic object comprises outputting the data in a textual form.

3. The computer-implemented method of claim 2 wherein receiving and capturing input comprises receiving and capturing audible input from the user.

4. The computer-implemented method of claim 3 wherein receiving and capturing input comprises receiving and capturing speech input from the user.

5. The computer-implemented method of claim 1 wherein receiving and capturing input comprises receiving and capturing handwriting input from the user.

6. The computer-implemented method of claim 1 wherein receiving and capturing input comprises receiving and capturing visual input from the user.

7. The computer-implemented method of claim 1 and further comprising rendering information to the user while the user is providing the input, the information being a function of the semantic object received.

8. The computer-implemented method of claim 7 wherein rendering information comprises acknowledging what has been received by rendering information as a function of the semantic information for the semantic object.

9. The computer-implemented method of claim 8 wherein rendering information comprises rendering information as a function of the data in accordance with the input that has been recognized.

10. The computer-implemented method of claim 9 wherein rendering information comprising rendering an option to the user as a function of the data in accordance with the input that has been received.

11. The computer-implemented method of claim 10 wherein rendering information comprises rendering a plurality of options to the user as a function of the data in accordance with the input that has been received.

12. The computer-implemented method of claim 10 wherein rendering information comprises rendering an audible prompt to the user.

13. The computer-implemented method of claim 11 wherein rendering information comprises rendering visual indications to the user.

14. The computer-implemented method of claim 13 wherein rendering information comprises rendering synchronized audible and visual indications to the user.

15. A computer readable media including instructions readable by a computing device which, when implemented, cause the computing device to handle information by performing steps comprising:

establishing a language model to perform recognition and understanding, the language model adapted to provide data in a format to be processed by a computer application and being in accordance with an input received and to provide semantic information for the input received;

receiving input from a user and capturing the input for processing;

processing the input with the language model to perform recognition on the input to ascertain semantic information pertaining to a first portion of the input and outputting a semantic object comprising data in a format to be processed by a computer application and being in accordance with the input that has been recognized and semantic information for the first portion, wherein performing recognition and outputting the semantic object are performed while capturing continues for subsequent portions of the input.

16. The computer readable media of claim 15 wherein the language model comprises a combination of an N-gram language model and a context-free grammar language model.

17. The computer readable media of claim 16 wherein establishing a language model comprises defining multiple grammars to be used for recognition.

18. The computer readable media of claim 17 wherein establishing a language model comprises using application program interfaces to define the multiple grammars to be used for recognition.

19. The computer readable media of claim 18 wherein establishing a language model comprises using SALT.

20. The computer readable media of claim 19 wherein processing the input comprise implementing a listen object of SALT in multiple mode.

21. The computer readable media of claim 15 wherein outputting the semantic object comprises outputting the data in a textual form.

22. The computer readable media of claim 21 wherein receiving and capturing input comprises receiving and capturing audible input from the user.

23. The computer readable media of claim 22 wherein receiving and capturing input comprises receiving and capturing speech input from the user.

24. The computer readable media of claim 15 wherein receiving and capturing input comprises receiving and capturing handwriting input from the user.

25. The computer readable media of claim 15 wherein receiving and capturing input comprises receiving and capturing visual input from the user.

26. The computer readable media of claim 15 and further comprising rendering information to the user while the user is providing the input, the information being a function of the semantic object received.

27. The computer readable media of claim 26 wherein rendering information comprises acknowledging what has been received by rendering information as a function of the semantic information for the semantic object.

28. The computer readable media of claim 27 wherein rendering information comprises rendering information as a function of the data in accordance with the input that has been recognized.

29. The computer readable media of claim 28 wherein rendering information comprising rendering an option to the user as a function of the data in accordance with the input that has been received.

30. The computer readable media of claim 29 wherein rendering information comprises rendering a plurality of options to the user as a function of the data in accordance with the input that has been received.

31. The computer readable media of claim 26 wherein rendering information comprises rendering an audible prompt to the user.

32. The computer readable media of claim 31 wherein rendering information comprises rendering visual indications to the user.

33. The computer readable media of claim 32 wherein rendering information comprises rendering synchronized audible and visual indications to the user.

34. A computer readable media including instructions readable by a computing device comprising:

a module adapted to receive input from a user and capture the input for processing, the module adapted to perform recognition on the input to ascertain semantic information pertaining to a first portion of the input and adapted to output a semantic object comprising data in a format to be processed by a computer application and being in accordance with the input that has been recognized and semantic information for the first portion, wherein performing recognition and outputting the semantic object are performed while capturing continues for subsequent portions of the input.

35. The computer readable media of claim 34 wherein the module is adapted to provide a semantic object that includes the data in a textual form.

36. The computer readable media of claim 35 wherein the module is adapted to capture audible input from the user.

37. The computer readable media of claim 36 wherein the module is adapted to capture speech input from the user.

38. The computer readable media of claim 34 wherein the module is adapted to capture handwriting input from the user.

39. The computer readable media of claim 34 wherein the module is adapted to capture visual input from the user.

40. A computer readable media including instructions readable by a computing device comprising:

a module configured to perform a selected task, the module adapted to receive semantic objects synchronously in accordance with a user'"'"'s input, each semantic object comprising data in a format to be processed by a computer application and in accordance with a portion of input from the user and semantic information pertaining to the portion of input from the user, the module adapted to take action as a function of processing the semantic objects synchronously.

41. The computer readable media of claim 40 wherein the module is adapted to process the data in a textual form.

42. The computer readable media of claim 40 wherein the module is adapted to take action comprising rendering information to the user while semantic objects related to subsequent portions of the user'"'"'s input are further received.

43. The computer readable media of claim 40 wherein the information rendered is a function of the semantic information for the semantic object.

44. The computer readable media of claim 41 wherein the information rendered is a function of the data in accordance with the input.

45. The computer readable media of claim 42 wherein the information rendered comprises an option to the user as a function of the semantic objects received.

46. The computer readable media of claim 45 wherein rendering information comprises rendering a plurality of options to the user as a function of the semantic objects received.

47. The computer readable media of claim 42 wherein information rendered comprises an audible prompt to the user.

48. The computer readable media of claim 47 wherein the information comprises rendering visual indications to the user.

49. The computer readable media of claim 48 wherein rendering information comprises rendering synchronized audible and visual indications to the user.