Speech Recognition Using Context-Aware Recognition Models

• US 20120323557A1
• Filed: 06/19/2012
• Published: 12/20/2012
• Est. Priority Date: 06/19/2011
• Status: Active Grant

• Current Assignee: Solventum Intellectual Properties Company (Solventum Corp.)
• Original Assignee: MModal IP LLC (3M Company)
• Inventors: Koll, Detlef, Finke, Michael
• Granted Patent: US 9,489,375 B2
• Time in Patent Office: Days
• US Class Current: 704/8
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 40/274 Converting codes to words; Guess-ahead of partial word inputs
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/454 Multi-language systems; Localisation; Internationalisation
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/0635 updating or merging of old and new templates; Mean values; Weighting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

Inputs provided into user interface elements of an application are observed. Records are made of the inputs and the state(s) the application was in while the inputs were provided. For each state, a corresponding language model is trained based on the input(s) provided to the application while the application was in that state. When the application is next observed to be in a previously-observed state, a language model associated with the application'"'"'s current state is applied to recognize speech input provided by a user and thereby to generate speech recognition output that is provided to the application. An application'"'"'s state at a particular time may include the user interface element(s) that are displayed and/or in focus at that time.

Claims

1. A computer-implemented method comprising:

receiving a first plurality of inputs into an application while the application is in a first state;

training a first language model based on the first plurality of inputs;

determining that the application is in the first state; and

applying the first language model to a first speech input in response to determining that the application is in the first state.

2. The method of claim 1, wherein receiving further comprises receiving the first plurality of inputs from a text-based input device.

3. The method of claim 1, wherein receiving further comprises receiving the first plurality of inputs from a pointing device.

4. The method of claim 1, wherein receiving further comprises receiving the first plurality of inputs from a speech input device.

5. The method of claim 1, wherein receiving further comprises receiving a speech-based input and a text-based input in the first plurality of inputs.

6. The method of claim 1, wherein receiving further comprises receiving a speech-based input and input from a pointing device in the first plurality of inputs.

7. The method of claim 1, further comprising:

receiving a second plurality of inputs into the application while the application is in a second state that differs from the first state;

training a second language model based on the second plurality of inputs;

determining that the application is in the second state; and

applying the second language model to second speech input in response to determining that the application is in the second state.

8. The method of claim 1, wherein training the first language model further comprises:

receiving a second plurality of inputs into a second copy of the application, while the second application is in the first state and executing on a different computing device than a computing device on which the first plurality of inputs is received; and

modifying the first language model based on the second plurality of inputs.

9. The method of claim 1, wherein training the first language model comprises:

identifying a pattern of use of a user interface associated with the first plurality of inputs; and

modifying the first language model based on the pattern of use.

10. The method of claim 1, further comprising modifying a resource accessed by the language model, based on the first plurality of inputs.

11. The method of claim 1, further comprising modifying an association between a word and a pronunciation of the word, the association accessible by the language model, based on the first plurality of inputs.

12. The method of claim 1, further comprising configuring a parameter governing whether to interpret an utterance as a grammar, based on the first plurality of inputs.

13. The method of claim 1, further comprising configuring a parameter governing whether to interpret an utterance as text, based on the first plurality of inputs.

14. The method of claim 1, further comprising:

identifying, in the plurality of inputs, a plurality of input values;

identifying a frequency with which each of the plurality of input values occurs in the plurality of inputs; and

training the first language model based on the identified frequency.

15. The method of claim 1, further comprising:

identifying, for one of the plurality of inputs, an input value;

determining that the input value is an instance of a concept;

identifying, in the plurality of inputs, a number of instances of the concept;

identifying a frequency with which the concept occurs in the plurality of inputs; and

training the first language model based on the identified frequency.

16. The method of claim 1, wherein training the first language model further comprises associating a probability with a word in the language model.

17. The method of claim 1, further comprising modifying the language model, based on a type of a user interface element provided by the application and receiving the first plurality of inputs.

18. The method of claim 1, further comprising:

receiving a second input into the application while the application is in the first state;

identifying the second input as an alternative spoken form of an input in the first plurality of inputs; and

modifying the first language model based on the second input.

19. The method of claim 1, further comprising:

identifying, by an operating system hooking component, a user interface element displayed in a foreground of a graphical user interface; and

associating the first language model with the identified user interface element.

20. The method of claim 1, further comprising:

identifying, by an operating system hooking component, a state of a user interface element displayed in a foreground of a graphical user interface; and

associating the first language model with the identified user interface element.

21. The method of claim 1, further comprising:

identifying, by an operating system hooking component, a user interface element having input focus; and

associating the first language model with the identified user interface element.

22. The method of claim 1, further comprising:

identifying, by an operating system hooking component, a user interface element into which one of the first plurality of inputs is provided; and

associating the first language model with the identified user interface element.

23. The method of claim 1, further comprising:

identifying, by an operating system hooking component, an application associated with a user interface element into which one of the first plurality of inputs is provided; and

associating the first language model with the identified application.

24. The method of claim 1, further comprising:

identifying, by an operating system hooking component, a plurality of user interface elements displayed by an application associated with a user interface element into which one of the first plurality of inputs is provided; and

associating the first language model with the identified plurality of user interface elements.

25. The method of claim 1, wherein determining further comprises:

analyzing application data to determine a state of the application;

comparing the determined state of the application to a state associated with the first language model; and

determining that the determined state and the state associated with the first language model are substantially the same state.

26. The method of claim 1, wherein determining further comprises:

comparing application data of the application to application data associated with the first language model; and

determining that the application data of the application and the application data associated with the first language model are substantially the same state.

27. The method of claim 1, wherein determining further comprises:

analyzing information provided by a context sharing application to determine a state of the application;

comparing the determined state of the application to a state associated with the first language model; and

determining that the determined state and the state associated with the first language model are substantially the same state.

28. The method of claim 1, further comprising providing, to the application, a result of applying the first language model to the first speech input.

29. The method of claim 28, wherein providing further comprises emulating a keyboard event to submit at least one of text and a control sequence to the application.

30. The method of claim 28, wherein providing further comprises:

posting the result to a clipboard buffer maintained by an operating system executing the application; and

receiving, by the application, the result from the clipboard buffer.

31. The method of claim 28, wherein providing further comprises:

generating an operating system message including the result; and

transmitting the operating system message to the application.

32. The method of claim 28, wherein providing further comprises emulating a pointing device event to submit at least one of text and a control sequence to the application.

33. The method of claim 28, wherein providing further comprises:

identifying, by an operating system hooking component, a user interface having input focus; and

providing the result to the user interface having input focus.

34. The method of claim 1, wherein applying further comprises applying the first language model to a first speech input after achieving a degree of confidence in a level of accuracy of the first language model.

35. An automated speech recognition system comprising:

means for receiving a first plurality of inputs into an application while the application is in a first state;

means for training a first language model based on the first plurality of inputs;

means for determining that the application is in the first state; and

means for applying the first language model to a first speech input in response to determining that the application is in the first state.

36. The automated speech recognition system of claim 35, further comprising an operating system hooking component retrieving the first plurality of inputs.

37. A non-transitory computer readable medium storing computer program instructions executable by at least one computer processor to perform a method, the method comprising:

receiving a first plurality of inputs into an application while the application is in a first state;

training a first language model based on the first plurality of inputs;

determining that the application is in the first state; and

applying the first language model to a first speech input in response to determining that the application is in the first state.

38. The computer readable medium of claim 37, wherein receiving further comprises receiving the first plurality of inputs from a text-based input device.

39. The computer readable medium of claim 37, wherein receiving further comprises receiving the first plurality of inputs from a pointing device.

40. The computer readable medium of claim 37, wherein receiving further comprises receiving the first plurality of inputs from a speech input device.

41. The computer readable medium of claim 37, wherein receiving further comprises receiving a speech-based input and a text-based input in the first plurality of inputs.

42. The computer readable medium of claim 37, wherein receiving further comprises receiving a speech-based input and input from a pointing device in the first plurality of inputs.

43. The computer readable medium of claim 37 wherein the method further comprises:

receiving a second plurality of inputs into the application while the application is in a second state that differs from the first state;

training a second language model based on the second plurality of inputs;

determining that the application is in the second state; and

applying the second language model to second speech input in response to determining that the application is in the second state.

44. The computer readable medium of claim 37, wherein training the first language model further comprises:

receiving a second plurality of inputs into a second copy of the application, while the second application is in the first state and executing on a different computing device than a computing device on which the first plurality of inputs is received; and

modifying the first language model based on the second plurality of inputs.

45. The computer readable medium of claim 37, wherein training the first language model comprises:

identifying a pattern of use of a user interface associated with the first plurality of inputs; and

modifying the first language model based on the pattern of use.

46. The computer readable medium of claim 37, wherein the method further comprises modifying a resource accessed by the language model, based on the first plurality of inputs.

47. The computer readable medium of claim 37, wherein the method further comprises modifying an association between a word and a pronunciation of the word, the association accessible by the language model, based on the first plurality of inputs.

48. The computer readable medium of claim 37, wherein the method further comprises configuring a parameter governing whether to interpret an utterance as a grammar, based on the first plurality of inputs.

49. The computer readable medium of claim 37, wherein the method further comprises configuring a parameter governing whether to interpret an utterance as text, based on the first plurality of inputs.

50. The computer readable medium of claim 37, wherein the method further comprises:

identifying, in the plurality of inputs, a plurality of input values;

identifying a frequency with which each of the plurality of input values occurs in the plurality of inputs; and

training the first language model based on the identified frequency.

51. The computer readable medium of claim 37, wherein the method further comprises:

identifying, for one of the plurality of inputs, an input value;

determining that the input value is an instance of a concept;

identifying, in the plurality of inputs, a number of instances of the concept;

identifying a frequency with which the concept occurs in the plurality of inputs; and

training the first language model based on the identified frequency.

52. The computer readable medium of claim 37, wherein training the first language model further comprises associating a probability with a word in the language model.

53. The computer readable medium of claim 37, wherein the method further comprises modifying the language model, based on a type of a user interface element provided by the application and receiving the first plurality of inputs.

54. The computer readable medium of claim 37, wherein the method further comprises:

receiving a second input into the application while the application is in the first state;

identifying the second input as an alternative spoken form of an input in the first plurality of inputs; and

modifying the first language model based on the second input.

55. The computer readable medium of claim 37, wherein the method further comprises:

identifying, by an operating system hooking component, a user interface element displayed in a foreground of a graphical user interface; and

associating the first language model with the identified user interface element.

56. The computer readable medium of claim 37, wherein the method further comprises:

identifying, by an operating system hooking component, a state of a user interface element displayed in a foreground of a graphical user interface; and

associating the first language model with the identified user interface element.

57. The computer readable medium of claim 37, wherein the method further comprises:

identifying, by an operating system hooking component, a user interface element having input focus; and

associating the first language model with the identified user interface element.

58. The computer readable medium of claim 37, wherein the method further comprises:

identifying, by an operating system hooking component, a user interface element into which one of the first plurality of inputs is provided; and

associating the first language model with the identified user interface element.

59. The computer readable medium of claim 37, wherein the method further comprises:

identifying, by an operating system hooking component, an application associated with a user interface element into which one of the first plurality of inputs is provided; and

associating the first language model with the identified application.

60. The computer readable medium of claim 37, wherein the method further comprises:

identifying, by an operating system hooking component, a plurality of user interface elements displayed by an application associated with a user interface element into which one of the first plurality of inputs is provided; and

associating the first language model with the identified plurality of user interface elements.

61. The computer readable medium of claim 37, wherein determining further comprises:

analyzing application data to determine a state of the application;

comparing the determined state of the application to a state associated with the first language model; and

determining that the determined state and the state associated with the first language model are substantially the same state.

62. The computer readable medium of claim 37, wherein determining further comprises:

comparing application data of the application to application data associated with the first language model; and

determining that the application data of the application and the application data associated with the first language model are substantially the same state.

63. The computer readable medium of claim 37, wherein determining further comprises:

analyzing information provided by a context sharing application to determine a state of the application;

comparing the determined state of the application to a state associated with the first language model; and

determining that the determined state and the state associated with the first language model are substantially the same state.

64. The computer readable medium of claim 37, wherein the method further comprises providing, to the application, a result of applying the first language model to the first speech input.

65. The computer readable medium of claim 64, wherein providing further comprises emulating a keyboard event to submit at least one of text and a control sequence to the application.

66. The computer readable medium of claim 64, wherein providing further comprises:

posting the result to a clipboard buffer maintained by an operating system executing the application; and

receiving, by the application, the result from the clipboard buffer.

67. The computer readable medium of claim 64, wherein providing further comprises:

generating an operating system message including the result; and

transmitting the operating system message to the application.

68. The computer readable medium of claim 64, wherein providing further comprises emulating a pointing device event to submit at least one of text and a control sequence to the application.

69. The computer readable medium of claim 64, wherein providing further comprises:

identifying, by an operating system hooking component, a user interface having input focus; and

providing the result to the user interface having input focus.

70. The computer readable medium of claim 37, wherein applying further comprises applying the first language model to a first speech input after achieving a degree of confidence in a level of accuracy of the first language model.