Common word graph based multimodal input

• US 20070239432A1
• Filed: 03/30/2006
• Published: 10/11/2007
• Est. Priority Date: 03/30/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhou, Jian-Lai, Liu, Peng, Soong, Frank
• Granted Patent: US 7,848,917 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 40/237 Lexical tools

Abstract

Multiple input modalities are selectively used by a user or process to prune a word graph. Pruning initiates rescoring in order to generate a new word graph with a revised best path.

Claims

1. A method for processing input received by a computing device comprising:

initializing a common word graph based on a language model;

receiving input and complementary information using a plurality of modalities, the complementary information corresponding to at least a portion of the input; and

modifying the word graph based on the input and the complementary information.

2. The method of claim 1 and further comprising:

outputting a hypothesis for the input based on the word graph.

3. The method of claim 2 wherein outputting the hypothesis comprises rendering the hypothesis to a user.

4. The method of claim 2 wherein modifying the word graph includes rescoring and pruning the word graph.

5. The method of claim 4 wherein the complementary information comprises corrective information received after receiving the input.

6. The method of claim 4 wherein the complementary information is received simultaneously with the input.

7. The method of claim 4 wherein rescoring the word graph includes averaging posterior probabilities from all modalities.

8. The method of claim 1 wherein modifying the word graph includes recording relevant nodes of each modality with respect to the word graph.

9. The method of claim 8 wherein modifying the word graph includes rescoring and pruning the word graph.

10. The method of claim 9 wherein modifying the word graph includes rescoring the word graph from a node corresponding to the complementary information to a furthest node available in the word graph.

11. A computer-readable medium having computer-executable instructions that when executed by a computer perform steps to process input received by the computer comprising the steps of:

receiving input using a first modality;

modifying a word graph based on the input; and

rendering a hypothesis for the input based on the word graph, and repeating the following steps until a desired hypothesis is obtained;

modifying the word graph based on complementary information received using a second modality, the complementary information corresponding to at least a portion of the input and the second modality being different than the first modality; and

rendering a new hypothesis for the input based on the word graph.

12. The computer-readable medium of claim 11 wherein the step of modifying the word graph includes pruning the word graph.

13. The computer-readable medium of claim 12 wherein the step of wherein modifying the word graph includes rescoring the word graph.

14. The computer-readable medium of claim 13 wherein the step of modifying the word graph includes recording relevant nodes of each modality with respect to the word graph and rescoring the word graph includes rescoring the word graph from a node corresponding to the complementary information to a furthest node available in the word graph.

15. A computing device:

a first component configured to provide input into the computing device using a first modality;

a second component configured to provide input into the computing device using a second modality; and

a recognizer configured to receive input from the first and the second component and configured to modify a common word graph based on input from the first component and input from the second component.

16. The computing device of claim 15 wherein the recognizer is configured to render a hypothesis based on the word graph.

17. The computing device of claim 16 wherein the recognizer is configured to modify the word graph by pruning and rescoring the word graph.

18. The computing device of claim 17 wherein the recognizer is configured to repetitively receive corrective information using the second modality, modify the common word graph, and render a new hypothesis based on the modified word graph until the new hypothesis is a desired hypothesis.