Combined speech and alternate input modality to a mobile device

• US 7,941,316 B2
• Filed: 10/28/2005
• Issued: 05/10/2011
• Est. Priority Date: 10/28/2005
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Hsu, Bo-June, Mahajan, Milind V., Acero, Alejandro
• Primary Examiner(s): Dorvil; Richemond
• Assistant Examiner(s): GODBOLD, DOUGLAS
• Application Number:
  US11/262,230
• Publication Number:
  US 20070100635A1
• Time in Patent Office: 2,020 Days
• Field of Search: 704/231, 704/235, 704/251
• US Class Current: 704/235
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

A method of entering information into a mobile device includes receiving a multi-word speech input from a user, performing speech recognition on the speech input to obtain a multi-word speech recognition result, and sequentially displaying, in a display, words in the speech recognition result for user confirmation or correction, by adding one word at a time to the display. A next word is only displayed after user confirmation or correct has been received for a previously displayed word that is immediately preceding the next word in the speech recognition result. The method also includes calculating a hypothesis lattice indicative of a plurality of speech recognition hypotheses based on the speech input and, prior to finishing calculating the hypothesis lattice and while continuing to calculate the hypothesis lattice, calculating a preliminary hypothesis lattice indicative of only partial speech recognition hypotheses based on the speech input and outputting the preliminary hypotheses lattice.

Claims

1. A method of entering information into a mobile device, comprising:

receiving a multi-word speech input from a user;

performing speech recognition on the speech input to obtain a multi-word speech recognition result; and

sequentially displaying, in a display, words in the speech recognition result for user confirmation or correction, by adding one word at a time to the display, regardless of how many words are in the speech recognition result, and only displaying a next word in the speech recognition result after user confirmation or correction has been received for a previously displayed word that is immediately preceding the next word in the speech recognition result;

wherein performing speech recognition comprises;

calculating a hypothesis lattice indicative of a plurality of speech recognition hypotheses based on the speech input;

prior to finishing calculating the hypothesis lattice and while continuing to calculate the hypothesis lattice, calculating a preliminary hypothesis lattice indicative of only partial speech recognition hypotheses based on the speech input and outputting the preliminary hypothesis lattice; and

wherein sequentially displaying the speech recognition result for user correction or confirmation comprises displaying the speech recognition result first using the preliminary hypothesis lattice, until the hypothesis lattice is completely calculated, and then displaying the speech recognition result using the completely calculated hypothesis lattice.

2. The method of claim 1 and further comprising:

receiving a user input indicative of user correction or confirmation of a given, displayed word.

3. The method of claim 2 wherein receiving a user input comprises:

receiving an alternate modality input indicative of user actuation of an input mechanism on the mobile device.

4. The method of claim 3 wherein sequentially displaying the speech recognition result, one word at a time comprises:

displaying a best hypothesis word; and

displaying a list of alternate words proximate the best hypothesis.

5. The method of claim 4 wherein receiving an alternate modality input comprises:

receiving a user selection of one of the alternate words.

6. The method of claim 4 wherein receiving an alternate modality input comprises:

receiving a user typing input indicative of a user inputting a desired word, one letter at a time.

7. The method of claim 1 wherein performing speech recognition comprises calculating a sequence of likely words corresponding to the speech input, and further comprising:

prior to adding the next word to the display of the speech recognition results, after the given displayed word, recalculating the hypothesis lattice based on the user input indicative of user correction or confirmation of the given displayed word.

8. The method of claim 7 wherein displaying comprises:

adding the next word to the display, based on the recalculation of the hypothesis lattice.

9. A mobile device, comprising:

a speech recognizer;

a user interface component receiving a speech recognition result from the speech recognizer indicative of recognition of a multi-word speech input and sequentially displaying words in the speech recognition result by displaying a next word in the speech recognition result and displaying a plurality of alternate words in a list as alternates to the displayed next word, wherein the next word in the speech recognition result and the plurality of alternate words are automatically displayed after a previously output word in the speech recognition result has been confirmed or corrected by a user and before any words following the next word in the speech recognition result are displayed, the user interface component adding only the next word and the plurality of alternate words to the display, regardless of how many additional words are in the speech recognition result following the next word, and wherein the user interface component is configured to receive a user input that identifies one of the alternate words in the list as being a correct word in the speech recognition result; and

wherein the speech recognizer provides the speech recognition result to the user interface component by calculating a likely recognition hypothesis lattice indicative of a plurality of speech recognition hypotheses based on the speech input, and wherein the speech recognizer is configured to recalculate the recognition hypothesis lattice based on the user input that identifies one of the alternate words in the list, and wherein a subsequent word following the identified correct word in the speech recognition result is displayed and a plurality of alternate words are provided for the subsequent word based on the recalculation of the recognition hypothesis lattice.

10. The mobile device of claim 9 and further comprising:

a display, the user interface component being configured to output the speech recognition result by sequentially displaying the speech recognition result on the display.

11. The mobile device of claim 10 wherein the speech recognizer provides the speech recognition result by outputting a preliminary recognition hypothesis lattice indicative of a preliminary, likely sequence of words and wherein the user interface component displays the speech recognition result to the user initially based on the preliminary, recognition hypothesis lattice.

12. The mobile device of claim 9, wherein the user confirmation comprising the user input does not modify the displayed word in the speech recognition result.

13. A mobile device, comprising:

a user actuable input modality component;

a display;

a speech recognition system configured to calculate a hypothesis lattice indicative of a plurality of speech recognition hypotheses based on a received multi-word speech input, wherein, prior to finishing calculating the hypothesis lattice and while calculating the hypothesis lattice, the speech recognition system calculates a preliminary hypothesis lattice indicative of only partial speech recognition hypotheses for the multi-word speech input and outputs the preliminary hypothesis lattice; and

a user interface component configured to display a list of words indicative of the received multi-word speech input, wherein each word in the list is sequentially displayed for user confirmation or correction prior to displaying a next word in the list, regardless of a number of words in the list, and wherein the next word in the list is displayed only after receiving user confirmation or correction for an immediately previous word in the list, wherein a portion of the list of words indicative of the received multi-word speech input is first displayed using the preliminary hypothesis lattice, until the hypothesis lattice is completely calculated, and then the list of words indicative of the received multi-word speech input is displayed using the completely calculated hypothesis lattice.

14. The user interface of claim 13 wherein the user confirmation is provided through the user actuable input modality component.

15. The user interface of claim 14 wherein the user actuable input modality component comprises one of a keypad, a soft keyboard, a handwriting input component, and a touch screen.

16. The user interface of claim 13 wherein the user interface component is configured to display, along with each word in the list, a set of alternate words for user selection.

17. The mobile device of claim 13, wherein the user confirmation comprising the user input does not modify the displayed word in the list of words.