Combined input processing for a computing device

• US 7,496,513 B2
• Filed: 06/28/2005
• Issued: 02/24/2009
• Est. Priority Date: 06/28/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Soong, Frank Kao-Ping, Tian, Ye, Zhou, Jian-Lai
• Primary Examiner(s): Hudspeth; David R
• Assistant Examiner(s): Rider; Justin W
• Application Number:
  US11/168,279
• Publication Number:
  US 20060290656A1
• Time in Patent Office: 1,337 Days
• Field of Search: 704/237, 704/260, 715/708, 715/811
• US Class Current: 704/260
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/0237 using prediction or retrieval techniques
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Input is received from at least two different input sources. Information from these sources are combined together to provide a result. In a particular example, input from one source corresponds to potential recognition candidates, and input from another source corresponds to other potential candidates. These candidates are combined together to select a result.

Claims

1. A computer-implemented method of determining and displaying a collection of text, the method comprising:

receiving a handwriting input from a first input source, wherein the handwriting input is an incomplete handwriting representation of the collection of text, the handwriting input being intentionally incomplete in that it is intentionally formed by a user to approximate only a limited portion of the collection of text, the limited portion being less than the entirety of the collection of text;

receiving a speech input from a second input source, wherein the speech input is a representation of the collection of text in its entirety;

determining the collection of text, the determination being based on a relative interrelation of recognition analysis performed on the handwriting input with recognition analysis performed on the speech input, wherein the relative interrelation is such that the determination is not independently dictated by the recognition analysis of either of the speech and handwriting inputs; and

displaying the collection of text on a display associated with a computing device.

2. The method of claim 1, wherein determining the collection of text comprises combining a handwriting recognition outcome probability with a speech recognition outcome probability, the handwriting recognition outcome probability and the speech recognition outcome probability each being a probability that a particular candidate string of text is the collection of text.

3. The method of claim 2, wherein neither of the handwriting and speech recognition outcome probabilities indicates an absolute certainty that the particular candidate string of text is the collection of text.

4. The method of claim 3, wherein determining the collection of text comprises utilizing the handwriting input as a basis for identifying a plurality of candidate strings of text, and wherein determining also comprises utilizing the speech input as a basis for assigning a likelihood to each of the plurality of candidate of strings, the likelihood being a probability that the candidate string is the collection of text, and wherein the likelihood is not an indication of absolute certainty that a particular candidate string is the collection of text.

5. The method of claim 1 wherein the input from the first input source corresponds to a plurality of potential candidates.

6. The method of claim 5 and further comprising:

selecting one candidate from the plurality of potential candidates as the result based on the input from the second source.

8. The method of claim 1, further comprising utilizing the determination to enhance the performance of the recognizer utilized to perform the recognition analysis on the handwriting input.

7. A method of identifying and displaying a text character, the method comprising:

receiving a first input corresponding to a plurality of different potential text character candidates;

receiving a second input, the second input being a speech input that is a representation of a specific one of the plurality of different potential text character candidates, the specific one being said text character;

determining the text character, the determination being based on a relative interrelation of recognition analysis performed on the first and second inputs, and wherein the relative interrelation involves processing the first input so as to identify the plurality of different potential text character candidates and processing the second input so as to determine the specific one of the different potential text character candidates; and

displaying the text character on a display associated with a computing device.

9. A computer-implemented method of identifying and displaying a collection of text, the method comprising:

receiving a handwriting input from a first input source, the handwriting input being a representation of at least a portion of the collection of text;

receiving a speech input from a second input source, the speech input being a representation of the collection of text;

performing handwriting recognition on the handwriting input so as to generate a plurality of handwriting recognition outcome candidates, each handwriting recognition outcome candidate being assigned a confidence score that indicates a likelihood that the candidate is the same as the collection of text;

performing speech recognition analysis on the speech input;

adjusting, based on a result of the speech recognition analysis, one or more of the confidence scores so as to increase the likelihood that the candidate is the same as the collection of text; and

determining whether any of the adjusted confidence scores is greater than a pre-established confidence threshold;

accepting as the collection of text a particular handwriting recognition outcome candidate associated with a confidence score greater than the pre-established confidence threshold;

displaying the particular recognition outcome candidate.

10. The method of claim 9, wherein adjusting comprises adjusting to a level that is still less than one hundred percent confidence.

11. The method of claim 9, wherein adjusting comprises adjusting to account for one or more confidence scores associated with a plurality of speech recognition outcome candidates.

12. The method of claim 9, wherein adjusting merging confidence scores associated with the handwriting recognition outcome candidates with confidence scores associated with a plurality of speech recognition outcome candidates generated based on the speech input as a result of the speech recognition analysis.