Multi-modal handwriting recognition correction

• US 7,506,271 B2
• Filed: 12/15/2003
• Issued: 03/17/2009
• Est. Priority Date: 12/15/2003
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wang, Xianfang, Wu, Jiang, Zhou, Jian-Lai, Wang, Jian, Yang, Hongyun, Zhu, Wenli
• Primary Examiner(s): Hong; Stephen
• Assistant Examiner(s): Termanini; Samir
• Application Number:
  US10/734,305
• Publication Number:
  US 20050128181A1
• Time in Patent Office: 1,919 Days
• Field of Search: 715/809
• US Class Current: 715/809
• CPC Class Codes: G06V 30/2272 with lexical matching
• CPC Class Codes: G06V 30/387 using human interaction, e.g. selection of the best displayed recognition candidate
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features

Abstract

Systems, methods, and computer-readable media for processing electronic ink receive an electronic ink input; convert the electronic ink input to a first machine-generated object using handwriting recognition; display the first machine-generated object on a display; receive speech input; convert the speech input to a second machine-generated object using speech recognition; generate a list of machine-generated objects based on the electronic ink input, the list including the first machine-generated object and alternative machine-generated objects and functioning as a dictionary for converting the speech input; and replace the first machine-generated object with the second machine-generated object. A user may confirm that the second machine-generated object should replace the first machine-generated object. The systems and methods may generate a list of alternative machine-generated object candidates to the first machine-generated object based on handwriting recognition of the electronic ink input alone or in combination with a statistical language model.

Claims

1. A method comprising:

receiving electronic ink input;

generating a list of machine-generated text candidates based on the electronic ink input, the list including a first machine-generated text candidate and alternative machine-generated text candidates;

converting the electronic ink input to the first machine-generated text candidate;

displaying the first machine-generated text candidate;

receiving speech input;

converting the speech input to second machine-generated text, wherein the second machine-generated text is one of the alternative machine-generated text candidates and the list of machine-generated text candidates functions as a dictionary used for converting the speech input pursuant to a statistical language model, to generate for every instance of a spoken word d_k in the dictionary, the similarity between d_k and list of machine-generated text candidates w_i,j being represented as afunction defined as the logarithmic value of a count of the matched characters w_i,j, d_k divided by a value of the sum of the length of w_i,j and d_k;

analyzing when the string similarity between d_k and w_i,j is very small, including the optimum constraint, with d_k being completely discounted as a candidate of w_i for converting the speech into text; and

replacing the first machine-generated text candidate with the second machine-generated text.

2. The method of claim 1, wherein the first machine-generated text candidate is a word.

3. The method of claim 1, wherein the first machine-generated text candidate is a portion of a word.

4. The method according to claim 1, further comprising receiving input selecting the first machine-generated text candidate prior to receiving the speech input.

5. The method according to claim 4, wherein the selecting includes touching a user input device to a digitizer screen at a location corresponding to the first machine-generated text candidate.

6. The method according to claim 4, wherein the first machine-generated text candidate is a group of words or part of a word.

7. The method according to claim 1, further including displaying the list of machine-generated text candidates prior to receiving the speech input.

8. The method according to claim 7, wherein said step of displaying the alternative machine-generated text candidates further includes displaying the alternative machine-generated text candidates in the list in an order based on a confidence level that each alternative machine-generated text candidate corresponds to the electronic ink input.

9. The method according to claim 1, wherein the alternative machine-generated text candidates include machine-generated text candidates based on the electronic ink input generated by a handwriting recognition engine.

10. The method according to claim 1, further comprising displaying the machine-generated text candidates generated by the handwriting recognition engine and subsequently displaying the machine-generated text candidates generated in accordance with the statistical language model.

11. The method according to claim 10, further comprising receiving input requesting the display of the machine-generated text candidates generated in accordance with the statistical language model while displaying the machine-generated text candidates generated by the handwriting recognition engine.

12. The method according to claim 1, wherein the alternative machine-generated text candidates include text candidates based on the electronic ink input generated by a statistical language model.

13. The method according to claim 1, wherein the step of converting the speech input to the second machine-generated text includes

determining if the speech input corresponds to one of the alternative machine-generated text candidates;

and

converting the speech input to the corresponding alternative machine-generated text candidate when the speech input corresponds to the alternative machine-generated text candidate.

14. The method according to claim 1, wherein further comprising the step of receiving an input confirming that the second machine-generated text candidate should replace the first machine-generated text candidate prior to performing said step of replacing.

15. A computer-readable medium including computer-executable instructions stored thereon for performing the method of claim 1.

45. The method of claim 1, wherein the speech input operates pursuant to a statistical language model, to generate for every instance of a spoken word d_k in the dictionary, the similarity between d_k and list of machine-generated text candidates w_i,j is defined as

$s_{\mathrm{sim}}$

( d k , w i , j ) = log ⁢

⁢

( c match ⁡

( w i , j , d k ) len ⁡

( w i , j ) + len ⁡

( d k ) )

with a C_match (w_i,j, d_k) being the count of the matched characters between d_k and w_i,j (considering order), the len(w_i,j) is the character count of word w_i,j, and the len(d_k) is the character count of word d_k, where the statistical score of d_k in string similarity is defined as

s_sim (d_k)=max(s_sim(d_k,w_i,j))

analyzing when the string similarity between d_k and w_i,j is very small, including the optimum constraint, with d_k being completely discounted as a candidate of w_i for converting the speech into text; and

replacing the first machine-generated text candidate with the second machine-generated text.

16. A method for recognizing an input, comprising:

receiving electronic ink input;

generating a list of machine-generated objects based on the electronic ink input, the list including a first machine-generated object and alternative machine-generated objects;

converting the electronic ink input to the first machine-generated object;

displaying the first machine-generated object;

receiving speech input;

converting the speech input to a second machine-generated object, wherein converting of the speech input is performed based on the list of machine-generated objects and wherein the second machine-generated object is one of the list of machine-generated objects pursuant to a statistical language model to generate every instance of a machine-generated object functioning as a dictionary, the similarity between the dictionary and the list of machine-generated objects being represented as a function defined as the logarithmic value of a count of the matched characters divides by the value of the sum of the length of machine-generated objects and every instance of the machine-generated object functioning as the dictionary, and analyzing when the string similarity between the length of machine-generated objects and every instance of the machine-generated object functioning as the dictionary is very small, including the optimum constraint, with every instance of the machine-generated Object functioning as the dictionary being completely discounted as a candidate of the list of machine-generated objects for converting speech into text; and

replacing the first machine-generated object with the second machine-generated object when the second machine-generated object is different from the first machine-generated object.

17. The method according to claim 16, further comprising the step of receiving input confirming that the second machine-generated object should replace the first machine- generated object prior to performing said step of replacing.

18. The method according to claim 16, further comprising receiving input selecting the first machine-generated object prior to receiving the speech input.

19. The method according to claim 16, further comprising displaying a list of alternative machine-generated objects on the display prior to receiving the speech input.

20. The method according to claim 19, wherein said step of displaying the alternative machine-generated objects further includes displaying the alternative machine-generated objects in the list in an order based on a confidence level that each alternative machine-generated object corresponds to the electronic ink input.

21. The method according to claim 16, wherein the alternative machine-generated objects include objects based on the electronic ink input generated by a handwriting recognition engine.

22. The method according to claim 16, further comprising displaying the machine-generated objects generated by the handwriting recognition engine and subsequently displaying the machine-generated objects generated in accordance with the statistical language model.

23. The method according to claim 22, receiving input requesting the display of the machine-generated objects generated in accordance wit the statistical language model while displaying the machine-generated objects generated by the handwriting recognition engine.

24. The method according to claim 16, wherein the alternative machine-generated objects include machine-generated objects based on the electronic ink input generated by a statistical language model.

25. The method according to claim 16, wherein the step of converting the speech input to the second machine-generated object includes

determining if the speech input corresponds to one of the alternative machine-generated objects;

and

converting the speech input to the corresponding alternative machine-generated object when the speech input corresponds to the alternative machine-generated object.

26. A computer-readable medium including computer-executable instructions stored thereon for performing the method of claim 16.

46. The method of claim 16, wherein the speech input operates pursuant to a statistical language model, to generate for every instance of a spoken word d_k in the dictionary, the similarity between d_k and list of machine-generated text candidates w_i,j is defined as

$s_{\mathrm{sim}}$

( d k , w i , j ) = log ⁢

⁢

( c match ⁡

( w i , j , d k ) len ⁡

( w i , j ) + len ⁡

( d k ) )

with a c_match (w_i,j, d_k) being the count of the matched characters between d_k and w_i,j (considering order), the len(w_i,j) is the character count of word w_i,j, and the len(d_k) is the character count of word d_k, where the statistical score of d_k in string similarity is defined as

s_sim (d_k)=Max(s_sim(d_k,w_i,j))

analyzing when the string similarity between d_k and w_i,j is very small, including the optimum constraint, with 4 being completely discounted as a candidate of w_i for converting the speech into text; and

replacing the first machine-generated text candidate with the second machine-generated text.

27. A system comprising:

a display;

a first input adapted to receive electronic ink input;

a second input adapted to receive speech input; and

a processor programmed and adapted to;

(a) convert the electronic ink input to first machine-generated text using handwriting recognition;

(b) display the first machine-generated text on the display;

(c) convert the speech input to second machine-generated text using speech recognition;

(d) generate a list of machine-generated text candidates based on the electronic ink input, the list including a first machine-generated text candidate and alternative machine-generated text candidates and pursuant to a statistical language model to generate every instance of an alternative machine-generated text candidates functioning as a dictionary, the similarity between the dictionary and the list of alternative machine-generated text candidates being represented as a function defined as the logarithmic value of a count of the matched characters divides by the value of the sum of the length of alternative machine-generated text candidates and every instance of the alternative machine-generated text candidate functioning as the dictionary, and analyzing when the string similarity between the length of alternative machine-generated text candidates and every instance of the alternative machine-generated text candidate functioning as the dictionary very small, including the optimum constraint, with every instance of the alternative machine-generated text candidate functioning as the dictionary being completely discounted as a candidate of the list of alternative machine-generated text candidates for converting speech into text and functioning as a dictionary for converting the speech input;

and (e) replace the first machine-generated text candidate with the second machine-generated text.

28. The system according to claim 27, wherein the first machine-generated text is a word.

29. The system according to claim 27, wherein one of the first or second inputs is adapted to receive a confirmation input that the second machine-generated text is correct before replacing the first machine-generated text with the second machine-generated text.

30. The system according to claim 27, wherein the first input is further adapted to receive a selection of a portion of the first machine-generated text for correction.

31. The system according to claim 30, wherein the selected portion is the entire first machine-generated text.

32. The system according to claim 27, wherein said processor is further programmed to display the alternative machine-generated text candidates on the display prior to the second input receiving the speech input.

33. The system according to claim 27, wherein said processor is further programmed to display the alternative machine-generated text candidates on the display in an order in the list based on a confidence level that the respective alternative machine-generated text candidate corresponds to the electronic ink input.

34. The system according to claim 27, wherein said processor is programmed to generate the alternative machine-generated text candidates based on the electronic ink input using a handwriting recognition engine.

35. The system according to claim 27, wherein the first input is further adapted to receive a request to display the machine-generated text candidates generated in accordance with the statistical language model while displaying the machine-generated text candidates generated by the handwriting recognition engine.

36. The system according to claim 27, wherein said processor is programmed to generate the alternative machine-generated text candidates based on the electronic ink input using a statistical language model.

37. The system according to claim 27, wherein said processor is further programmed to determine if the speech input corresponds to one of the alternative machine-generated text candidates;

and convert the speech input to the corresponding alternative machine-generated text candidate when the speech input corresponds to the alternative machine-generated text candidate.

47. The method of claim 27, wherein the speech input operates pursuant to a statistical language model, to generate for every instance of a spoken word d_k in the dictionary, the similarity between d_k and list of machine-generated text candidates w_i,j is defined as

$s_{\mathrm{sim}}$

( d k , w i , j ) = log ⁢

⁢

( c match ⁡

( w i , j , d k ) len ⁡

( w i , j ) + len ⁡

( d k ) )

with a c_match (w_i,j, d_k) being the count of the matched characters between d_k and w_i,j (considering order), the len(w_i,j) is the character count of word w_i,j , and the len(d_k) is the character count of word d_k, where the statistical score of d_k in string similarity is defined as

s_sim(d_k)=max(s_sim(d_k,w_i,j))

analyzing when the string similarity between d_k and w_i,j is very small, including the optimum constraint, with d_k being completely discounted as a candidate of w_i for converting the speech into text; and

replacing the first machine-generated text candidate with the second machine-generated text.

38. A system for recognizing an input, comprising:

a display;

a first input adapted to receive an electronic ink input;

a second input adapted to receive speech input; and

a processor programmed and adapted to;

(a) convert the electronic ink input to a first machine-generated object using handwriting recognition;

(b) display the first machine-generated object on the display;

(c) generate a list of machine-generated objects based on the electronic ink input, the list including the first machine-generated object and alternative machine-generated objects pursuant to a statistical language model to generate every instance of an alternative machine-generated objects functioning as a dictionary, the similarity between the dictionary and the list of alternative machine-generated objects being represented as a function defined as the logarithmic value of a count of the matched characters divides by the value of the sum of the length of alternative machine-generated objects and every instance of the alternative machine-generated object functioning as the dictionary, and analyzing when the string similarity between the length of alternative machine-generated objects and every instance of the alternative machine-generated object functioning as the dictionary is very small, including the optimum constraint, with every instance of the alternative machine-generated object functioning as the dictionary being completely discounted as a candidate of the list of alternative machine-generated objects for converting speech into text and;

(d) convert the speech input to a second machine-generated object using speech recognition, wherein the conversion of the speech input is performed based on the list of machine-generated objects and wherein the second machine-generated object is one of the list of machine-generated objects; and

(e) replace the first machine-generated object with the second machine-generated object when the second machine-generated object is different from the first machine-generated object.

39. The system according to claim 38, wherein one of the first or second inputs is adapted to receive a confirmation input that the second machine-generated object is correct before replacing the first machine-generated object with the second machine-generated object.

40. The system according to claim 38, wherein said processor is further programmed to display the alternative machine-generated objects on the display prior to the second input receiving the speech input.

41. The system according to claim 38, wherein said processor is programmed to generate the alternative machine-generated objects based on the electronic ink input using a handwriting recognition engine.

42. The system according to claim 41, wherein said processor is programmed to generate the alternative machine-generated objects based on the electronic ink input using a statistical language model.

43. The system according to claim 42, wherein the first input is further adapted to receive a request to display the machine-generated objects generated in accordance with the statistical language model while displaying the machine-generated objects generated by the handwriting recognition engine.

44. The system according to claim 41, wherein said processor is further programmed to determine if the speech input corresponds to one of the alternative machine-generated objects;

and convert the speech input to the corresponding alternative machine-generated object when the speech input corresponds to the alternative machine-generated object.

48. The method of claim 38, wherein the speech input operates pursuant to a statistical language model, to generate for every instance of a spoken word d_k in the dictionary, the similarity between d_k and list of machine-generated text candidates w_i,j is defined as

$s_{\mathrm{sim}}$

( d k , w i , j ) = log ⁢

⁢

( c match ⁡

( w i , j , d k ) len ⁡

( w i , j ) + len ⁡

( d k ) )

with a c_match (w_i,j, d_k) being the count of the matched characters between d_k and w_i,j (considering order), the len(w_i,j) is the character count of word w_i,j, and the len(d_k)is the character count of word d_k, where the statistical score of d_k in string similarity is defined as

s_sim(d_k)=max(s_sim(d_k,w_i,j))

analyzing when the string similarity between d_kand w_i,j is very small, including the optimum constraint, with d_k being completely discounted as a candidate of w_i for converting the speech into text; and

replacing the first machine-generated text candidate with the second machine-generated text.