Multi-modal handwriting recognition correction

• US 20050128181A1
• Filed: 12/15/2003
• Published: 06/16/2005
• Est. Priority Date: 12/15/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wang, Xianfang, Wu, Jiang, Zhou, Jian-Lai, Wang, Jian, Yang, Hongyun, Zhu, Wenli
• Granted Patent: US 7,506,271 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06V 30/2272 with lexical matching
• CPC Class Codes: G06V 30/387 using human interaction, e.g. selection of the best displayed recognition candidate
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features

Abstract

Systems, methods, and computer-readable media for processing electronic ink receive an electronic ink input; convert the electronic ink input to a first machine-generated object using handwriting recognition; display the first machine-generated object on a display; receive speech input; convert the speech input to a second machine-generated object using speech recognition; generate a list of machine-generated objects based on the electronic ink input, the list including the first machine-generated object and alternative machine-generated objects and functioning as a dictionary for converting the speech input; and replace the first machine-generated object with the second machine-generated object. The machine-generated objects may correspond to words, lines, and/or other groupings of machine-generated text. A user may confirm that the second machine-generated object should replace the first machine-generated object and the system will perform the replacement. The systems and methods may generate a list of alternative machine-generated object candidates to the first machine-generated object based on handwriting recognition of the electronic ink input alone or in combination with a statistical language model.

Claims

1. A method comprising:

receiving electronic ink input;

generating a list of machine-generated text candidates based on the electronic ink input, the list including a first machine-generated text candidate and alternative machine-generated text candidates;

converting the electronic ink input to the first machine-generated text candidate;

displaying the first machine-generated text candidate;

receiving speech input;

converting the speech input to second machine-generated text, wherein the second machine-generated text is one of the alternative machine-generated text candidates and the list of machine-generated text candidates functions as a dictionary used for converting the speech input; and

replacing the first machine-generated text candidate with the second machine-generated text.

2. The method of claim 1, wherein the first machine-generated text candidate is a word.

3. The method of claim 1, wherein the first machine-generated text candidate is a portion of a word.

4. The method according to claim 1, further comprising receiving input selecting the first machine-generated text candidate prior to receiving the speech input.

5. The method according to claim 4, wherein the selecting includes touching a user input device to a digitizer screen at a location corresponding to the first machine-generated text candidate.

6. The method according to claim 4, wherein the first machine-generated text candidate is a group of words or part of a word.

7. The method according to claim 1, further including displaying the list of machine-generated text candidates prior to receiving the speech input.

8. The method according to claim 7, wherein said step of displaying the alternative machine-generated text candidates further includes displaying the alternative machine-generated text candidates in the list in an order based on a confidence level that each alternative machine-generated text candidate corresponds to the electronic ink input.

9. The method according to claim 1, wherein the alternative machine-generated text candidates include machine-generated text candidates based on the electronic ink input generated by a handwriting recognition engine.

10. The method according to claim 9, wherein the alternative machine-generated text candidates include machine-generated text candidates based on the electronic ink input generated in accordance with a statistical language model.

11. The method according to claim 10, further comprising displaying the machine-generated text candidates generated by the handwriting recognition engine and subsequently displaying the machine-generated text candidates generated in accordance with the statistical language model.

12. The method according to claim 11, further comprising receiving input requesting the display of the machine-generated text candidates generated in accordance with the statistical language model while displaying the machine-generated text candidates generated by the handwriting recognition engine.

13. The method according to claim 1, wherein the alternative machine-generated text candidates include text candidates based on the electronic ink input generated by a statistical language model.

14. The method according to claim 1, wherein the step of converting the speech input to the second machine-generated text includes determining if the speech input corresponds to one of the alternative machine-generated text candidates;

and

converting the speech input to the corresponding alternative machine-generated text candidate when the speech input corresponds to the alternative machine-generated text candidate.

15. The method according to claim 1, wherein further comprising the step of receiving an input confirming that the second machine-generated text candidate should replace the first machine-generated text candidate prior to performing said step of replacing.

16. A computer-readable medium including computer-executable instructions stored thereon for performing the method of claim 1.

17. A method, comprising:

receiving electronic ink input;

generating a list of machine-generated objects based on the electronic ink input, the list including a first machine-generated object and alternative machine-generated objects;

converting the electronic ink input to the first machine-generated object;

displaying the first machine-generated object;

receiving speech input;

converting the speech input to a second machine-generated object, wherein the second machine-generated object is one of the alternative machine-generated objects and the list of machine-generated objects functions as a dictionary used for converting the speech input; and

replacing the first machine-generated object with the second machine-generated object.

18. The method according to claim 17, further comprising the step of receiving input confirming that the second machine-generated object should replace the first machine-generated object prior to performing said step of replacing.

19. The method according to claim 17, further comprising receiving input selecting the first machine-generated object prior to receiving the speech input.

20. The method according to claim 17, further comprising displaying a list of alternative machine-generated objects on the display prior to receiving the speech input.

21. The method according to claim 20, wherein said step of displaying the alternative machine-generated objects further includes displaying the alternative machine-generated objects in the list in an order based on a confidence level that each alternative machine-generated object corresponds to the electronic ink input.

22. The method according to claim 17, wherein the alternative machine-generated objects include objects based on the electronic ink input generated by a handwriting recognition engine.

23. The method according to claim 22, wherein the alternative machine-generated objects include machine-generated objects based on the electronic ink input generated in accordance with a statistical language model.

24. The method according to claim 23, further comprising displaying the machine-generated objects generated by the handwriting recognition engine and subsequently displaying the machine-generated objects generated in accordance with the statistical language model.

25. The method according to claim 24, receiving input requesting the display of the machine-generated objects generated in accordance with the statistical language model while displaying the machine-generated objects generated by the handwriting recognition engine.

26. The method according to claim 17, wherein the alternative machine-generated objects include machine-generated objects based on the electronic ink input generated by a statistical language model.

27. The method according to claim 17, wherein the step of converting the speech input to the second machine-generated object includes

determining if the speech input corresponds to one of the alternative machine-generated objects;

and

converting the speech input to the corresponding alternative machine-generated object when the speech input corresponds to the alternative machine-generated object.

28. A computer-readable medium including computer-executable instructions stored thereon for performing the method of claim 17.

29. A system comprising:

a display;

a first input adapted to receive electronic ink input;

a second input adapted to receive speech input; and

a processor programmed and adapted to;

(a) convert the electronic ink input to first machine-generated text using handwriting recognition;

(b) display the first machine-generated text on the display;

(c) convert the speech input to second machine-generated text using speech recognition;

(d) generate a list of machine-generated text candidates based on the electronic ink input, the list including a first machine-generated text candidate and alternative machine-generated text candidates and functioning as a dictionary for converting the speech input; and

(e) replace the first machine-generated text candidate with the second machine-generated text.

30. The system according to claim 29, wherein the first machine-generated text is a word.

31. The system according to claim 29, wherein one of the first or second inputs is adapted to receive a confirmation input that the second machine-generated text is correct before replacing the first machine-generated text with the second machine-generated text.

32. The system according to claim 29, wherein the first input is further adapted to receive a selection of a portion of the first machine-generated text for correction.

33. The system according to claim 32, wherein the selected portion is the entire first machine-generated text.

34. The system according to claim 29, wherein said processor is further programmed to display the alternative machine-generated text candidates on the display prior to the second input receiving the speech input.

35. The system according to claim 29, wherein said processor is further programmed to display the alternative machine-generated text candidates on the display in an order in the list based on a confidence level that the respective alternative machine-generated text candidate corresponds to the electronic ink input.

36. The system according to claim 29, wherein said processor is programmed to generate the alternative machine-generated text candidates based on the electronic ink input using a handwriting recognition engine.

37. The system according to claim 36, wherein said processor is programmed to generate the alternative machine-generated text candidates based on the electronic ink input using a statistical language model.

38. The system according to claim 37, wherein the first input is further adapted to receive a request to display the machine-generated text candidates generated in accordance with the statistical language model while displaying the machine-generated text candidates generated by the handwriting recognition engine.

39. The system according to claim 29, wherein said processor is programmed to generate the alternative machine-generated text candidates based on the electronic ink input using a statistical language model.

40. The system according to claim 29, wherein said processor is further programmed to determine if the speech input corresponds to one of the alternative machine-generated text candidates;

and convert the speech input to the corresponding alternative machine-generated text candidate when the speech input corresponds to the alternative machine-generated text candidate.

41. A system comprising:

a display;

a first input adapted to receive an electronic ink input;

a second input adapted to receive speech input; and

a processor programmed and adapted to;

(a) convert the electronic ink input to a first machine-generated object using handwriting recognition;

(b) display the first machine-generated object on the display;

(c) convert the speech input to a second machine-generated object using speech recognition;

(d) generate a list of machine-generated objects based on the electronic ink input, the list including the first machine-generated object and alternative machine-generated objects and functioning as a dictionary for converting the speech input; and

(e) replace the first machine-generated object with the second machine-generated object.

42. The system according to claim 41, wherein one of the first or second inputs is adapted to receive a confirmation input that the second machine-generated object is correct before replacing the first machine-generated object with the second machine-generated object.

43. The system according to claim 41, wherein said processor is further programmed to display the alternative machine-generated objects on the display prior to the second input receiving the speech input.

44. The system according to claim 41, wherein said processor is programmed to generate the alternative machine-generated objects based on the electronic ink input using a handwriting recognition engine.

45. The system according to claim 44, wherein said processor is programmed to generate the alternative machine-generated objects based on the electronic ink input using a statistical language model.

46. The system according to claim 45, wherein the first input is further adapted to receive a request to display the machine-generated objects generated in accordance with the statistical language model while displaying the machine-generated objects generated by the handwriting recognition engine.

47. The system according to claim 41, wherein said processor is programmed to generate the alternative machine-generated objects based on the electronic ink input using a statistical language model.

48. The system according to claim 41, wherein said processor is further programmed to determine if the speech input corresponds to one of the alternative machine-generated objects;

and convert the speech input to the corresponding alternative machine-generated object when the speech input corresponds to the alternative machine-generated object.