Handwriting symbol recognition accuracy using speech input

• US 8,077,975 B2
• Filed: 02/26/2008
• Issued: 12/13/2011
• Est. Priority Date: 02/26/2008
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ma, Lei, Shi, Yu, Soong, Frank Kao-ping
• Primary Examiner(s): DULANEY, KATHLEEN YUAN
• Application Number:
  US12/037,095
• Publication Number:
  US 20090214117A1
• Time in Patent Office: 1,386 Days
• Field of Search: 382/182, 382/187, 382/228, 382/229, 345/156, 715/708, 704/255
• US Class Current: 382/187
• CPC Class Codes: G06V 30/333 Preprocessing; Feature extraction
• CPC Class Codes: G06V 30/36 Matching; Classification
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features

Abstract

Described is a bimodal data input technology by which handwriting recognition results are combined with speech recognition results to improve overall recognition accuracy. Handwriting data and speech data corresponding to mathematical symbols are received and processed (including being recognized) into respective graphs. A fusion mechanism uses the speech graph to enhance the handwriting graph, e.g., to better distinguish between similar handwritten symbols that are often misrecognized. The graphs include nodes representing symbols, and arcs between the nodes representing probability scores. When arcs in the first and second graphs are determined to match one another, such as aligned in time and associated with corresponding symbols, the probability score in the second graph for that arc is used to adjust the matching probability score in the first graph. Normalization and smoothing may be performed to correspond the graphs to one another and to control the influence of one graph on the other.

Claims

1. One or more computer-readable storage devices having computer-executable instructions, which when executed by one or more processing devices, cause the one or more processing devices to perform:

receiving handwriting input including handwritten mathematical symbols;

receiving speech input including spoken words corresponding to spoken mathematical symbols;

recognizing the handwriting input to identify candidate handwritten mathematical symbols;

recognizing the speech input to identify candidate spoken words;

processing the candidate handwritten mathematical symbols into a handwriting graph including nodes representing the candidate handwritten mathematical symbols and arcs representing probabilities that the candidate handwritten mathematical symbols are correct;

processing the candidate spoken words into a speech graph by;

accessing a lookup table with the candidate spoken words recognized from the speech input to identify candidate spoken mathematical symbols; and

including nodes in the speech graph representing the candidate spoken mathematical symbols and arcs representing probabilities that the candidate spoken mathematical symbols identified from the lookup table are correct; and

enhancing the handwriting graph into an enhanced graph by adjusting individual probabilities in the handwriting graph based on individual matching probabilities in the speech graph.

2. The one or more computer-readable storage devices of claim 1 wherein recognizing the speech input comprises generating a word lattice that includes the candidate spoken words, and the word lattice is converted into the speech graph.

3. The one or more computer-readable storage devices of claim 1 having further computer-executable instructions, which, when executed by one or more processing devices, cause the one or more processing devices to perform:

performing symbol matching and time-alignment matching to identify the individual matching probabilities in the speech graph.

4. The one or more computer-readable storage devices of claim 1 having further computer-executable instructions which when executed by the one or more processing devices, cause the one or more processing devices to perform:

outputting the enhanced graph from which a mathematical expression is computed.

5. The one or more computer-readable storage devices of claim 1, wherein the lookup table comprises a first word and a second word that both identify an individual candidate spoken mathematical symbol.

6. The one or more computer-readable storage devices of claim 5, the first word comprising the word “

plus,”

the second word comprising the word “

add,” and

the individual candidate spoken mathematical symbol comprising the symbol “

+.”

7. A method comprising:

receiving handwriting input including handwritten mathematical symbols;

receiving speech input including a spoken word corresponding to a spoken mathematical symbol;

recognizing the handwriting input to identify candidate handwritten mathematical symbols;

recognizing the speech input to identify a candidate spoken word;

processing the candidate handwritten mathematical symbols into a handwriting graph reflecting probabilities, represented by arcs, that the candidate handwritten mathematical symbols, represented by nodes, correctly identify the handwritten mathematical symbols;

processing the candidate spoken word by;

accessing a lookup table with the candidate spoken word recognized from the speech input to identify a candidate spoken mathematical symbol; and

generating a speech graph reflecting a probability, represented by an arc, that the candidate spoken mathematical symbol, represented by a node, correctly identifies the spoken mathematical symbol; and

fusing the handwriting graph and the speech graph to create an enhanced graph by adjusting individual probabilities of the handwriting graph based on individual matching probabilities in the speech graph,

wherein at least the fusing is performed by a processing device.

8. The method according to claim 7, wherein the lookup table comprises a second word that also identifies the candidate spoken mathematical symbol.

9. The method according to claim 8, the candidate spoken word comprising the word “

plus,”

the second word comprising the word “

add,” and

the candidate spoken mathematical symbol comprising the symbol “

+.”

10. The method of claim 7, wherein recognizing the speech input comprises generating a word lattice that includes the candidate spoken word and converting the word lattice into the speech graph.

11. The method of claim 7, further comprising:

performing symbol matching and time-alignment matching to identify the individual matching probabilities in the speech graph.

12. The method of claim 7, further comprising:

outputting the enhanced graph, from which a mathematical expression is computed.

13. A system comprising:

a handwriting recognizer configured to;

receive handwriting input including handwritten mathematical symbols;

recognize the handwriting input to identify candidate handwritten mathematical symbols; and

generate a handwriting graph reflecting probabilities, represented by arcs, that the candidate handwritten mathematical symbols, represented by nodes, correctly identify the handwritten mathematical symbols;

a speech recognizer configured to;

receive speech input including a spoken word corresponding to a spoken mathematical symbol;

recognize the speech input to identify a candidate spoken word;

access a lookup table with the candidate spoken word recognized from the speech input to identify a candidate spoken mathematical symbol; and

generate a speech graph reflecting a probability, represented by an arc, that the candidate spoken mathematical symbol, represented by a node, correctly identifies the spoken mathematical symbol;

a fusion mechanism configured to fuse the handwriting graph and the speech graph to create an enhanced graph by adjusting individual probabilities of the handwriting graph based on individual matching probabilities in the speech graph; and

at least one processing device configured to execute the handwriting recognizer, the speech recognizer, or the fusion mechanism.

14. The system of claim 13, wherein the speech recognizer is configured to recognize the speech input by generating a word lattice that includes the candidate spoken word and converting the word lattice into the speech graph.

15. The system of claim 13, wherein the fusion mechanism is further configured to perform symbol matching and time-alignment matching to identify the individual matching probabilities in the speech graph.

16. The system of claim 13, wherein the fusion mechanism is further configured to output the enhanced graph, from which a mathematical expression is computed.

17. The system of claim 13, wherein the lookup table comprises a second word that also identifies the candidate spoken mathematical symbol.

18. The system of claim 17, the candidate spoken word comprising the word “

plus,”

the second word comprising the word “

add,” and

the candidate spoken mathematical symbol comprising the symbol “

+.”