Combining online and offline recognizers in a handwriting recognition system

• US 7,953,279 B2
• Filed: 06/28/2007
• Issued: 05/31/2011
• Est. Priority Date: 06/28/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Chen, Xinjian, Zhang, Dongmei, Zou, Yu, Wang, Jian, Han, Shi, Chang, Ming
• Primary Examiner(s): Seth; Manav
• Application Number:
  US11/823,644
• Publication Number:
  US 20090003706A1
• Time in Patent Office: 1,433 Days
• Field of Search: 382/101, 382/102, 382155-161, 382181-231, 706/20, 715/268
• US Class Current: 382/187
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06F 18/29 Graphical models, e.g. Bayesian networks
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding

Abstract

Described is a technology by which online recognition of handwritten input data is combined with offline recognition and processing to obtain a combined recognition result. In general, the combination improves overall recognition accuracy. In one aspect, online and offline recognition is separately performed to obtain online and offline character-level recognition scores for candidates (hypotheses). A statistical analysis-based combination algorithm, an AdaBoost algorithm, and/or a neural network-based combination may determine a combination function to combine the scores to produce a result set of one or more results. Online and offline radical-level recognition may be performed. For example, a HMM recognizer may generate online radical scores used to build a radical graph, which is then rescored using the offline radical recognition scores. Paths in the rescored graph are then searched to provide the combined recognition result, e.g., corresponding to the path with the highest score.

Claims

1. At least one computer storage medium that is not a signal or carrier wave per se, the at least one computer storage medium storing computer-executable instructions that, when executed by a computer, cause the computer to perform a method comprising combining online recognition scores corresponding to character or radical level recognition of handwritten input data, with offline recognition scores corresponding to character or radical level recognition of the handwritten input data, to obtain a combined recognition result, and outputting the combined recognition result.

2. The at least one computer storage medium of claim 1 wherein combining online recognition scores with offline recognition scores comprises performing online recognition to obtain an online recognition result set corresponding to scores for whole character ink data, performing offline recognition to obtain an offline recognition result set corresponding to scores for whole character ink data, and combining the online recognition result set with the offline recognition result set via a combination mechanism to obtain the combined recognition result.

3. The at least one computer storage medium of claim 2 wherein combining the online recognition result set with the offline recognition result set comprises using a statistical analysis-based process to produce the function to combine the online scores with the offline scores.

4. The at least one computer storage medium of claim 2 wherein combining the online recognition result set with the offline recognition result set comprises applying an AdaBoost algorithm to produce the function to combine the online scores with the offline scores.

5. The at least one computer storage medium of claim 2 wherein combining the online recognition result set with the offline recognition result set comprises using artificial neural network-based combination to produce the function to combine the online scores with the offline scores.

6. The at least one computer storage medium of claim 1 wherein combining online recognition scores with offline recognition scores comprises, performing online recognition via a radical recognizer to obtain online recognition scores corresponding to radical ink data, building a radical graph representing candidates based on boundary data of the radical ink and the online radical scores, performing offline recognition via a radical recognizer to obtain offline recognition scores corresponding to the radical ink, combining the online radical recognition scores with the offline radical recognition scores to obtain a new combined radical score for each candidate of each radical ink data and updating the corresponding score in the radical graph based on the new combined radical score, and processing paths in the graph to determine the combined recognition result.

7. The at least one computer storage medium of claim 6 wherein the combined recognition result comprises a single result or multiple results, and wherein processing the paths in the graph to determine the combined recognition result comprises searching for an optimal path in the graph having a highest score or ranking by their path scores.

8. The at least one computer storage medium of claim 1 wherein combining the online recognition scores with offline recognition scores comprises, recognizing the handwritten radical data as online radical scores and offline radical scores.

9. In a computing environment, a system comprising an online recognizer implemented by a computer and configured to process an handwritten input to produce first data corresponding to online recognition results, and a combination mechanism implemented by the computer and configured to process the first data and second data to output a final recognition result set, wherein an offline recognizer processes the handwritten input to produce the second data, and wherein the second data corresponds to offline recognition results.

10. The system of claim 9 wherein the online recognizer produces the first data as a first set of character level scores, and wherein the offline recognizer produces the second data as a second set of character level scores.

11. The system of claim 10 wherein the combination mechanism determines a function that mathematically processes the scores to produce the final recognition result set.

12. The system of claim 11 wherein the combination mechanism comprises a statistical analysis-based process to determine the function that mathematically combines the scores.

13. The system of claim 11 wherein the scores correspond to features, and wherein the combination mechanism comprises an AdaBoost process to determine the function that mathematically combines the scores.

14. The system of claim 11 wherein the combination mechanism comprises an artificial neural network-based combination process.

15. The system of claim 14 wherein the artificial neural network-based combination process comprises a back propagation algorithm.

16. The system of claim 9 wherein the online recognizer produces the first data as radical level data used in constructing a radical graph, wherein the offline recognizer produces the second data as radical level data, and wherein the combination mechanism uses the second data to re-score the radical graph data in the graph, and further comprising a search mechanism that processes paths in the graph to produce the final recognition result set.

17. The system of claim 9 wherein the offline recognizer produces the second data via radical-based relation features, position features, or duration features, or any combination of radical-based relation features, position features, or duration features.

18. In a computing environment, a method comprising:

receiving, by a computer, handwritten input data;

performing online recognition of the handwritten input data to produce online recognition scores;

performing, by the computer, offline recognition of the handwritten input data to produce offline recognition scores; and

outputting a combined recognition result based on the online recognition scores and the offline recognition scores.

19. The method of claim 18 wherein the online and offline recognition scores comprise character level recognition scores associated with hypotheses, and wherein outputting the combined recognition result comprises combining online scores with offline scores using a combining function determined by a statistical analysis based process, an AdaBoost process, or an artificial neural network-based process.

20. The method of claim 18 wherein the online and offline recognition scores comprise radical level recognition scores, and further comprising, generating a radical graph from the online radical recognition scores, rescoring the radical graph using the offline radical recognition scores, and further comprising, searching paths in the radical graph to determine the combined recognition result.