Freeform digital ink annotation recognition

• US 20060050969A1
• Filed: 09/03/2004
• Published: 03/09/2006
• Est. Priority Date: 09/03/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Shilman, Michael, Bargeron, David M., Wei, Zile
• Granted Patent: US 7,574,048 B2
• Time in Patent Office: Days
• US Class Current: 382/224
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 40/103 Formatting, i.e. changing of presentation of documents automatic justification G06F40/189; automatic line break hyphenation G06F40/191
• CPC Class Codes: G06F 40/171 by use of digital ink
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/1444 Selective acquisition, locating or processing of specific regions, e.g. highlighted text, fiducial marks or predetermined fields

Abstract

The present invention leverages classification type detectors and/or context information to provide a systematic means to recognize and anchor annotation strokes, providing reflowable digital annotations. This allows annotations in digital documents to be archived, shared, searched, and easily manipulated. In one instance of the present invention, an annotation recognition method obtains an input of strokes that are grouped, classified, and anchored to underlying text and/or points in a document. Additional instances of the present invention utilize linguistic content, domain specific information, anchor context, and document context to facilitate in correctly recognizing an annotation.

Claims

1. A system that facilitates recognition, comprising:

a component that receives at least one input containing digital ink strokes; and

a recognition component that identifies and groups at least a subset of the digital ink strokes from the input into at least one recognized annotation.

2. The system of claim 1, the recognition component recognizes the annotation in real-time.

3. The system of claim 1, the recognition component employs background processes to obtain the recognized annotation.

4. The system of claim 1 further comprising a classification component that classifies the annotation into at least one type of a group of extensible types.

5. The system of claim 4, the classification component employs at least one type detector.

6. The system of claim 1 further comprising an anchoring component that anchors the annotation to an appropriate portion of the input.

7. The system of claim 6 further comprising a reflow component that employs annotation anchor points to automatically and logically reflow an annotation when the input is manipulated.

8. The system of claim 1, the input comprising at least one selected from the group consisting of digital information from a digital writing device and digital information from a scanned input.

9. The system of claim 1, the input further comprising a document context relating to the digital ink strokes.

10. The system of claim 9, the recognition component utilizes the document context to facilitate in recognizing at least one annotation.

11. The system of claim 9, the document context comprising a tree structure.

12. The system of claim 1, the recognition component further provides an annotation indicator for the recognized annotation to identify a recognized annotation location when the input is summarized.

13. The system of claim 1 further comprising a linguistic analysis component that employs a linguistic context of the input to facilitate the recognition component in proper recognition of at least one annotation.

14. The system of claim 1, the recognition component further employs domain specific information to facilitate in determining at least one recognized annotation.

15. The system of claim 14, the domain specific information comprising at least one selected from the group consisting of user specific information, professional genre information, and annotation guideline information.

16. The system of claim 1 further comprising an action determination component that determines at least one action indicated by the recognized annotation.

17. The system of claim 1 further comprising a resolution component that facilitates the recognition component by maximizing a number of explained strokes, maximizing an overall confidence, and minimizing a number of hypotheses for the input.

18. The system of claim 17, the resolution component optimizes recognition of an annotation via employment of an energy function given by:

$\begin{array}{cc}\begin{array}{c}E=<br><br>\underset{i}{\sum <br><br>}<br><br>{\mathrm{confidence}}_i+\mathrm{\alpha <br><br>}<br><br><br><br>\mathrm{explained}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{strokes}<br><br>-\\ <br><br>\mathrm{\beta <br><br>}<br><br><br><br>\mathrm{hypotheses}<br><br>;<br><br>\end{array}& \left(\mathrm{Eq}.\text{\hspace{1em}<br><br>}<br><br>1\right)\end{array}$

where α and

β

are empirically-determined weights.

19. The system of claim 18, the resolution component further employs dynamic program to facilitate in a resolution of the energy function.

51. A computer readable medium having stored thereon computer executable components of the system of claim 1.

53. A device employing the system of claim 1 comprising at least one selected from the group consisting of a computer, a server, and a handheld electronic device.

20. A method for facilitating recognition, comprising:

receiving at least one input containing digital ink strokes; and

identifying and grouping at least a subset of the digital ink strokes from the input into at least one recognized annotation.

21. The method of claim 20, the identifying and grouping of the subset of the digital ink strokes to form a recognized annotation occurring in real-time.

22. The method of claim 20 further comprising:

classifying the annotation into at least one type of a group of extensible types.

23. The method of claim 22, the group of extensible types including at least one selected from the group consisting of horizontal range type, vertical range type, container type, connector type, symbol type, writing type, and drawing type.

24. The method of claim 22 further comprising:

employing at least one type detector to facilitate in classifying the annotation.

25. The method of claim 24, the type detector comprising at least one selected from the group consisting of horizontal range type detector, vertical range type detector, container type detector, connector type detector, symbol type detector, writing type detector, and drawing type detector.

26. The method of claim 22 further comprising:

employing machine learning to facilitate in classifying the annotation.

27. The method of claim 20 further comprising:

anchoring the annotation to an appropriate portion of the input.

28. The method of claim 27 further comprising:

employing an annotation anchor point to automatically and logically reflow an annotation when the input is manipulated.

29. The method of claim 20, the input comprising at least one selected from the group consisting of digital information from a digital writing device and digital information from a scanned input.

30. The method of claim 20, the input comprising digital ink strokes obtained from a scanning process.

31. The method of claim 20, the input further comprising a document context relating to the digital ink strokes.

32. The method of claim 31 further comprising:

utilizing the document context to facilitate in recognizing at least one annotation.

33. The method of claim 31, the document context comprising a tree structure.

34. The method of claim 20 further comprising:

providing an annotation indicator for the recognized annotation to identify a recognized annotation location when the input is summarized.

35. The method of claim 20 further comprising:

employing a linguistic context of the input to facilitate in proper recognition of at least one annotation.

36. The method of claim 20 further comprising:

utilizing domain specific information to facilitate in determining at least one recognized annotation.

37. The method of claim 36, the domain specific information comprising at least one selected from the group consisting of architectural diagrams and geographic maps.

38. The method of claim 20 further comprising:

recognizing annotations on at least one media type selected from the group consisting of a typewritten text, a photographic image, a geometric vector graphics image, a digital ink handwriting, and a digital ink drawing.

39. The method of claim 20 further comprising:

determining at least one action indicated by the recognized annotation.

40. The method of claim 20 further comprising:

employing a plurality of candidate annotations to facilitate in recognizing an annotation.

41. The method of claim 20 further comprising:

utilizing a stylistic and/or non-stylistic characteristic of the digital ink strokes to facilitate in recognizing the annotation.

42. The method of claim 41, the stylistic characteristic comprising at least one selected from the group consisting of a color of a digital ink stroke, a thickness of a digital ink stroke, a pen tip style employed to create a digital ink stroke, a transparency level of a digital ink stroke, and a viscosity level of a digital ink stroke.

44. The method of claim 20 further comprising:

providing searchable recognized annotations based, at least in part, on digital ink strokes.

45. The method of claim 44 further comprising:

employing collaborative filtering techniques to facilitate in determining a desired search result.

46. The method of claim 20 further comprising:

determining an optimum resolution to facilitate in determining the recognized annotation via maximizing a number of explained strokes, maximizing an overall confidence, and minimizing a number of hypotheses for the input.

47. The method of claim 46 further comprising:

employing an energy function to facilitate in determining the optimum resolution, the energy function given by;

$\begin{array}{cc}\begin{array}{c}E=<br><br>\underset{i}{\sum <br><br>}<br><br>{\mathrm{confidence}}_i+\mathrm{\alpha <br><br>}<br><br><br><br>\mathrm{explained}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{strokes}<br><br>-\\ <br><br>\mathrm{\beta <br><br>}<br><br><br><br>\mathrm{hypotheses}<br><br>;<br><br>\end{array}& \left(\mathrm{Eq}.\text{\hspace{1em}<br><br>}<br><br>1\right)\end{array}$

where α and

β

are empirically-determined weights.

48. The method of claim 47 further comprising:

utilizing dynamic programming to facilitate in a resolution of the energy function.

52. A device employing the method of claim 20 comprising at least one selected from the group consisting of a computer, a server, and a handheld electronic device.

43. The method of 41, the non-stylistic characteristic comprising at least one selected from the group consisting of a timestamp on a digital ink stroke and a serial number on a pen tip cursor utilized to create a digital ink stroke.

49. A system that facilitates recognition, comprising:

means for receiving at least one input containing digital ink strokes; and

means for identifying and grouping at least a subset of the digital ink strokes from the input into at least one recognized annotation.

50. A data packet, transmitted between two or more computer components, that facilitates recognition, the data packet comprising, at least in part, information relating to an annotation recognition system that utilizes, at least in part, a digital ink stroke input to recognize at least one annotation.