Digital ink annotation process and system for recognizing, anchoring and reflowing digital ink annotations

• US 20040252888A1
• Filed: 06/13/2003
• Published: 12/16/2004
• Est. Priority Date: 06/13/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Shilman, Michael, Bargeron, David M., Wei, Zile, Moscovich, Tomer
• Granted Patent: US 7,218,783 B2
• Time in Patent Office: Days
• US Class Current: 382/188
• CPC Class Codes: G06F 40/171 by use of digital ink
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32

Abstract

A digital ink annotation process and system for processing digital documents and digital ink annotations therein. The process and system maintain an annotation'"'"'s position within a document such that the original intent and meaning of the annotation is preserved. This is true even if the document is edited, resized, displayed on a different device or otherwise modified. The digital ink annotation process includes automatic and manual grouping of digital ink strokes within a document to define digital ink annotations, classifying the annotations according to annotation type, and anchoring the annotations to appropriate regions or positions in a document. The process further includes reflowing the annotations in a new document layout such that the annotations conform and adapt to the new layout while preserving the original intents and meanings of the annotations. A digital ink annotation system includes a classification module, an anchoring module, a reflow module and a clean-up module to implement the digital ink annotation process.

Claims

1. A method for processing a digital document containing a digital ink annotation, comprising:

classifying the digital ink annotation to generate an annotation classification;

anchoring the digital ink annotation to a region in the digital document with which the digital ink annotation is associated to generate an annotation anchor; and

reflowing the digital ink annotation in a new layout of the digital document.

2. The method as set forth in claim 1, wherein the digital ink annotation is comprised of digital ink strokes and further comprising:

grouping the digital ink strokes created by a user together to define the digital ink annotation; and

classifying the digital ink annotation based on the grouped ink strokes.

3. The method as set forth in claim 1, further comprising classifying the digital ink annotation based on at least two different criteria.

4. The method as set forth in claim 3, wherein the at least two criteria include:

(a) annotation characteristics; and

(b) contextual information in the digital document.

5. The method as set forth in claim 3, further comprising defining a confidence estimate of the annotation classification that determines whether the user should be consulted.

6. The method as set forth in claim 5, further comprising determining that the confidence estimate is low for the annotation classification.

7. The method as set forth in claim 5, further comprising determining that the confidence estimate for a grouping of digital ink strokes falls below a threshold and that manual grouping should be used.

8. The method as set forth in claim 1, further comprising defining anchoring rules for each classification of digital ink annotations.

9. The method as set forth in claim 1, further comprising applying anchoring rules for each classification of digital ink annotation to generate the annotation anchor.

10. The method as set forth in claim 1, further comprising determining the validity of the annotation anchor based on the presence or absence of the anchor context for the digital ink annotation in the digital document.

11. The method as set forth in claim 10, further comprising determining whether a user interaction is desired based on the validity of the annotation anchor.

12. The method as set forth in claim 1, further comprising processing the digital ink annotation to conform to the new layout such that the meaning and intent of the original digital ink annotation is preserved.

13. The method as set forth in claim 12, further comprising rendering the digital ink annotation on a display device to conform to the new layout.

14. A computer-readable medium having computer-executable instructions for performing the method recited in claim 1.

15. A method for processing digital ink strokes made in a digital document, comprising:

grouping the digital ink strokes to define a digital ink annotation;

classifying the digital ink annotation based on annotation characteristics and contextual information in the digital document; and

reflowing the digital ink annotation a modified version of the digital document having a new layout such that an intent and meaning of the digital ink strokes are preserved.

16. The method as set forth in claim 15, further comprising grouping the digital ink strokes based on a temporal ordering of the digital ink strokes.

17. The method as set forth in claim 15, further comprising grouping the digital ink strokes based on a spatial ordering of the digital ink strokes.

18. The process as set forth in claim 15, further comprising grouping the digital ink strokes based on a combination of:

(a) a temporal ordering of the digital ink strokes; and

(b) a spatial ordering of the digital ink strokes.

20. One or more computer-readable media having computer-readable instructions thereon which, when executed by one or more processors, cause the one or more processors to implement the method of claim 15.

21. A computer-readable medium having computer-executable instructions for adapting a digital ink annotation in a digital document to a new layout in a modified digital document, comprising:

grouping digital ink strokes within the digital document to define the digital ink annotation;

anchoring the digital ink annotation based on the annotation classification to generate a digital document with properly flowed annotations;

subsequently modifying the digital document with properly flowed annotations by an external modification process to generate the modified digital document having the new layout that is different from the digital document with properly flowed annotations; and

reflowing the digital annotations to conform to the new layout of the modified digital document.

22. The computer-readable medium of claim 21, further comprising grouping the digital ink strokes based on at least one of:

(a) a temporal ordering of the digital ink strokes;

(b) a spatial ordering of the digital ink strokes.

23. The computer-readable medium of claim 21, wherein the digital ink annotations include active reading annotations that are made by a user to emphasize certain content, make comments, or record thoughts and impressions within the digital document.

24. The computer-readable medium of claim 21, wherein the digital ink annotation includes editing annotations that are used to indicate errors in the digital document.

25. A process for reflowing digital ink strokes from an original digital document having a first layout in a modified digital document having a second layout that is different from the first layout, comprising:

grouping the digital ink strokes to define a digital ink annotation;

classifying the digital ink annotation based on annotation characteristics and contextual information within the original digital document;

anchoring the digital ink annotation to a region in the original digital document associated with the digital ink annotation;

determining the second layout of the modified digital document;

processing the digital ink annotation to conform to the second layout; and

rendering the digital ink annotation in the modified digital document such that the digital ink annotation conforms to the second layout and retains a same meaning and intent in the modified digital document as was present in the original digital document.

26. The process as set forth in claim 25, further comprising classifying the digital ink annotation as an underline annotation.

27. The process as set forth in claim 26, further comprising anchoring the underline annotation to the characters in the region that are underlined.

28. The process as set forth in claim 25, further comprising classifying the digital ink annotation as a highlight annotation.

29. The process as set forth in claim 25, further comprising classifying the digital ink annotation as a marginalia annotation.

30. The process as set forth in claim 29, wherein the marginalia annotation is defined as any digital ink annotation stroke that is in a margin or white space of the digital document.

31. The process as set forth in claim 25, further comprising classifying the digital ink annotation as a circle annotation.

32. The process as set forth in claim 31, wherein the circle annotation is at least one of:

(a) an inline circle;

(b) a region circle.

33. The process as set forth in claim 32, wherein the inline circle circumscribes a single line of text.

34. The process as set forth in claim 32, wherein the region circle circumscribes multiple partial line of text and other digital document content.

35. The process as set forth in claim 25, further comprising classifying the digital ink annotation as a margin bracket annotation.

36. The process as set forth in claim 35, wherein the margin bracket annotation is defined as groups of ink strokes that form approximately vertical lines a margin of the digital document.

37. The process as set forth in claim 25, further comprising classifying the digital ink annotation as a callout annotation.

38. The process as set forth in claim 37, wherein the callout annotation is defined as an annotation that groups links regions by of the digital document by going from one position to another position.

39. The process as set forth in claim 37, wherein the callout annotation exhibits some curvature.

40. The process as set forth in claim 37, wherein the callout annotation is at an angle to horizontal and to vertical directions in the digital document.

41. The process as set forth in claim 37, wherein ends of the callout annotation are adjacent to at least one of:

(a) text of the digital document;

(b) content of the digital document;

(c) digital ink strokes from other annotations.