Architecture for ink annotations on web documents

• US 7,284,192 B2
• Filed: 06/24/2004
• Issued: 10/16/2007
• Est. Priority Date: 06/24/2004
• Status: Active Grant

• Current Assignee: Arlington Technologies, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Avaya Technology Corporation Miami Lakes FLA US
• Inventors: Ramachandran, Sriram, Kashi, Ramanujan S.
• Primary Examiner(s): HUYNH, CONG LAC T
• Application Number:
  US10/877,004
• Publication Number:
  US 20050289452A1
• Time in Patent Office: 1,209 Days
• Field of Search: 715/512, 715/541, 715/500
• US Class Current: 715/232
• CPC Class Codes: G06F 40/143 Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/171 by use of digital ink

Abstract

Digital ink annotations of digital documents are captured, processed, and stored for later retrieval. Elements within the digital documents are found and data concerning them are stored, in order that they may be associated with the digital ink annotation and serve as anchors for the digital ink annotations within the digital documents being annotated. Furthermore, a markup language schema is used for the efficient, standardized storage of the digital ink annotations. By these means, digital ink annotations are stored efficiently, retrieved, and rendered accurately across different types of browsers and operating systems, as well as across different size browser windows and output devices.

Claims

1. A method for annotating a markup language document with a visual representation of a digital ink annotation and storing said digital ink annotation, comprising the steps of:

acquiring digital ink by capturing input from a user who is annotating a markup language document;

rendering the digital ink and the markup language document in a browser window, wherein said rendered digital ink forms a digital ink annotation of said markup language document;

finding a longest allowable point-to-point segment by applying a test criterion to two non-adjoining digital ink points in a line of the digital ink annotation to compress a number of digital ink points comprising the digital ink annotation, wherein, if the test criterion is satisfied, the digital ink points between the two non-adjoining digital ink points are deleted and replaced with a straight-line segment between the two non-adjoining digital ink points;

searching for at least one element or a portion of at least one element which is unique on the markup language document in at least one characteristic to associate at least one element in the markup language document with the digital ink annotation; and

storing the digital ink annotation wherein the associated at least one element acts as an anchor in the markup language document for the digital ink annotation.

2. The method of claim 1, wherein a document object model is used in the step of associating at least one element of the markup language document with the digital ink annotation of the markup language document.

3. The method of claim 2, wherein the document object model comprises a W3C DOM standard.

4. The method of claim 1, wherein the at least one element comprises one of an image element and a text element.

5. The method of claim 1, wherein the at least one characteristic is textual.

6. The method of claim 5, wherein being unique in at least one textual characteristic comprises at least one of:

(i) being a unique word in the markup language document; and

(ii) being a unique character string in the markup language document.

7. The method of claim 1, further comprising the steps of:

determining at least one of a shape and location of the digital ink annotation; and

associating at least one element in the markup language document with the digital ink annotation based at least partly on the determined at least one of shape and location.

8. The method of claim 7, wherein the step of associating at least one element in the markup language document with the digital ink annotation based at least partly on the determined at least one of shape and location comprises the step of:

classifying an annotation type of the digital ink annotation based at least partly on the determined at least one of shape and location of the digital ink annotation.

9. The method of claim 7, wherein the step of associating at least one element in the markup language document with the digital ink annotation based at least partly on the determined at least one of shape and location comprises the step of:

analyzing at least one element in the markup language document which is near the determined location of the digital ink annotation.

10. The method of claim 1, further comprising the step of:

analyzing at least one element in the markup language document in a manner appropriate to a type of the at least one element being analyzed.

11. The method of claim 1, further comprising the steps of:

obtaining a reference position of at least one element in the markup language document; and

determining a relative reference position of the digital ink annotation in relation to the obtained reference position.

12. The method of claim 11, wherein the step of storing the digital ink annotation comprises the step of:

storing the obtained reference position of the at least one element; and

storing the determined relative reference position of the digital ink annotation in relation to the obtained reference position of the at least one element.

13. The method of claim 1, wherein the step of storing the digital ink annotation comprises the step of:

using a markup language schema to create a digital ink annotation file in markup language.

14. The method of claim 13, wherein a markup language tag in the markup language schema comprises at least one of:

a tag for a type of digital ink annotation;

a tag for a type of the markup language document; and

a tag for a location of the digital ink annotation.

15. The method of claim 13, wherein a markup language tag in the markup language schema comprises a tag for a location of the associated at least one element on the markup language document.

16. The method of claim 13, wherein a markup language tag in the markup language schema comprises a tag for a location of the digital ink annotation relative to the associated at least one element.

17. The method of claim 13, wherein a markup language tag in the markup language schema comprises a tag for a type of the associated at least one element.

18. The method of claim 13, wherein a markup language tag in the markup language schema comprises at least one of:

a tag for ink points comprising the digital ink annotation, a tag for identifying the user making the digital ink annotation, a tag for identifying the markup language document, and a tag for indicating a spatial size of the markup language document.

19. The method of claim 13, further comprising the steps of:

retrieving the digital ink annotation file in markup language from storage;

rendering the retrieved digital ink annotation and the markup language document;

wherein the retrieved digital ink annotation file is used to appropriately render the digital ink annotation in relation to the markup language document.

20. The method of claim 1, further comprising the steps of:

retrieving the digital ink annotation from storage; and

rendering the retrieved digital ink annotation and the markup language document;

wherein the retrieved digital ink annotation is rendered by using the associated at least one element as an anchor in the markup language document.

21. The method of claim 1, wherein the step of storing the digital ink annotation comprises the step of:

storing a portion of the markup language document associated with the digital ink annotation.

22. The method of claim 1, wherein the digital ink annotation is handwritten by the user using an input device.

23. The method of claim 1, wherein the step of finding a longest allowable point-to-point segment comprises the step of:

(a) applying the test criterion to an initial digital point and a test digital point located at least one digital point away from the initial digital point;

(b) if the test criterion is satisfied, making a next digital point in the line the test digital point and returning to step (a).

24. The method of claim 23, wherein the step of finding a longest allowable point-to-point segment further comprises the step of:

(c) if the test criterion is not satisfied, taking a penultimate digital point immediately before the test digital point, deleting the digital points between the initial digital point and the penultimate point, and replacing them with a straight line segment between the initial digital point and the penultimate point.

25. The method of claim 24, wherein the test criterion is whether a sum of at least one deviation distance from each digital point between the initial digital point and the test digital point to a straight line segment between the initial digital point and the test digital point is less than a maximum allowable area deviation per length unit.

26. The method of claim 1, wherein the method is implemented on at least one of a desktop computer, a laptop computer, a Personal Digital Assistant (PDA) or other hand-held computer, a cellular telephone, and any device capable of running a markup language document browser.

27. A method for annotating an HTML or XML document with a visual representation of a digital ink annotation and storing said digital ink annotation using a XML annotation schema, comprising the steps of:

acquiring digital ink by capturing input from a user who is annotating an HTML/XML document;

rendering the digital ink and the HTML/XML document in a browser window, wherein said rendered digital ink forms a digital ink annotation of said HTML/XML document;

finding a longest allowable point-to-point segment by applying a test criterion to two non-adjoining digital ink points in a line of the digital ink annotation to compress a number of digital ink points comprising the digital ink annotation, wherein, if the test criterion is satisfied, the digital ink points between the two non-adjoining digital ink points are deleted and replaced with a straight-line segment between the two non-adjoining digital ink points;

using a run-time document object model (DOM) of the HTML/XML document to find at least one HTML/XML element or portion of at least one HTML/XML element in the HTML/XML document related to the digital ink annotation by uniqueness in at least one characteristic; and

creating a digital ink association HTML/XML file using a XML annotation schema, wherein at least one found HTML/XML element acts as an anchor in the HTML/XML document for the digital ink annotation.

28. The method of claim 27, wherein the DOM comprises a W3C DOM standard.

29. The method of claim 27, wherein the uniqueness in at least one characteristic comprises location in relation to the digital ink annotation, and wherein the step of using the run-time DOM to find at least one HTML/XML element or portion of at least one HTML/XML element related to the digital ink annotation by said uniqueness in at least one characteristic comprises the steps of:

determining a location of the digital ink annotation; and

querying the run-time DOM of the HTML/XML document for HTML/XML elements near the determined location of the digital ink annotation.

30. The method of claim 27, wherein the step of creating a digital ink association HTML/XML file using a XML annotation schema, wherein at least one found HTML/XML clement acts as an anchor in the HTML/XML document for the digital ink annotation, comprises the step of:

storing data comprising a location of the digital ink annotation relative to the at least one found HTML/XML element.

31. The method of claim 27, wherein the step of using a run-time document object model (DOM) of the HTML/XML document to find at least one HTML/XML clement or portion of at least one HTML/XML element in the HTML/XML document related to the digital ink annotation by uniqueness in at least one characteristic comprises the step of:

inserting programming code into the run-time DOM of the HTML/XML document, wherein said inserted programming code parses the run-time DOM.

32. The method of claim 27, wherein the digital ink annotation is handwritten by the user using an input device.

33. The method of claim 27, wherein a markup language tag in the XML annotation schema comprises at least one of:

a tag for a type of digital ink annotation;

a tag for a type of the markup language document; and

a tag for a location of the digital ink annotation.

34. The method of claim 27, wherein a markup language tag in the XML annotation schema comprises a tag for a location of the associated at least one element on the markup language document.

35. The method of claim 27, wherein a markup language tag in the XML annotation schema comprises a tag for a location of the digital ink annotation relative to the associated at least one element.

36. The method of claim 27, wherein a markup language tag in the XML annotation schema comprises a tag for a type of the associated at least one element.

37. The method of claim 27, wherein a markup language tag in the XML annotation schema comprises at least one of:

a tag for ink points comprising the digital ink annotation, a tag for identifying the user making the digital ink annotation, a tag for identifying the markup language document, and a tag for indicating a spatial size of the markup language document.

38. The method of claim 27, wherein the step of finding a longest allowable point-to-point segment comprises the step of:

(a) applying the test criterion to an initial digital point and a test digital point located at least one digital point away from the initial digital point;

(b) if the test criterion is satisfied, making a next digital point in the line the test digital point and returning to step (a).

39. The method of claim 38, wherein the step of finding a longest allowable point-to-point segment further comprises the step of:

(c) if the test criterion is not satisfied, taking a penultimate digital point immediately before the test digital point, deleting the digital points between the initial digital point and the penultimate point, and replacing them with a straight line segment between the initial digital point and the penultimate point.

40. The method of claim 39, wherein the test criterion is whether a sum of at least one deviation distance from each digital point between the initial digital point and the test digital point to a straight line segment between the initial digital point and the test digital point is less than a maximum allowable area deviation per length unit.

41. The method of claim 27, wherein the method is implemented on at least one of a desktop computer, a laptop computer, a Personal Digital Assistant (PDA) or other hand-held computer, a cellular telephone, and any device capable of running a markup language document browser.

42. The method of claim 27, wherein the at least one HTML/XML element comprises one of an image element and a text element.

43. A method for retrieving and rendering a stored visual representation of a digital ink annotation on a markup language document, comprising the step steps of:

finding a longest allowable point-to-point segment by applying a test criterion to two non-adjoining digital ink points in a line of the digital ink annotation to compress a number of digital ink points comprising the digital ink annotation. wherein, if the test criterion is satisfied, the digital ink points between the two non-adjoining digital ink points are deleted and replaced with a straight-line segment between the two non-adjoining digital ink points; and

rendering the stored digital ink annotation and at least a portion of a markup language document which was annotated by the stored digital ink annotation in a browser window, wherein the stored digital ink annotation comprises data identifying at least one element in the markup language document and data concerning a location of the digital ink annotation relative to the at least one element which is unique on the markup language document in at least one characteristic;

wherein the at least one element acts as an anchor in the markup language document for the rendering of the digital ink annotation.

44. The method of claim 43, wherein a document object model was used to associate said at least one element of the markup language document with the digital ink annotation of the markup language document.

45. The method of claim 44, wherein the document object model comprises a W3C DOM standard.

46. The method of claim 43, wherein the stored digital ink annotation was created and stored using a markup language schema.

47. The method of claim 46, wherein a markup language tag in the markup language schema comprises at least one of:

a tag for a type of digital ink annotation, a tag for a type of the markup language document, and a tag for a location of the digital ink annotation.

48. The method of claim 46, wherein said data identifying at least one element in the markup language document comprises a markup language tag in the markup language schema for a location of the at least one element on the markup language document.

49. The method of claim 46, wherein said data concerning the location of the digital ink annotation relative to the at least one element comprises a markup language tag in the markup language schema for a location of the digital ink annotation relative to the at least one element.

50. The method of claim 46, wherein said data identifying at least one element in the markup language document comprises a markup language tag in the markup language schema for a type of the at least one element.

51. The method of claim 46, wherein a markup language tag in the markup language schema comprises at least one of:

a tag for ink points comprising the digital ink annotation, a tag for identifying the user making the digital ink annotation, a tag for identifying the markup language document, and a tag for indicating a spatial size of the markup language document.

52. The method of claim 43, wherein the markup language comprises at least one of XML and HTML.

53. The method of claim 43, wherein the method is implemented on at least one of a desktop computer, a laptop computer, a Personal Digital Assistant (PDA) or other hand-held computer, a cellular telephone, and any device capable of running a markup language document browser.

54. The method of claim 43, wherein said at least one element comprises one of an image element and a text element.