Electronic input device, system, and method using human-comprehensible content to automatically correlate an annotation of a paper document with a digital version of the document

• US 7,546,524 B1
• Filed: 03/30/2005
• Issued: 06/09/2009
• Est. Priority Date: 03/30/2005
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Kessel, Steven, Bryar, Colin M., Siegel, Hilliard B.
• Primary Examiner(s): RIES, LAURIE ANNE
• Application Number:
  US11/095,763
• Time in Patent Office: 1,532 Days
• Field of Search: 715/229, 715/230, 715/268
• US Class Current: 715/230
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes

Abstract

An electronic input device such as an electronic pen is provided to annotate a paper document. The input device records an annotation and an image of human-comprehensible content in the document sufficient to identify the document and possibly a location in the document. The human-comprehensible content is used to locate a digital version of the document and determine a corresponding location of the annotation in the digital version of the document. A computer system such as a server system may receive and store the annotation in association with the digital version of the document. The server system may further augment the digital version of the document with the annotation and send the augmented version to an output device for display and/or printing.

Claims

1. An electronic input device for use in annotating a document, comprising:

(a) an imaging component for obtaining an image of human-comprehensible content in a paper version of a document sufficient to identify a location in the document;

(b) a recording component for recording an annotation to be associated with the identified location in the document;

(c) a recognition component configured to analyze the annotation and determine if the annotation includes an electronically-executable instruction; and

(d) a data communication component configured to communicate data from the electronic input device to a computer system.

2. The electronic input device of claim 1, wherein the recording component is a writing tip of the electronic input device.

3. The electronic input device of claim 2, wherein the writing tip includes a sensor for recording movements of the writing tip.

4. The electronic input device of claim 3, wherein the sensor records movements of the writing tip in a two-dimensional X-Y plane.

5. The electronic input device of claim 2, further comprising an ink deposit connected to the writing tip.

6. The electronic input device of claim 5, further comprising a switch for enabling or disabling the flow of ink from the ink deposit through the writing tip.

7. The electronic input device of claim 1, wherein the recording component includes an audio device that is capable of capturing an audible annotation.

8. The electronic input device of claim 1, wherein the imaging component includes a camera.

9. The electronic input device of claim 1, wherein the imaging component records a spatial relationship among the human-comprehensible content in the document.

10. The electronic input device of claim 1, further comprising a processor for processing the human-comprehensible content in the image obtained by the imaging component.

11. The electronic input device of claim 1, wherein the electronically-executable instruction starts with one or more predefined characters or words that are recognized as indicating the executable instruction.

12. A computing system for correlating an annotation of a paper version of a document with a digital version of the document, comprising:

(a) an interface for communicating with one or more input devices;

(b) a storage containing digital versions of documents; and

(c) a processor in communication with the interface and the storage, wherein the processor is configured to;

(i) receive through the interface, from an input device, an image of human-comprehensible content in a document, wherein the human-comprehensible content is sufficient to identify a digital version of the document;

(ii) use the image of human-comprehensible content to locate in the storage a digital version of the document;

(iii) receive through the interface, from the input device, an annotation of a paper version of the document;

(iv) apply a recognition routine to the annotation and determine if the annotation includes an electronically-executable instruction; and

(v) store the annotation in association with the digital version of the document.

13. The computer system of claim 12, wherein the processor is configured to locate a digital version of the document using a digital signature derived from the image of human-comprehensible content.

14. The computer system of claim 12, wherein the processor is configured to locate a digital version of the document using a hash key generated from the image of human-comprehensible content.

15. The computer system of claim 12, wherein the processor is configured to locate a digital version of the document using the human-comprehensible content in the image.

16. The computer system of claim 12, wherein the processor is configured to locate a digital version of the document using a spatial relationship among the human-comprehensible content in the image.

17. The computing system of claim 16, wherein the processor is further configured to use the spatial relationship to identify the location of the annotation in the document.

18. The computing system of claim 12, wherein the processor is configured to store the annotation as any one of a text file, an audio file, or an image file.

19. The computing system of claim 12, wherein the processor is further configured to execute the electronically-executable instruction.

20. The computing system of claim 12, wherein the processor is further configured to augment the digital version of the document with the annotation.

21. The computing system of claim 20, wherein the processor is configured to cause the annotation to appear in the digital version of the document at approximately the same location as the annotation appears on the paper version of the document.

22. The computing system of claim 20, wherein the processor is configured to cause the annotation to appear as a footnote to the digital version of the document with a footnote indicator appearing at approximately the same location as the annotation appears on the paper version of the document.

23. The computing system of claim 20, wherein the processor is further configured to print the augmented digital version of the document.

24. The computer system of claim 12, wherein the electronically-executable instruction starts with one or more predefined characters or words that are recognized as indicating the executable instruction.

25. A computer-implemented method for processing an annotation of a document, comprising:

(a) receiving an image of human-comprehensible content in a paper version of a document, wherein the human-comprehensible content is sufficient to identify a digital version of the document;

(b) processing the human-comprehensible content to locate a digital version of the document;

(c) receiving an annotation of the paper version of the document;

(d) determining if the annotation includes an electronically-executable instruction; and

(e) storing the annotation in association with the digital version of the document.

26. The computer-implemented method of claim 25, wherein receiving the annotation includes receiving an image of the annotation as it appears on the paper version of the document.

27. The computer-implemented method of claim 25, wherein processing the human-comprehensible content produces a digital signature that is usable to locate the digital version of the document.

28. The computer-implemented method of claim 25, wherein processing the human-comprehensible content produces a hash key that is usable to locate the digital version of the document.

29. The computer-implemented method of claim 25, wherein processing the human-comprehensible content uses the human-comprehensible content itself to locate the digital version of the document.

30. The computer-implemented method of claim 25, further comprising determining a spatial relationship among the human-comprehensible content and using the spatial relationship to locate the digital version of the document.

31. The computer-implemented method of claim 30, further comprising using the spatial relationship to identify the location of the annotation in the document.

32. The computer-implemented method of claim 25, wherein storing the annotation includes storing the annotation as any one of a text file, an audio file, or an image file.

33. The computer-implemented method of claim 25, further comprising executing the electronically-executable instruction.

34. The computer-implemented method of claim 25, further comprising augmenting the digital version of the document with the annotation.

35. The computer-implemented method of claim 34, wherein augmenting the digital version of the document with the annotation includes causing the annotation to appear in the digital version of the document at approximately the same location as the annotation appears on the paper version of the document.

36. The computer-implemented method of claim 34, wherein augmenting the digital version of the document with the annotation includes causing the annotation to appear as a footnote in the digital version of the document with a footnote indicator appearing at approximately the same location as the annotation appears on the paper version of the document.

37. The computer-implemented method of claim 34, further comprising printing the augmented digital version of the document.

38. The computer-implemented method of claim 25, wherein the electronically-executable instruction starts with one or more predefined characters or words that are recognized as indicating the executable instruction.

39. A computer-readable medium containing executable instructions that, when executed by a computing apparatus, cause the computing apparatus to:

(a) receive an image of human-comprehensible content in a paper version of a document, wherein the human-comprehensible content is sufficient to locate a digital version of the document;

(b) locate a digital version of the document using the image of human-comprehensible content;

(c) receive an annotation of the paper version of the document;

(d) apply a recognition routine to the annotation to determine if the annotation includes an electronically-executable instruction; and

(e) store the annotation in association with the digital version of the document.

40. The computer-readable medium of claim 39, further comprising computer-executable instructions that, when executed by the computing apparatus, cause the computing apparatus to use a spatial relationship among the human-comprehensible content to identify the location of the annotation in the document.

41. The computer-readable medium of claim 39, wherein the computer-executable instructions further cause the computing apparatus to execute the electronically-executable instruction determined to be in the annotation.

42. The computer-readable medium of claim 39, wherein the computer-executable instructions cause the computing apparatus to display the annotation in the digital version of the document at approximately the same location as the annotation appears on the paper version of the document.

43. The computer-readable medium of claim 39, wherein the computer-executable instructions cause the computing apparatus to display the annotation as a footnote in the digital version of the document with a footnote indicator appearing at approximately the same location as the annotation appears on the paper version of the document.

44. The computer-readable medium of claim 39, wherein the computer-executable instructions further cause the computing apparatus to print the digital version of the document with the annotation.

45. The computer-readable medium of claim 39, wherein the electronically-executable instruction within the annotation starts with one or more predefined characters or words that are recognized as indicating the executable instruction.

46. A computer-implemented method for processing an annotation of a document, comprising:

(a) recording an annotation of a paper version of a document;

(b) obtaining an image of human-comprehensible content in the document, wherein the human-comprehensible content is sufficient to identify a digital version of the document and further identify a location of the annotation in the document;

(c) locating a digital version of the document using the image of human-comprehensible content;

(d) identifying the location of the annotation in the document using the image of human-comprehensible content;

(e) determining if the annotation includes an electronically-executable instruction; and

(f) storing the annotation and location in association with the digital version of the document.

47. The computer-implemented method of claim 46, wherein the image of human-comprehensible content is obtained at approximately the same time the annotation is recorded.

48. The computer-implemented method of claim 46, further comprising, at the location of the annotation in the document, augmenting the digital version of the document with the stored annotation.

49. The computer-implemented method of claim 48, wherein augmenting the digital version of the document with the annotation includes causing the annotation to appear in the digital version of the document at approximately the same location as the annotation appears on the paper version of the document.

50. The computer-implemented method of claim 48, wherein augmenting the digital version of the document with the annotation includes causing the annotation to appear as a footnote in the digital version of the document with a footnote indicator appearing at approximately the same location as the annotation appears on the paper version of the document.

51. The computer-implemented method of claim 48, further comprising printing the augmented digital version of the document.

52. The computer-implemented method of claim 46, wherein the electronically-executable instruction starts with one or more predefined characters or words that are recognized as indicating the executable instruction.