Stroke localization and binding to electronic document

• US 7,580,576 B2
• Filed: 06/02/2005
• Issued: 08/25/2009
• Est. Priority Date: 06/02/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wang, Qiang, Dang, Yingnong, Wu, Jiang, Wang, Jian, Chen, Liyong, Liu, Youjun
• Primary Examiner(s): Bali; Vikkram
• Assistant Examiner(s): Brooks; Julian D
• Application Number:
  US11/142,910
• Publication Number:
  US 20060274948A1
• Time in Patent Office: 1,545 Days
• Field of Search: 382/314, 382/315, 382/186, 382/181, 382/187, 382/188, 382/202, 382/203, 382/306, 345179-181
• US Class Current: 382/188
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry

Abstract

A method and apparatus for capturing information encoded within a surface, such as location information or document metadata, and associating the information with a document is described. The captured information may be obtained by a camera associated with a pointing or writing device having a camera, such as an image-capturing pen.

Claims

1. A process for associating an ink stroke with an electronic document comprising the steps of:

generating a first two-dimensional array of bits;

generating a second two-dimensional array of bits;

combining the first two-dimensional array and the second two-dimensional array to generate a combined array, wherein the first two-dimensional array and the second two-dimensional array at least partially overlap in the combined array and wherein a start position of the second two-dimensional array in the combined array is shifted, by an amount that is based on a document identifier associated with an electronic document, relative to a start position of the first two-dimensional array in the combined array;

printing the combined array on a paper document

receiving a first captured image of the paper document from a camera associated with a pen;

receiving a second captured image of the paper document from the camera associated with the pen;

obtaining encoded information from the first captured image and the second captured image;

decoding said information from the first captured image and the second captured image, wherein the decoded information includes location information and a document identifier, wherein location information is decoded at least in part by decoding a portion of the first two-dimensional array contained within an image and wherein a document identifier is decoded at least in part by determining the shift of a portion of the second two-dimensional array contained within an image relative to a portion of the first two-dimensional array contained within the image;

forming a representation of an ink stroke based on the decoded location information from the first captured image and the second captured image;

identifying an electronic version of the paper document based on the decoded document identifier from said decoding step;

storing the formed representation of the ink stroke in association with the identified electronic version of the paper document; and

displaying the electronic version of the paper document, including displaying the ink stroke based on the formed representation stored in association with the electronic version of the identified document.

2. The process according to claim 1, further comprising the step of:

pre-processing at least one captured image.

3. The process according to claim 1, further comprising the step of:

performing pattern analysis on at least one captured image.

4. The process according to claim 1, further comprising the step of:

recognizing one or more symbols from said obtaining encoded information step.

5. The process according to claim 1, wherein said forming step includes mapping said ink stroke to a document.

6. A system for associating pen movement with a document comprising:

an input that receives images of the document from a camera associated with a pen, wherein each image contains a combined array comprised of a first two-dimensional array of bits and a second two-dimensional array of bits, wherein a start position of the second two-dimensional array in the combined array is shifted, by an amount that is based on a document identifier associated with the document, relative to a start position of the first two-dimensional array in the combined array;

at least one processor that receives the images, the processor obtaining encoded information from the images, decoding the information to determine location information and a document identifier, and forming an ink stroke based on the identified location information, wherein location information is determined at least in part by decoding a portion of the first two-dimensional array contained within an image and wherein the document identifier is decoded at least in part by determining the shift of a portion of the second two-dimensional array contained within a first image relative to a portion of the first two-dimensional array contained within the first image; and

a component that stores the ink stroke in association with the document so that the associated ink stroke can be output along with the document.

7. The system according to claim 6, wherein the images are of a printed document and wherein the ink stroke is stored in association with an electronic version of the printed document.

8. The system according to claim 6, said at least one processor including a decoder that decodes the information.

9. The system according to claim 6, said at least one processor including an image preprocessor that adjusts lighting of at least one captured image.

10. The system according to claim 6, said at least one processor including a pattern analyzer that analyzes a pattern found in at least one image.

11. The system according to claim 6, said at least one processor including a symbol recognizer that recognizes a pattern in at least one image as a symbol that may be decoded.

12. The system according to claim 6, said at least one processor including a decoder that decodes a symbol found in at least one image.

13. The system according to claim 6, said at least one processor including a document mapper that maps a location determined from information in the images into a stroke that may be associated with said document.

14. The system according to claim 6, wherein the encoded information includes metadata.

15. A pen that captures images, said images being processed to associate a stroke with an electronic document, the pen comprising:

a component for capturing images of a surface containing an EIC symbol array and an output of the electronic document, the EIC symbol array including encoded position information and an encoded document identifier, wherein the EIC symbol array is comprised of a first two-dimensional array of bits and a second two-dimensional array of bits, wherein a start position of the second two-dimensional array is shifted, by an amount that is based on a document identifier associated with an electronic document, relative to a start position of the first two-dimensional array;

a component for decoding an EIC symbol within captured images to determine the position of the pen at the time an image was captured, wherein the position is determined at least in part by decoding the first two-dimensional array contained within an EIC symbol;

a component for generating an ink stroke based at least in part on the determined positions of captured images;

a component for decoding an EIC symbol within captured images to determine a document identifier, wherein the document identifier is decoded at least in part by determining the shift of the second two-dimensional array contained within an EIC symbol relative to the first two-dimensional array contained within the EIC symbol;

a component for identifying the electronic document based at least in part on the decoded document identifier; and

a component for binding the generated ink stroke to the electronic document so that the ink stroke can be identified when the electronic document is retrieved.

16. The pen of claim 15, wherein the document is an electronic version of a printed document and wherein the surface is the printed document.