Method and apparatus for capturing images of a document with interaction

• US 7,711,207 B2
• Filed: 07/28/2004
• Issued: 05/04/2010
• Est. Priority Date: 08/07/2003
• Status: Expired due to Fees

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Pilu, Maurizio, Pollard, Stephen Bernard
• Primary Examiner(s): Bali; Vikkram
• Assistant Examiner(s): Park; Soo Jin
• Application Number:
  US10/900,910
• Publication Number:
  US 20050078871A1
• Time in Patent Office: 2,106 Days
• Field of Search: 382200-230, 382/103, 382/293, 382/295
• US Class Current: 382/293
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: H04N 1/195 the array comprising a two-dimensional array or a combination of two-dimensional arrays
• CPC Class Codes: H04N 1/19594 using a television camera or a still video camera
• CPC Class Codes: H04N 2201/0416 Performing a pre-scan
• CPC Class Codes: H04N 2201/0436 Scanning a picture-bearing surface lying face up on a support H04N2201/0434 takes precedence; using cylindrical picture bearing surfaces H04N1/06; with manual scanning H04N1/107
• CPC Class Codes: H04N 2201/046 Actively compensating for disturbances, e.g. vibrations

Abstract

The user assistance system provides a system and method for capturing images of a document and interaction of a primary user with the document in an interaction session. Briefly described, one embodiment comprises an image capture means adapted to capture an initial image of the document and at least one subsequent, additional image of the document during an interaction session, the image being mapped to a known co-ordinate system, interaction capture means for capturing the interaction of a user with the document whereby to determine at least one co-ordinate of a pointer used for the interaction relative to the same co-ordinate system defined for the initial captured image, and processing means for determining an appropriate transform that maps the additional image onto the original image.

Claims

1. An apparatus for capturing images of a document and interaction of a primary user with the document in an interaction session, the apparatus comprising:

image capture means adapted to capture an initial image of the document at an initial position and at least one subsequent, additional image of the document during an interaction session, wherein the document in the additional image has been moved to a new position that is different than the initial position, the initial captured image being mapped to a known co-ordinate system;

interaction capture means for capturing the interaction of a user with the document, whereby the interaction capture means determines at least one co-ordinate of a pointer used for the interaction relative to the same co-ordinate system defined for the initial captured image;

processing means for determining an appropriate transform that maps the additional image onto the initial image; and

transmitting means for transmitting to a remote site the initial image and said at least one coordinate of the pointer, wherein the initial image having a pointer at said at least one coordinate of the pointer is displayable at the remote site, wherein the processing means is additionally adapted to apply the transform to the at least one captured co-ordinate of the pointer to produced at least one mapped pointer co-ordinate which is transmitted to the remote site.

2. Apparatus according to claim 1, wherein the interaction capture means determines a plurality of co-ordinates of the pointer.

3. Apparatus according to claim 1, in which more than one additional image is captured and in which image analysis means is provided arranged to analyse the additional image to determine if the document has moved within a field of view of the image capture device, a transform being produced for an additional image in the event that the document in the additional image has moved.

4. Apparatus according to claim 1, wherein the processing means is additionally adapted to apply the transform to co-ordinates of the additional image to transform the additional images to the same co-ordinates for the initial image and thereby produce mapped document co-ordinates.

5. Apparatus according to claim 1, wherein the processing means is adapted to generate a data set of information that includes the initial captured image and the mapped co-ordinates.

6. Apparatus according to claim 5, in which the generated data set includes only a single, initial, captured image of the document in addition to the mapped co-ordinates.

7. Apparatus according to claim 5, which includes storage means for storing the generated data set on a storage device.

8. Apparatus according to claim 5, wherein the transmission means is adapted to transmit the generated data across a network to a remote device.

9. Apparatus according to claim 1, which is arranged to capture the initial captured image at a higher resolution than the additional image.

10. Apparatus according to claim 1, in which the interaction capture means comprises a camera.

11. Apparatus according to claim 1, in which the interaction capture means comprises a graphics tablet on which the document is placed which is adapted to co\-operate with a pointer which is moved across the document and which produces a signal indicative of the position of the pointer relative to a set of co-ordinates.

12. Apparatus according to claim 1, in which the interaction capture means comprises a graphics tablet on which the document is placed which is adapted to co\-operate with mouse which is moved across the document and which produces a signal indicative of the position of the mouse relative to a set of co-ordinates.

13. A system for capturing images of a document and interaction of a primary user with the document in an interaction session and the display of the document and interaction at a remote location, the system comprising:

image capture means adapted to capture an initial high-resolution image of the document at an initial position and at least one subsequent, additional image of the document during an interaction session, wherein the document in the additional image has been moved to a new position that is different than the initial position;

interaction capture means for capturing the interaction of a user with the document to determine at least one co-ordinate of a pointer used for the interaction relative to a co-ordinate system of the image capture device;

processing means for determining an appropriate transform that maps the additional image onto the initial captured image;

data generating means for generating a data set indicative of the initial captured image and the mapped co-ordinates; and

transmission means adapted to transmit the generated data set across the network to a remote device, wherein the remote device comprises a composite image generating means which generates a plurality of composite images in which a pointer icon is superimposed on the initial captured image of the document which is sent across the network,

wherein the processing means further comprises means for applying the transform to the at least one captured co-ordinate of the pointer to produced at least one mapped co-ordinate which is transmitted to the remote device.

14. A system according to claim 13, wherein the interaction capture means determines a plurality of co-ordinates of the pointer.

15. A system according to claim 13, wherein the processing means further comprises means for applying the transform to co-ordinates of the additional images to produced mapped co-ordinates.

16. A system according to claim 13, in which more than one additional image is captured and in which image analysis means is provided adapted to analyse the additional images to determine if the document has moved within the field of view of the image capture device, a transform being produced for an additional image in the event that the document in the additional image has moved.

17. A system according to claim 16, in which the remote device comprises a display adapted to display the composite images.

18. The system according to claim 17, in which mapped pointer information is transmitted across the network in real time and a new composite image is produced and displayed each time a new set of pointer co-ordinates are received from the network.

19. The system according to claim 13, in which more than one remote device is connected to the network at any one time.

20. The system according to claim 13, in which the data set indicative of the captured image that is sent across the network comprises compressed data which encodes the captured image.

21. A method of capturing the interaction of a user with a document, comprising:

capturing an initial image of the document using an image capture device at an initial position and at least one subsequent, additional image of the document during an interaction session, wherein the document in the additional image has been moved to a new position that is different than the initial position, the initial captured image being mapped to a known co-ordinate system;

capturing interaction of a user with the document to determine at least one co-ordinate of a pointer used for the interaction relative to the same co-ordinate system of defined for the initial captured image;

capturing at least one additional image of the document;

analyzing the at least one additional image to determine if the document has moved within the field of view of the image capture device;

determining a transform mapping the additional image onto the initial image in the event that the document in the additional image is determined to have moved; and

a processor applying the transform to a captured co-ordinate of the pointer to produce at least one mapped pointer co-ordinate.

22. The method of claim 21, further comprising generating a data set of information indicative of the image and the mapped co-ordinate of the pointer.

23. A method according to claim 22, further comprising sending the data set of information across a network.

24. The method according to claim 23, wherein the additional images are captured at a lower resolution than the initial image.

25. A method according to claim 22, further comprising storing the information in memory.

26. A computer readable storage medium which stores a computer program which when running on a processing apparatus causes the processing apparatus to:

(a) capture an initial image of a document using an image capture device at an initial position and at least one subsequent, additional image of the document during an interaction session, wherein the document in the additional image has been moved to a new position that is different than the initial position, the initial captured image being mapped to a known co-ordinate system;

(b) capture the interaction of a user with the document to determine co-ordinates of a pointer used for the interaction relative to the same co-ordinate system defined for the initial captured image;

(c) capture at least one additional image of the document;

(d) analyse the additional image to determine if the document has moved within the field of view of the image capture device;

(e) determine an appropriate transform that maps the additional image onto the initial image in the event that the additional image has moved; and

(f) apply the transform to the captured co-ordinates of the pointer to produced mapped pointer co-ordinates.