Interactive copying system

• US 5,511,148 A
• Filed: 04/29/1994
• Issued: 04/23/1996
• Est. Priority Date: 04/30/1993
• Status: Expired due to Term

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Wellner, Pierre D.
• Primary Examiner(s): Bayerl, Raymond J.
• Assistant Examiner(s): Garcia, Gabriel I.
• Application Number:
  US08/236,758
• Time in Patent Office: 725 Days
• Field of Search: 395/106, 395/144, 395/145, 395/146, 395/147, 345/173, 345/156, 358/478, 358/350, 358/537
• US Class Current: 358/1.6
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: H04N 1/00347 with another still picture apparatus, e.g. hybrid still picture apparatus circuits or arrangements for control or supervision between image input and image output device H04N1/32
• CPC Class Codes: H04N 1/00381 Input by recognition or interpretation of visible user gestures manual input means, e.g. digitisers, writing tablets H04N1/00392; gesture input to computers G06F3/017; recognition algorithms G06V40/20
• CPC Class Codes: H04N 2201/0082 Image hardcopy reproducer H04N2201/0091 - H04N2201/0094 take precedence
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 2201/0089 Image display device
• CPC Class Codes: H04N 2201/0436 Scanning a picture-bearing surface lying face up on a support H04N2201/0434 takes precedence; using cylindrical picture bearing surfaces H04N1/06; with manual scanning H04N1/107

Abstract

A system for generating new documents (20) from originals (4) containing text and/or images (22,28) employing e.g. a camera-projector system (6,8) focussed on a work surface (2), in conjunction with a copier or printer (14). In use, an original paper document (4) lying on the surface (2) becomes part of the interface to the copier machine as various interactions are carried out on the text or images (22,28). Feedback to the user is provided by projection of an image (21,24) onto the surface or onto the original, or using some other visual display. Text or images (22,28) may be selected and manipulated by the user, e.g. by pointing with fingers and tapping on the surface.

Claims

1. A copying system, comprising:

a work surface;

means for displaying images on the work surface;

a camera, focussed on the work surface, for generating video signals representing in electronic form image information present within the field of view of the camera;

processing means for recognizing one or more manual operations relating to the image information which are executed by a user within the field of view of the camera, and for performing electronic operations, corresponding to said manual operations, on the electronic form to produce a modified electronic form;

the displaying means being adapted to display, under the control of the processing means, simultaneously with or subsequent to said electronic operations, images defined by said manual operations; and

wherein said images defined by said manual operations include an image of a newly created document.

2. The copying system according to claim 1, wherein the modified electronic form includes an electronic version of the newly created document.

3. The copying system according to claim 2, further including means for printing a document corresponding to at least part of said modified electronic form.

4. The copying system according to claim 1, further including means, for sensing vibrational signals on the surface;

the processing means being adapted to recognize a tap or strike by a user on the surface.

5. The copying system according to claim 1, wherein the processing means includes a frame grabber for storing video frames and differencing means for establishing the difference between pixel data values of corresponding pixels in successive video frames, for obtaining resultant video frame data showing the difference, and for displaying the resultant video frame data.

6. The copying system according to claim 5, wherein the processing means includes thresholding means, for converting multi-bit per pixel video frame data to 1 bit per pixel video frame data.

7. The copying system according to claim 6, wherein the thresholding means is adapted for carrying out said converting operation based on an estimate equal to a moving average of pixel intensities in a local area.

8. The copying system according to claim 7, wherein the local area comprises 1/nth of the width of a video frame, where n is about 8.

9. The copying system according to claim 1, wherein the processing means includes a frame grabber, for storing video frames, and means for calibrating positions in the frame grabber relative to positions within the display.

10. The copying system according to claim 9, wherein the calibrating means includes means for projecting a mark at four points in the display and carrying out said calibration by means of a four point mapping, given by

11. The copying system according to claim 1, wherein the processing means includes means for determining whether the user is right- or left-handed.

12. An interactive image reproduction system, comprising:

two or more workstations interconnected by a communications link, each workstation comprising a system according to claim 1, each workstation being adapted for displaying the video output from the camera of each other workstation.

13. A system according to claim 12, further including an audio or videoconferencing link between the workstations.

14. A method of generating documents, comprising:

providing a work surface, means for displaying images on the work surface, and a camera focussed on the work surface, said camera generating video signals representing in electronic form image information present within the field of view of the camera;

recognizing one or more manual operations relating to the image information which are executed by a user within the field of view of the camera;

performing electronic operations, corresponding to said manual operations, on the electronic form to produce a modified electronic form;

displaying, simultaneously with or subsequent to the act of performing electronic operations, images defined by said manual operations; and

wherein said images defined by said manual operations include an image of a newly created document.

15. The method according to claim 14, wherein the modified electronic form includes an electronic version of the newly created document.

16. The method according to claim 14, further comprising:

supplying to a printing device said electronic version; and

printing out said newly created document.

17. The method according to claim 16, wherein said manual operations include selecting a portion of text or image information in a document located within the field of view of the camera.

18. The method according to claim 17, wherein said manual operations include designating two or more extremities of a shape encompassing said selected portion of text or image information.

19. The method according to claim 18, wherein the images defined by said manual operations include an outline of, or a shaded area coincident with, said shape.

20. The method according to claim 18, wherein said shape is a rectangle.

21. The method according to claim 18, wherein said manual operations include pointing with two or more fingers at corners of said shape.

22. The method according to claim 18, wherein said extremities are designated using a stylus in association with a position sensing tablet on the surface.

23. The method according to claim 17, wherein said manual operations include designating a text or image unit in the document by pointing a finger at it.

24. The method according to claim 23, wherein said manual operations include designating a successively larger text or image unit in the document by tapping on the surface.

25. The method according to claim 17, wherein said manual operations include confirming a text or image selection by tapping on the surface.

26. The method according to claim 17, wherein said manual operations include copying the selected text or image to a location in a new document displayed on the surface by pointing at the selected text or image using a finger or stylus and dragging the finger or stylus across the surface to said location in the new document, and dropping the selected text or image at said location by tapping on the surface.

27. The method according to claim 17, wherein said manual operations include changing the dimensions of selected text or image by changing the separation of finger tips of the user defining extremities of the selected text or image.

28. The method according to claim 17, wherein said manual operations include placing paper signs within the field of view of the camera, the signs defining operations to be performed on selected text or image information.

29. A copying system, comprising:

a work surface;

means for displaying images on the work surface;

a camera, focussed on the work surface, for generating video signals representing in electronic form image information present in at least first and second documents within the field of view of the camera;

processing means for recognizing one or more manual operations which are executed by a user within the field of view of the camera and represent the transfer of image information from the first document to the second document, and for performing electronic operations, corresponding to said manual operations, on the electronic form of said second document to produce a modified electronic form;

the displaying means being adapted to display, under the control of the processing means, simultaneously with or subsequent to said electronic operation performing step, images defined by said manual operations.

30. The copying system of claim 29, further including means for scanning the second document to generate an electronic version of the second document;

wherein the processing means includes means for recognizing the positions of the transferred image information in said electronic version;

the system further including means for printing said transferred image information on said second document.

31. A product comprising:

image input circuitry for receiving signals from an image input device;

image output circuitry for providing data defining output images to an image output device; and

processing circuitry connected for receiving signals from the image input circuitry and for providing data to the image output circuitry;

the processing circuitry being operable to;

obtain first output image data defining a first output image that shows a document;

provide the first output image data to the image output circuitry to cause the image output device to present the first output image on a work surface;

a working area being on or adjacent the work surface;

the working area being an area in which a user can perform manual operations;

receive an input image signal from the image input device through the image input circuitry, the input image signal defining an input image set that includes at least one input image, each input image in the set showing the work surface and the working area;

the input image set showing a first manual operation being performed in the working area;

the first manual operation indicating image-related information;

use the input image signal to obtain manual operation data;

the manual operation data indicating the image-related information;

use the manual operation data to obtain second output image data defining a version of the first output image that shows a modified version of the document; and

provide the second output image data to the image output circuitry to cause the image output device to present the version of the first output image on the work surface.

32. The product of claim 31, further comprising a printer, the processor being connected for providing data defining images to the printer.

33. The product of claim 31, further comprising a scanner;

the processor being connected for receiving data defining images from the scanner.

34. A method comprising:

obtaining first output image data defining a first output image that shows a document;

providing the first output image data to cause an image output device to present the first output image on a work surface;

a working area being on or adjacent the work surface;

the working area being an area in which a user can perform manual operations;

receiving an input image signal from an image input device, the input image signal defining an input image set that includes at least one input image, each input image in the set showing the work surface and the working area;

the input image set showing a first manual operation being performed in the working area;

the first manual operation indicating image-related information;

using the input image signal to obtain manual operation data;

the manual operation data indicating the image-related information;

using the manual operation data to automatically obtain second output image data defining a version of the first output image that shows a modified version of the document; and

providing the second output image data to cause the image output device to present the version of the first output image on the work surface.

35. The method of claim 34 in which a paper document is on the work surface;

the first manual operation indicating a part of the paper document;

the act of using the manual operation data to automatically obtain second output image data comprising;

obtaining part image data defining an image showing the indicated part of the paper document; and

using the part image data to obtain the second output image data so that the version of the first output image shows a copy of the indicated part of the paper document.

36. The method of claim 34, further comprising:

receiving an input sound signal indicating an acoustic effect of a second manual operation in the working area;

the act of using the input image signal to obtain manual operation data being performed in response to the input sound signal.

37. A product comprising:

image input circuitry for receiving signals from a camera that is positioned to capture images of a work surface and a working area on or adjacent the work surface;

image output circuitry for providing data defining output images to a display device that is positioned to present images on the work surface;

the working area being an area in which a user can perform manual operations indicating operations that would modify images presented on the work surface by the display device; and

processing circuitry connected for receiving signals from the image input circuitry and for providing data to the image output circuitry;

the processing circuitry being operable to;

obtain first output image data defining a first output image that shows a document;

provide the first output image data to the image output circuitry to cause the display device to present the first output image on the work surface;

receive an input image signal from the camera through the image input circuitry, the input image signal defining an input image set that includes at least one input image, each input image in the set showing the work surface and the working area;

the input image set showing a first manual operation being performed in the working area;

the first manual operation indicating a first document modifying operation that would modify the document;

use the input image signal to obtain modification data;

the modification data indicating the first document modifying operation;

the act of using the input image signal comprising;

performing image processing on the input image signal to recognize the first manual operation; and

obtaining the modification data based on recognizing the first manual operation;

using the modification data to obtain second output image data defining a version of the first output image;

the version of the first output image showing a modified version of the document in which the first document modifying operation has been performed; and

provide the second output image data to the image output circuitry to cause the display device to present the version of the first output image on the work surface.

38. The product of claim 37 in which the image output circuitry is for providing data defining output images to a display device that is a projector.