Camera-based document scanning system using multiple-pass mosaicking

• US 20020186425A1
• Filed: 06/01/2001
• Published: 12/12/2002
• Est. Priority Date: 06/01/2001
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Kang, Sing Bing, Ulichney, Robert Alan, Dufaux, Frederic
• Granted Patent: US 7,046,401 B2
• Time in Patent Office: Days
• US Class Current: 358/497
• CPC Class Codes: G06T 3/147 using affine transformations
• CPC Class Codes: G06T 5/006 Geometric correction correction of chromatic aberrations adapted to be used in an image pickup device containing an electronic image sensor H04N23/10; detecting, correcting, reducing or removing artefacts resulting only from the lens unit, e.g. flare, shading, vignetting or "cos4" H04N25/61
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/207 for motion estimation over a hierarchy of resolutions multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06V 10/10 Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: H04N 1/195 the array comprising a two-dimensional array or a combination of two-dimensional arrays
• CPC Class Codes: H04N 1/19594 using a television camera or a still video camera
• CPC Class Codes: H04N 1/3876 Recombination of partial images to recreate the original image
• CPC Class Codes: H04N 2201/0414 Scanning an image in a series of overlapping zones
• CPC Class Codes: H04N 2201/0436 Scanning a picture-bearing surface lying face up on a support H04N2201/0434 takes precedence; using cylindrical picture bearing surfaces H04N1/06; with manual scanning H04N1/107

Abstract

A camera-based document scanning system produces electronic versions of documents, based on a plurality of images of discrete portions of the documents. The system compares each pair of consecutive images and derives motion parameters that indicate the relative motion between each pair of consecutive images. The system utilizes the derived motion parameters to align and merge each image with respect to the previous images, thereby building a single, mosaic image of the document. In the illustrative embodiment, the motion parameters are derived by minimizing a sum of squared differences equation on a pixel-by-pixel basis.

Claims

1. A method for generating an electronic version of a document, the method comprising the steps of:

receiving a plurality of digital, electronic images of the document;

generating a corrected image from each received image;

deriving one or more motion parameters for each pair of consecutive, corrected images, the motion parameters indicating the relative motion between the consecutive, corrected images;

aligning each image relative to the previous images based on the derived motion parameters; and

blending each image into the previous images so as to produce the electronic version of the document.

2. The method of claim 1 wherein the digital, electronic images are produced by a digital video camera.

3. The method of claim 1 wherein two or more series of digital, electronic images of the document are received, whereby each series of images corresponds to a respective sweep of the document by the video camera, the method further comprising the steps of:

merging the images from each series together to form a composite, mosaic image of the respective sweeps, and

merging consecutive mosaic sweep images together to form the electronic version of the document.

4. The method of claim 3 wherein the step of deriving the one or more motion parameters comprises the step of minimizing a sum of squares differences between each pair of consecutive images.

5. The method of claim 4 wherein

in the corrected image frames include a plurality of pixels, and

the sum of squares differences is applied on a pixel-by-pixel basis.

6. The method of claim 5 wherein the sum of squares differences is substantially given by the following equation:

7. The method of claim 6 wherein the motion model has eight motion parameters, m₀, m₁, m₂, m₃, m₄, m₅, m₆ and m₇ defined by:

i′

=(m₀+m₂i+m₃j)/(m₆i+m₇j+1)j′

=(m₁+m₄i+m₅j)/(m₆i+m₇j+1)

8. The method of claim 7 further comprising the step of subsampling the received images, and wherein it is assumed that:

m₂=m₅=1, andm₃=m₄=m₆=m₇=0.

9. The method of claim 8 wherein the step of subsampling the received images comprises the step of discarding one out of two pixels in both the horizontal and vertical directions from each received image.

10. The method of claim 7 wherein it is assumed that:

m₅=m₂m₄=−

m₃, andm₆=m₇=0.

11. The method of claim 7 wherein

the received image frames have YUV color space components, where Y corresponds to luminance and U and V correspond to chrominance,

the motion parameters are derived only for the Y component of the corrected images, and

the derived motion parameters for the Y component are scaled for the U and V components.

12. The method of claim 6 further comprising the step of performing a spline-based registration on consecutive mosaic sweep images.

13. The method of claim 2 wherein the step of generating a corrected image comprises the steps of:

building at least one look-up table (LUT) having, for each pixel of the received image frames, a corresponding entry containing a correction factor; and

applying the corresponding correction factors to the pixels of the received image frames to produce the corrected images.

14. The method of claim 9 wherein the correction factors stored at the at least one LUT correct for off-axis illumination and radial lens distortion.

16. The system of claim 15 wherein the digital electronic images of the document are received from a digital video camera.

17. The system of claim 16 wherein the at least one motion estimation engine is configured to derive the one or more motion parameters by minimizing a sum of squares differences between each pair of consecutive images.

18. The system of claim 17 further comprising at least one look-up table (LUT) containing, for each pixel of the received images, a corresponding entry containing a correction factor, and wherein

the image correction engine utilizes the correction factors stored at the at least one LUT to produce the corrected images.

19. The system of claim 18 wherein

the digital video camera is mounted to a stand, and

the correction factors stored at the at least one LUT correct for off-axis illumination and radial lens distortion of the video camera, and for tilt of the video camera relative to the stand.

20. The system of claim 19 wherein the at least one motion estimation engine includes an image pyramid having a plurality of levels, each level of the image pyramid configured to perform an iterative gradient descent operation and a convergence operation on consecutive images to produce the one or more motion parameters, the motion parameters from a given level being used as a starting point for the iterative gradient descent and convergence operations of the next lower level of the pyramid.

15. A system for generating an electronic version of a document, the system comprising:

an image correction engine configured to receive a plurality of digital, electronic images of the document and to generate a corrected image from each received image;

at least one motion estimation engine configured to compare consecutive, corrected images and to derive for each pair of consecutive corrected images one or more motion parameters defining the relative motion between the respective images; and

at least one alignment and blending image configured to use the derived motion parameters to align and blend consecutive images to produce the electronic version of the document.

21. A computer system for use in scanning a document, the computer system comprising:

a base;

a display panel pivotally attached to the base;

a digital camera mounted to the display panel;

an image correction engine configured to receive a plurality of images of the document from the digital camera, and further configured to generate a corrected image from each received image;

at least one motion estimation engine configured to compare consecutive, corrected images and to derive for each pair of consecutive corrected images one or more motion parameters defining the relative motion between the respective images; and

at least one alignment and blending image configured to use the derived motion parameters to align and blend consecutive images to produce a scanned image of the entire document.

22. The computer system of claim 21 wherein the at least one motion estimation engine is configured to derive the one or more motion parameters by minimizing a sum of squares differences between each pair of consecutive images.

23. The computer system of claim 22 further comprising at least one look-up table (LUT) containing, for each pixel of the received images, a corresponding entry containing a correction factor, and wherein

the image correction engine utilizes the correction factors stored at the at least one LUT to produce the corrected images.

24. The computer system of claim 23 wherein the at least one motion estimation engine includes an image pyramid having a plurality of levels, each level of the image pyramid configured to perform an iterative gradient descent operation and a convergence operation on consecutive images to produce the one or more motion parameters, the motion parameters from a given level being used as a starting point for the iterative gradient descent and convergence operations of the next lower level of the pyramid.