Camera-based document scanning system using multiple-pass mosaicking

• US 7,046,401 B2
• Filed: 06/01/2001
• Issued: 05/16/2006
• Est. Priority Date: 06/01/2001
• Status: Expired due to Term

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Kang, Sing Bing, Ulichney, Robert Alan, Dufaux, Frederic
• Primary Examiner(s): Coles, Edward
• Assistant Examiner(s): Safaipour, Houshang
• Application Number:
  US09/872,725
• Publication Number:
  US 20020186425A1
• Time in Patent Office: 1,810 Days
• Field of Search: 358/473, 358/478, 358/450, 358/284, 358/241, 358/474, 382/305, 382/275, 382/284, 382/107, 382/241, 348/241
• US Class Current: 358/450
• CPC Class Codes: G06T 3/147 using affine transformations
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/207 for motion estimation over a hierarchy of resolutions multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06V 10/10 Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: H04N 1/195 the array comprising a two-dimensional array or a combination of two-dimensional arrays
• CPC Class Codes: H04N 1/19594 using a television camera or a still video camera
• CPC Class Codes: H04N 1/3876 Recombination of partial images to recreate the original image
• CPC Class Codes: H04N 2201/0414 Scanning an image in a series of overlapping zones
• CPC Class Codes: H04N 2201/0436 Scanning a picture-bearing surface lying face up on a support H04N2201/0434 takes precedence; using cylindrical picture bearing surfaces H04N1/06; with manual scanning H04N1/107

Abstract

A camera-based document scanning system produces electronic versions of documents, based on a plurality of images of discrete portions of the documents. The system compares each pair of consecutive images and derives motion parameters that indicate the relative motion between each pair of consecutive images. The system utilizes the derived motion parameters to align and merge each image with respect to the previous images, thereby building a single, mosaic image of the document. In the illustrative embodiment, the motion parameters are derived by minimizing a sum of squared differences equation on a pixel-by-pixel basis.

Claims

1. A method for generating an electronic version of a document, the method comprising:

receiving a plurality of digital, electronic images of the document;

generating a corrected image from each received image;

deriving one or more motion parameters for each pair of consecutive, corrected images, the motion parameters indicating the relative motion between the consecutive, corrected images, the motion parameters are derived by minimizing the sum of squares differences between each pair of consecutive images;

aligning each image relative to the previous images based on the derived motion parameters; and

blending each image into the previous images so as to produce the electronic version of the document;

wherein the sum of squares differences is substantially given by the following equation;

$E=\underset{i,j}{\sum <br><br>}<br><br>e^2<br><br>\left(i,j\right)$

wherein, e(i, j)=I(i, j)−

I′

(i′

, j′

); and

wherein (i, j) and (i′

, j′

) are corresponding pixel locations in a previous image and a current image, respectfully.

2. The method of claim 1 wherein the digital, electronic images are produced by a digital video camera.

3. The method of claim 1 wherein two or more series of digital, electronic images of the document are received, whereby each series of images corresponds to a respective sweep of the document by the video camera, the method further comprising:

merging the images from each series together to form a composite, mosaic image of the respective sweeps; and

merging consecutive mosaic sweep images together to form the electronic version of the document.

4. The method of claim 1 wherein:

the corrected image frames include a plurality of pixels; and

the sum of squares differences is applied on a pixel-by-pixel basis.

5. The method of claim 1 wherein the motion model has eight motion parameters, m₀, m₁, m₂, m₃, m₄, m₅, m₆ and m₇ defined by:

i′

=(m₀+m₂i+m₃j)/(m₆i+m₇j+1); and

j′

=(m₁+m₄i+m₅j)/(m₆i+m₇j+1).

6. The method of claim 5 further comprising subsampling the received images, wherein:

m₂=m₅=1; and

m₃=m₄=m₆=m₇=0.

7. The method of claim 6 wherein subsampling the received images comprises discarding one out of two pixels in both the horizontal and vertical directions from each received image.

8. The method of claim 5 wherein:

m₅=m₂;

m₄=m₃; and

m₆=m₇=0.

9. The method of claim 5 wherein:

the received image frames have YUV color space components, wherein Y corresponds to luminance and U and V correspond to chrominance;

the motion parameters are derived only for the Y component of the corrected images; and

the derived motion parameters for the Y component are scaled for U and V components.

10. The method of claim 1 further comprising performing a spline-based registration on consecutive mosaic sweep images.

11. The method of claim 2 wherein the generating a corrected image comprises:

building at least one look-up table having, for each pixel of the received image frames, a corresponding entry containing a correction factor; and

applying the corresponding correction factors to the pixels of the received image frames to produce the corrected images.

12. The method of claim 7 wherein the correction factors stored at the at least one look-up table correct for off-axis illumination and radial lens distortion.

13. A method for generating an electronic version of a document, the method comprising:

receiving a plurality of digital, electronic images of the document, said electronic images being produced by a digital video camera;

generating a corrected image from each received image, said generating comprising building at least one look-up table having, for each pixel of the received image frames, a corresponding entry containing a correction factor; and

applying the corresponding correction factors to the pixels of the received image frames to produce the corrected images;

deriving one or more motion parameters for each pair of consecutive, corrected images, the motion parameters indicating the relative motion between the consecutive, corrected images;

aligning each image relative to the previous images based on the derived motion parameters; and

blending each image into the previous images so as to produce the electronic version of the document.

14. The method of claim 13 wherein two or more series of digital, electronic images of the document are received, whereby each series of images corresponds to a respective sweep of the document by the video camera, the method further comprising:

merging the images from each series together to form a composite, mosaic image of the respective sweeps; and

merging consecutive mosaic sweep images together to form the electronic version of the document.

15. The method of claim 14 wherein the step of deriving the one or more motion parameters comprises minimizing a sum of squares differences between each pair of consecutive images.

16. The method of claim 15 wherein the corrected image frames include a plurality of pixels, and the sum of squares differences is applied on a pixel-by-pixel basis.

17. The method of claim 16 wherein the sum of squares differences is substantially given by the following equation:

$E=\underset{i,j}{\sum <br><br>}<br><br>e^2<br><br>\left(i,j\right)$

wherein, e(i, j)=I(i, j)−

I′

(i′

, j′

); and

wherein (i, j) and (i′

, j′

) are corresponding pixel locations in a previous image and a current image, respectfully.

18. The method of claim 17 wherein the motion model has eight motion parameters, m₀, m₁, m₂, m₃, m₄, m₅, m₆ and m₇ defined by:

i′

=(m₀+m₂i+m₃j)/(m₆i+m₇j+1); and

j′

=(m₁+m₄i+m₅j)/(m₆+m₇j+1).

19. The method of claim 17 further comprising subsampling the received images wherein m₁ and m₅ are equal to one and m₃, m₄, m₆, and m₇ are equal to zero.

20. The method of claim 19 wherein the step of subsampling the received images comprised the step of discarding one out of two pixels in both the horizontal and vertical directions from each received image.

21. The method of claim 20 wherein the correction factors stored at the at least one look-up table correct for off-axis illumination and radial lens distortion.

22. The method of claim 18 wherein:

m₂ equals m₅;

m₃ equals −

m₄; and

m₆ and m₇ are equal to zero.

23. The method of claim 18 wherein:

the received image frames have YUV color space components, where Y corresponds to luminance and U and V correspond to chrominance;

the motion parameters and derived only for the Y component of the corrected images; and

the derived motion parameters for the Y component are scaled for U and V components.

24. The method of claim 17 further comprising performing a spline-based registration on consecutive mosaic sweep images.