Systems and methods for three dimensional geometric reconstruction of captured image data

• US 9,208,536 B2
• Filed: 09/19/2014
• Issued: 12/08/2015
• Est. Priority Date: 09/27/2013
• Status: Active Grant

• Current Assignee: Kofax Incorporated
• Original Assignee: Kofax Incorporated
• Inventors: Macciola, Anthony, Ma, Jiyong, Shustorovich, Alexander, Thrasher, Christopher W., Amtrup, Jan
• Primary Examiner(s): Patel, Jayesh A
• Application Number:
  US14/491,901
• Publication Number:
  US 20150093018A1
• Time in Patent Office: 445 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes

Abstract

In various embodiments, methods, systems, and computer program products for processing digital images captured by a mobile device are disclosed. Myriad features enable and/or facilitate processing of such digital images using a mobile device that would otherwise be technically impossible or impractical, and furthermore address unique challenges presented by images captured using a camera rather than a traditional flat-bed scanner, paper-feed scanner, or multifunction peripheral. Notably, the presently disclosed systems and techniques enable three-dimensional reconstruction of objects depicted in image captured using a camera of a mobile device. The reconstruction corrects or compensates for perspective distortion caused by camera-based capture.

Claims

1. A method of reconstructing a digital image, the method comprising:

receiving the digital image comprising a digital representation of an object bounded by a tetragon;

dividing the digital image into a plurality of digital image segments;

performing a segmentation analysis on the plurality of digital image segments;

correcting curvature in the tetragon to form a quadrilateral based at least in part on the segmentation analysis;

correcting projective effects in the quadrilateral to form a rectangle based at least in part on the segmentation analysis; and

estimating an absolute size of the object based on intrinsic parameters of a capture device used to capture the digital image.

2. The method as recited in claim 1, wherein correcting the curvature comprises transforming one or more portions of the tetragon based at least in part on intrinsic tetragon coordinates using a four-point algorithm.

3. The method as recited in claim 1, wherein the digital image is characterized by a capture angle of about 30 degrees or more deviation from normal with respect to the object.

4. The method as recited in claim 1, wherein correcting the projective effects comprises one or more of singular value decomposition (SVD) and direct linear transformation (DLT).

5. The method as recited in claim 1, further comprising estimating a two-dimensional pixel position for each of up to four pixels in the digital image, each of the up to four pixels corresponding to one of four corners of one or more of the object and the tetragon.

6. The method as recited in claim 5, wherein estimating the two-dimensions pixel position for each of the four pixels is based on a predetermined correspondence between each of the four pixels and four corresponding pixels of a reference image,

wherein the reference image depicts the digital representation of the object from a perspective corresponding to a capture angle normal to the object, and

wherein each of the four corresponding pixels of the reference image corresponds to one of four corners of the digital representation of the object in the reference image.

7. The method as recited in claim 1, further comprising estimating a two-dimensional pixel position for each of four corners of one or more of the object and the tetragon, and

wherein at least one of the four corners is characterized by being one or more of:

obstructed in the digital image;

not depicted in the digital image; and

blurred beyond recognition in the digital image.

8. The method as recited in claim 1,

wherein correcting the projective effects comprises estimating a homography transform H,

wherein estimating H comprises detecting one or more point correspondences p_i P_i′

with p_i=(x_i,y_i,1)^T,

wherein estimating H comprises detecting four point correspondences, each point correspondence p_iP_i′

corresponding to a corner position p_i of the tetragon and a respective corner position P′

_i of the rectangle, and

wherein H satisfies the expression λ

p_i=HP′

_i, where

λ

=a focal distance between a three-dimensional reference coordinate position (X, Y, Z) and a corresponding reference coordinate position of a capture device;

p_i=(x_i,y_i,1)^T;

P′

_I=(X_i,Y_i,1)^T, and

 

i is the respective index feature of the point p or P′

in the corresponding image.

9. The method as recited in claim 1, wherein correcting the projective effects comprises performing a non-linear refinement.

10. The method as recited in claim 1, further comprising outputting the reconstructed digital image, wherein the reconstructed digital image is characterized by a pixel location error of about 5 pixels or less relative to a scanned 500-DPI image of the object captured using a capture angle normal to the object.

11. The method as recited in claim 1, wherein at least one side of the tetragon is characterized by a second degree or higher order polynomial.

12. The method as recited in claim 1, wherein correcting the projective effects comprises normalizing coordinates of point-correspondences between the digital image and a reconstructed image.

13. The method as recited in claim 1, wherein the object is a document characterized by a known height-to-width ratio.

14. The method as recited in claim 1, wherein the estimating employs an intrinsic parameter matrix A representative of the intrinsic capture device parameters.

15. The method as recited in claim 14, wherein the intrinsic parameter matrix A is characterized by an expression

16. The method as recited in claim 15, wherein a=f/dx

wherein b=f/dy,

wherein f is the focal length,

wherein dx is a horizontal scaling factor,

wherein dy is a vertical scaling factor,

wherein c is a skew parameter, and

wherein (d, e) are the coordinates of a principal point in the digital image.

17. The method as recited in claim 15, wherein c =0, wherein either:

(d, e) are the coordinates of a principal point in the digital image, or

d=0 and e=0.

18. A system, comprising:

a processor configured to execute logic; and

logic configured to receive a digital image comprising a digital representation of an object bounded by a tetragon;

logic configured to correct curvature in the tetragon to form a quadrilateral; and

logic configured to correct projective effects in the quadrilateral to form a rectangle,

wherein correcting the projective effects is based at least in part on estimating corner positions of the object; and

logic configured to estimate an absolute size of the object based on intrinsic parameters of a capture device used to capture the digital image, and

wherein at least one corner of the object for which the corner position is estimated is characterized by being one or more of;

obstructed in the digital image;

not depicted in the digital image; and

blurred beyond recognition in the digital image.

19. A computer program product comprising a non-transitory computer readable storage medium having computer readable program code stored thereon, the computer readable program code comprising:

computer readable program code configured to receive a digital image comprising a digital representation of an object bounded by a tetragon;

computer readable program code configured to correct curvature in the tetragon to form a quadrilateral;

computer readable program code configured to correct projective effects in the quadrilateral to form a rectangle; and

computer readable program code configured to estimate an absolute size of the object based on intrinsic parameters of a capture device used to capture the digital image.