Automatic rectification of distortions in images

• US 9,064,309 B2
• Filed: 10/29/2013
• Issued: 06/23/2015
• Est. Priority Date: 10/29/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Chaudhury, Krishnendu, DiVerdi, Stephen J.
• Primary Examiner(s): Tabatabai, Abolfazl
• Application Number:
  US14/066,633
• Publication Number:
  US 20150117787A1
• Time in Patent Office: 602 Days
• Field of Search: 382/199, 382/254, 382/276, 382/289, 382/290, 382/309, 382/310, 348/746, 348/806
• US Class Current: 1/1
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Abstract

Implementations relate to relate to rectification of distortion in an image. In some implementations, a method includes extracting edgelets from an image, each edgelet defined by a location of a pixel having an edge depicted in the image and defined by a direction of the edge. The method finds at least one vanishing point in the image based on intersections of pairs of the edges of the edgelets. Each vanishing point is selected based on a plurality of the edgelets in the image having edges approximately aligned with the vanishing point. Found vanishing points are used to transform the image to reduce image distortion.

Claims

1. A method comprising:

extracting a plurality of edgelets from an image, each edgelet defined by a location of a pixel having an edge depicted in the image and each edgelet defined by a direction of the edge;

finding at least one vanishing point in the image based on intersections of pairs of the edges of the edgelets, wherein the at least one vanishing point is selected based on a plurality of the edgelets in the image having edges approximately aligned with the at least one vanishing point; and

using the at least one vanishing point to transform the image to reduce distortion in the image.

2. The method of claim 1 wherein each edgelet is associated with an edge line extending through the location in the direction of the edgelet, and wherein finding at least one vanishing point in the image is based on intersections of pairs of the edge lines of the edgelets, wherein the at least one vanishing point is selected based on a plurality of the edgelets in the image having edge lines approximately aligned with the at least one vanishing point.

3. The method of claim 2 wherein finding each of the at least one vanishing point is based on a weighted majority of the edgelets in the image having edge lines at least approximately aligned with the at least one vanishing point.

4. The method of claim 3 wherein finding each of the at least one vanishing point includes checking a plurality of sets of randomly-selected edgelets and candidate vanishing points formed by the sets of randomly-selected edgelets for the weighted majority of the aligned edge lines.

5. The method of claim 2 wherein finding each of the at least one vanishing point includes:

randomly selecting a pair of the edgelets and finding a candidate vanishing point at the intersection of the edge lines of the selected pair of edgelets;

determining a contribution to a score for the pair of edgelets from each of the other edgelets based on an alignment of the edge lines of the other edgelets to the candidate vanishing point;

repeating the randomly selecting and determining a contribution for a plurality of other pairs of the edgelets and associated vanishing points; and

selecting the candidate vanishing point having the highest score.

6. The method of claim 5 wherein each edgelet also is defined by a strength of the edge, wherein the strength is indicative of a magnitude of a discontinuity in pixel values between the pixel and at least one neighboring pixel, and

wherein the contribution to the score for the pair of edgelets from each of the other edgelets is also based on a strength of each of the other edgelets.

7. The method of claim 5 wherein finding each of the at least one vanishing point includes re-estimating the found vanishing point more accurately using an estimation of a best intersection point of edgelets that contributed to the score of the found vanishing point.

8. The method of claim 1 wherein the direction of the edgelet is a direction indicated by examining a neighborhood of pixels surrounding the pixel having the edge.

9. The method of claim 1 wherein each edgelet also is defined by a strength of the edge, wherein the strength is indicative of a magnitude of a discontinuity in pixel values between the pixel and at least one neighboring pixel.

10. The method of claim 9 wherein the strength of the edgelet is based on a magnitude of an eigenvalue of a matrix based on the pixel and the direction of the edgelet is based on an eigenvector corresponding to the eigenvalue.

11. The method of claim 1 wherein extracting a plurality of edgelets includes using a corner detector to detect pixels having edges and pixels having corners, wherein each of the pixels having an edge is associated with a different edgelet and the pixels having corners are excluded from association with edgelets.

12. The method of claim 1 wherein transforming the image includes warping the image to cause parallel linear features in a scene captured in the image to become more parallel to each other as depicted in the image, wherein the warping of the image is performed using an affine rectification of the image using only two found vanishing points.

13. The method of claim 12 wherein the transformation of the image includes an alignment transformation using a rotation matrix to adjust an orientation of the image.

14. The method of claim 13 wherein the alignment transformation includes aligning one of the found vanishing points with an axis of the image.

15. The method of claim 1 wherein the transformation of the image includes performing a transformation that is a product of:

an affine rectification to cause parallel linear features in a scene captured in the image to become more parallel to each other as depicted in the image; and

an alignment transformation to align one or more depicted image features with a horizontal or vertical axis of the image.

16. The method of claim 1 further comprising performing at least one correctness check on the transformed image to determine whether the transformation of the image is to be discarded, wherein the correctness check is based on at least one of:

image size distortion, face distortion, and horizon distortion within the transformed image.

17. A method comprising:

detecting a plurality of edges in an image;

finding two vanishing points in the image based on intersections of pairs of the edges, wherein the two vanishing points are selected based on a plurality of the edges approximately aligned with the two vanishing points; and

using the two vanishing points and no additional vanishing points to transform the image to reduce distortion in the image, wherein the transformation includes;

an affine rectification to cause parallel linear features in a scene captured in the image to become more parallel to each other as depicted in the image; and

an alignment transformation to align an orientation of one or more depicted image features with a horizontal or vertical axis of the image.

18. The method of claim 17 wherein detecting the plurality of edges includes extracting a plurality of edgelets from an image, each edgelet defined by a location of a pixel having an edge depicted in the image and defined by a direction, wherein the direction of the edgelet is a direction indicated by examining a neighborhood of pixels surrounding the pixel having the edge, and wherein each edgelet is associated with an edge line extending through the location in the direction of the edgelet.

19. A system comprising:

a storage device; and

at least one processor accessing the storage device and operative to perform operations comprising;

extracting a plurality of edgelets from an image, each edgelet defined by a location of a pixel having an edge depicted in the image and each edgelet defined by a direction, wherein each edgelet is associated with an edge line extending through the location in the direction of the edgelet;

finding at least one vanishing point in the image based on intersections of pairs of the edge lines of the edgelets, wherein the at least one vanishing point is selected based on a plurality of the edgelets in the image having edge lines approximately aligned with the at least one vanishing point; and

using the at least one vanishing point to transform the image to reduce distortion in the image.

20. The system of claim 19 wherein finding each of the at least one vanishing point includes:

randomly selecting a pair of the edgelets and finding a candidate vanishing point at the intersection of the edge lines of the selected pair of edgelets;

determining a contribution to a score for the pair of edgelets from each of the other edgelets based on an alignment of the edge lines of the other edgelets to the candidate vanishing point; and

repeating the randomly selecting and determining a contribution for a plurality of other pairs of the edgelets and selecting a vanishing point having the highest score.