Rectifying camera pairs

• US 9,503,677 B1
• Filed: 11/29/2012
• Issued: 11/22/2016
• Est. Priority Date: 11/29/2012
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Ramaswamy, Sharadh, Ivanchenko, Volodymyr V.
• Primary Examiner(s): Tran, Thai
• Assistant Examiner(s): Braniff, Christopher T
• Application Number:
  US13/689,126
• Time in Patent Office: 1,454 Days
• US Class Current: 1/1
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 7/00 Television systems details H04N3/00, H04N5/00; methods or arrangements, for coding, decoding, compressing or decompressing digital video signals H04N19/00; selective content distribution H04N21/00

Abstract

The quality of stereoscopic imaging using cameras with wide angle lenses can be improved using various calibration approaches to determine distortions or misalignments between the cameras. A calibration object such as a checkered grid with a determined inclination between portions can be used to provide lateral calibration as well as depth information. Intersections of the checkered regions can be used to determine intersection points for the calibration object in the image. Regions about at least a portion of these points can be analyzed, where the regions are sized such that features of the calibration object are substantially linear. Nearest neighbor points can be identified, and a local orientation determined in order to determine a relationship of each of the nearest points. Once identified, the points in each image can be correlated to corresponding points of the calibration object.

Claims

1. A computer-implemented method, comprising:

capturing a first image including a first representation of a calibration object using a first camera;

capturing a second image including a second representation of the calibration object using a second camera separated a distance from the first camera, the calibration object including a pattern of alternating regions;

locating respective points in each of the first image and the second image corresponding to intersections of the alternating regions of the pattern;

for each point of at least a portion of the respective points in each of the first image and the second image;

analyzing a region including the point to identify nearest neighboring points, a size of the region being determined such that a respective measure of straightness of intersecting lines intersecting at the point is less than or greater than a predetermined measure of straightness by at most a specified value and such that an angle of the intersecting lines is less than or greater than ninety degrees by at most a specified number of degrees;

determining a respective relationship of the nearest neighboring points relative to the point;

determining a local orientation of the nearest neighboring points relative to a default coordinate system; and

determining a respective correlation of each of the nearest neighboring points with a corresponding point on the calibration object based at least in part upon the respective relationship of the nearest neighboring points relative to the point and the local orientation of the nearest neighboring points; and

storing information from the respective correlation for reducing at least one of misalignment or distortion effects from subsequent pairs of images captured by the first camera and the second camera, each subsequent pair of images capable of being used to generate a stereoscopic image display.

2. The computer-implemented method of claim 1, further comprising:

applying a default model, to each of the first image and the second image to remove respective distortion due to a type of the first camera and the second camera, before locating the respective points in each of the first image and the second image corresponding to the intersections of the alternating regions of the pattern.

3. The computer-implemented method of claim 1, wherein the calibration object has a first portion and a second portion joined along a vertical axis, the first portion being at an angle with respect to the second portion.

4. The computer-implemented method of claim 1, wherein the calibration object has two intersecting regions of a predetermined intensity enabling a respective anchor point of the intersecting regions to be identified in each of the first image and the second image.

5. The computer-implemented method of claim 4, further comprising:

locating the respective anchor point in each of the first image and the second image.

6. The computer-implemented method of claim 1, wherein locating the respective points in each of the first image and the second image corresponding to the intersections of the alternating regions is performed using a corners detection algorithm.

7. The computer-implemented method of claim 1, further comprising:

using a second local orientation of second nearest neighboring points of an adjacent point in the calibration object relative to the default coordinate system as a default orientation for at least one of the nearest neighboring points; and

determining a respective local orientation for each set of nearest neighboring points.

8. The computer-implemented method of claim 1, further comprising:

using a second size of an adjacent region as an estimated size for the region.

9. The computer-implemented method of claim 7, wherein relationship information stored for the respective points in each of the first image and the second image includes vector data indicating at least one of:

relationships among the respective points, relative orientations of the respective points, or relative regions sizes for the respective points.

10. The computer-implemented method of claim 1, wherein the pattern is a regular checker pattern.

11. The computer-implemented method of claim 1, wherein each of the first camera and the second camera includes a respective wide-angle lens having a respective field of view of at least ninety degrees.

12. A computing device, comprising:

at least one device processor;

a first camera having a first field of view;

a second camera separated a distance on the computing device from the first camera, the second camera having a second field of view that at least partially overlaps the first field of view; and

a memory device including instructions that, when executed by the at least one device processor, causes the computing device to;

capture a first image including a first representation of a calibration object using the first camera, the calibration object including a pattern of alternating regions;

capture a second image including a second representation of the calibration object using the second camera;

locate respective points in each of the first image and the second image corresponding to intersections of the alternating regions of the pattern;

for each point of at least a portion of the respective points in each of the first image and the second image;

analyze a region including the point to identify nearest neighboring points, a size of the region being determined such that a respective measure of straightness of intersecting lines intersecting at the point is less than or greater than a predetermined measure of straightness by at most a specified value and such that an angle of the intersecting lines is less than or greater than ninety degrees by at most a specified number of degrees;

determine a respective relationship of the nearest neighboring points relative to the point;

determine a local orientation of the nearest neighboring points relative to a default coordinate system; and

determine a respective correlation of each of the nearest neighboring points with a corresponding point on the calibration object based at least in part upon the respective relationship of the nearest neighboring points relative to the point and the local orientation of the nearest neighboring points; and

store information from the respective correlation for reducing at least one of misalignment or distortion effects from subsequent pairs of images captured by the first camera and the second camera, each subsequent pair of images capable of being used to generate a stereoscopic image display.

13. The computing device of claim 12, wherein the calibration object has a first portion and a second portion joined along a vertical axis, the first portion being at an angle with respect to the second portion, and wherein the calibration object has two intersecting regions of a predetermined intensity enabling a respective anchor point of the two intersecting regions to be identified in each of the first image and the second image.

14. The computing device of claim 13, wherein the instructions when executed further cause the computing device to:

locate the respective anchor point in each of the first image and the second image,

wherein the respective correlation of each of the nearest neighboring points is based at least in part on a respective location of each of the nearest neighboring points relative to the respective anchor point.

15. The computing device of claim 12, wherein the instructions when executed further cause the computing device to:

estimate the size of the region based at least in part upon a second size of an adjacent region.

16. The computing device of claim 12, wherein each of the first camera and the second camera includes a respective wide-angle lens having a respective field of view of at least ninety degrees.

17. The computing device of claim 12, wherein each set of nearest neighboring points lies along one or more lines of the calibration object including a respective anchor point.

18. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor of a computing device, cause the computing device to:

capture a first image including a first representation of a calibration object using a first camera, the calibration object including a pattern of alternating regions;

capture a second image including a second representation of the calibration object using a second camera separated a distance from the first camera;

locate respective points in each of the first image and the second image corresponding to intersections of the alternating regions of the pattern;

for each point of at least a portion of the respective points in each of the first image and the second image;

analyze a region including the point to identify nearest neighboring points, a size of the region being determined such that a respective measure of straightness of intersecting lines intersecting at the point is less than or greater than a predetermined measure of straightness by at most a specified value and such that an angle of the intersecting lines is less than or greater than ninety degrees by at most a specified number of degrees;

determine a respective relationship of the nearest neighboring points relative to the point;

determine a local orientation of the nearest neighboring points relative to a default coordinate system; and

determine a respective correlation of each of the nearest neighboring points with a corresponding point on the calibration object based at least in part upon the respective relationship of the nearest neighboring points relative to the point and the local orientation of the nearest neighboring points; and

store information from the respective correlation for reducing at least one of misalignment or distortion effects from subsequent pairs of images captured by the first camera and the second camera, each subsequent pair of images capable of being used to generate a stereoscopic image display.

19. The non-transitory computer-readable storage medium of claim 18, wherein the calibration object has a first portion and a second portion joined along a vertical axis, the first portion being at a predetermined angle with respect to the second portion, the calibration object having two intersecting regions of a predetermined intensity enabling a respective reference point of the two intersecting regions to be identified in each of the first image and the second image, and wherein the instructions when executed further cause the computing device to:

locate the respective reference point in each of the first image and the second image, wherein the respective points in each of the first image and the second image are located based at least in part on a respective location of each of the respective points relative to the respective reference point.

20. The non-transitory computer-readable storage medium of claim 18, wherein the instructions when executed further cause the computing device to:

locate a respective anchor point in each of the first image and the second image, wherein the respective correlation of each of the nearest neighboring points is based at least on part on a respective location of each of the nearest neighboring points relative to the respective anchor point.

21. The non-transitory computer-readable storage medium of claim 18, wherein the instructions when executed further cause the computing device to:

estimate the size of the region based at least in part upon a second size of an adjacent region.

22. A computer-implemented method, comprising:

capturing a first image including a first representation of a calibration object using a first camera, the calibration object including alternating regions;

capturing a second image including a second representation of the calibration object using a second camera separated a distance from the first camera;

determining a first point in the first image corresponding to an intersection of the alternating regions;

identifying first neighboring points of the first point in a first region of the first image that has a first size such that a first respective measure of straightness of first intersecting lines intersecting at the first point is less than or greater than a predetermined measure of straightness by at most a specified value and such that a first angle of the first intersecting lines is less than or greater than ninety degrees by at most a specified number of degrees;

determining a first orientation of the first neighboring points relative to a default coordinate system;

determining a second point in the second image corresponding to the intersection;

identifying second neighboring points of the second point in a second region of the second image that has a second size such that a second respective measure of straightness of second intersecting lines intersecting at the second point is less than or greater than the predetermined measure of straightness by at most the specified value and such that a second angle of the second intersecting lines is less than or greater than ninety degrees by at most the specified number of degrees;

determining a second orientation of the second neighboring points relative to the default coordinate system;

determining one or more adjustment values for aligning the first point and the second point based at least in part upon the first orientation and the second orientation; and

determining stereoscopic image data based at least in part upon a third image captured by the first camera, a fourth image captured by the second camera, and the one or more adjustment values.

23. The computer-implemented method of claim 22, wherein the calibration object is a physical object having a predetermined inclination between at least a first portion and a second portion.

24. The computer-implemented method of claim 22, further comprising:

determining a respective set of one or more adjustment values for aligning each intersection in the first image to a corresponding intersection in the second image.

25. The computer-implemented method of claim 24, further comprising:

determining, in a sequential pattern, respective neighboring points for each intersection in the first image,

wherein the sequential pattern is one of a spiral pattern, a horizontal line pattern, or a vertical line pattern.

26. The computer-implemented method of claim 22, wherein determining the first point in the first image includes:

locating two regions in the first image each having a predetermined intensity,

wherein the first point is an intersection of the two regions.