Target-Less Auto-Alignment Of Image Sensors In A Multi-Camera System

• US 20150271483A1
• Filed: 03/20/2014
• Published: 09/24/2015
• Est. Priority Date: 03/20/2014
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Sun, Yajie, Macmillan, Timothy, Chawla, Sumit, Martin, Elizabeth Torres
• Granted Patent: US 9,197,885 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 25/48 Increasing resolution by shifting the sensor relative to the scene
• CPC Class Codes: H04N 25/76 Addressed sensors, e.g. MOS or CMOS sensors

Abstract

A pair of cameras having an overlapping field of view is aligned based on images captured by image sensors of the pair of cameras. A pixel shift is identified between the images. Based on the identified pixel shift, a calibration is applied to one or both of the pair of cameras. To determine the pixel shift, the camera applies correlation methods including edge matching. Calibrating the pair of cameras may include adjusting a read window on an image sensor. The pixel shift can also be used to determine a time lag, which can be used to synchronize subsequent image captures.

Claims

1. A computer-implemented method for determining a pixel shift between an image pair, the method comprising:

accessing a first image and a second image of the image pair captured at a substantially same time, the images comprising image data representative of an overlapping field of view between the image sensors;

identifying, by one or more processors, one or more edges in the image data corresponding to the first image and the second image;

matching the identified one or more edges in the image data corresponding to the first image to the identified one or more edges in the image data corresponding to the second image; and

determining a pixel shift between the image pair based, at least in part, on the matching of edges.

2. The method of claim 1, wherein the image data comprises a luma component for each image, and wherein identifying the one or more edges in the image data corresponding to an image comprises:

determining edge magnitude and edge phase of the image data based on the luma component; and

identifying one or more edges in the image data based, at least in part, on the determined edge magnitude and edge phase of the image data.

3. The method of claim 2, wherein identifying the one or more edges in the image data comprises:

identifying a first set of image data having an edge magnitude greater than an edge magnitude threshold;

identifying a second set of image data having an edge phase within a threshold of a target edge phase value; and

identifying the one or more edges in the image data based on the identified first set of image data and the identified second set of image data.

4. The method of claim 2, wherein identifying the one or more edges in the image data further comprises:

determining an edge length for each of a plurality of candidate edges; and

identifying one or more candidate edges having an edge length greater than an edge length threshold.

5. The method of claim 2, wherein matching the identified one or more edges in the image data corresponding to the first image to the identified one or more edges in the image data corresponding to the second image comprises:

determining edge lengths for the identified one or more edges; and

matching the one or more edges based, at least in part, on the determined edge magnitude of the one or more edges and the determined edge lengths.

6. The method of claim 1, further comprising:

before identifying one or more edges in the image data, compressing the first and second images by downsampling the first and second images by a downsampling factor.

7. The method of claim 6, wherein the determined pixel shift is multiplied by the downsampling factor.

8. The method of claim 1, further comprising:

smoothing the first and second images prior to identifying the one or more edges in the image data corresponding to the first image and the second image.

9. A non-transitory computer readable medium comprising instructions for determining a pixel shift between an image pair, the instructions executable by a processor, the instructions comprising instructions for:

accessing a first image and a second image of the image pair captured at a substantially same time, the images comprising image data representative of an overlapping field of view between the image sensors;

identifying, by one or more processors, one or more edges in the image data corresponding to the first image and the second image;

matching the identified one or more edges in the image data corresponding to the first image to the identified one or more edges in the image data corresponding to the second image; and

determining a pixel shift between the image pair based, at least in part, on the matching of edges.

10. The computer readable medium of claim 9, wherein the image data comprises a luma component, and wherein the instructions for identifying the one or more edges in the image data comprise instructions for:

determining edge magnitude and edge phase of the image data based on the luma component; and

identifying one or more edges in the image data based, at least in part, on the determined edge magnitude and edge phase of the image data.

11. The computer readable medium of claim 10, wherein the instructions for identifying the one or more edges in the image data comprise instructions for:

identifying a first set of image data having an edge magnitude greater than an edge magnitude threshold;

identifying a second set of image data having an edge phase within a threshold of a target edge phase value; and

identifying the one or more edges in the image data based on the identified first set of image data and the identified second set of image data.

12. The computer readable medium of claim 10, wherein the instructions for identifying the one or more edges in the image data comprise instructions for:

determining an edge length for each of a plurality of candidate edges; and

identifying one or more candidate edges having an edge length greater than an edge length threshold.

13. The computer readable medium of claim 10, wherein the instructions for matching the identified one or more edges from the first image to the identified one or more edges in the second image comprise instructions for:

determining edge lengths for the identified one or more edges; and

matching the one or more edges based, at least in part, on the determined edge magnitude of the one or more edges and the determined edge lengths.

14. The computer readable medium of claim 9, further comprising instructions for:

before identifying one or more edges in the image data, compressing the first and second images by downsampling the first and second images by a downsampling factor.

15. A camera system comprising at least one processor and a non-transitory computer readable medium comprising instructions for determining a pixel shift between an image pair, the instructions executable by the at least one processor, the instructions comprising instructions for:

accessing a first image and a second image of the image pair captured at a substantially same time, the images comprising image data representative of an overlapping field of view between the image sensors;

identifying, by one or more processors, one or more edges in the image data corresponding to the first image and the second image;

matching the identified one or more edges in the image data corresponding to the first image to the identified one or more edges in the image data corresponding to the second image; and

determining a pixel shift between the image pair based, at least in part, on the matching of edges.

16. The camera system of claim 15, wherein the image data comprises a luma component, and wherein the instructions for identifying the one or more edges in the image data comprise instructions for:

determining edge magnitude and edge phase of the image data based on the luma component; and

identifying one or more edges in the image data based, at least in part, on the determined edge magnitude and edge phase of the image data.

17. The camera system of claim 16, wherein the instructions for identifying the one or more edges in the image data comprise instructions for:

identifying a first set of image data having an edge magnitude greater than an edge magnitude threshold;

identifying a second set of image data having an edge phase within a threshold of a target edge phase value; and

identifying the one or more edges in the image data based on the identified first set of image data and the identified second set of image data.

18. The camera system of claim 16, wherein the instructions for identifying the one or more edges in the image data comprise instructions for:

determining an edge length for each of a plurality of candidate edges; and

identifying one or more candidate edges having an edge length greater than an edge length threshold.

19. The camera system of claim 16, wherein the instructions for matching the identified one or more edges from the first image to the identified one or more edges in the second image comprise instructions for:

determining edge lengths for the identified one or more edges; and

matching the one or more edges based, at least in part, on the determined edge magnitude of the one or more edges and the determined edge lengths.

20. The camera system of claim 15, further comprising instructions for:

before identifying one or more edges in the image data, compressing the first and second images by downsampling the first and second images by a downsampling factor.