Auto-alignment of image sensors in a multi-camera system

• US 8,988,509 B1
• Filed: 03/20/2014
• Issued: 03/24/2015
• Est. Priority Date: 03/20/2014
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Campbell, Scott Patrick, Newman, David A., Sun, Yajie, Macmillan, Timothy
• Primary Examiner(s): Lee, Michael
• Application Number:
  US14/221,254
• Time in Patent Office: 369 Days
• Field of Search: 348 36- 39, 348/211.2, 348/218.1, 348/388, 348/177, 348 46- 50
• US Class Current: 348/47
• CPC Class Codes: H04N 13/00 Stereoscopic video systems; Multi-view video systems; Details thereof
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 25/531 by controlling rolling shutters in CMOS SSIS
• CPC Class Codes: H04N 5/265 Mixing

Abstract

A pair of cameras having an overlapping field of view is aligned based on images captured by image sensors of the pair of cameras. A pixel shift is identified between the images. Based on the identified pixel shift, a calibration is applied to one or both of the pair of cameras. To determine the pixel shift, the camera applies correlation methods including edge matching. Calibrating the pair of cameras may include adjusting a read window on an image sensor. The pixel shift can also be used to determine a time lag, which can be used to synchronize subsequent image captures.

Claims

1. A computer-implemented method for synchronizing a pair of image sensors, the method comprising:

capturing an image with each image sensor at a substantially same time, the image sensors having a rolling shutter direction and an overlapping field of view;

identifying, by one or more processors, a pixel shift between the captured images based on captured image data representative of the overlapping field of view; and

calibrating, based on the identified pixel shift, at least one image sensor to synchronize subsequent image capture by the image sensors by adjusting a read window of the at least one image sensor in the rolling shutter direction, the read window comprising a portion of the image sensor configured to capture light incident upon the portion to produce captured image data during an image capture.

2. The method of claim 1, wherein identifying the pixel shift comprises:

correlating the captured image data representative of the overlapping field of view by shifting at least one image by a first number of pixels such that a measure of difference between pixels of the captured image data representative of the overlapping field of view is substantially minimized;

wherein the first number of pixels comprises the identified pixel shift.

3. The method of claim 2, wherein correlating the captured image data representative of the overlapping field of view comprises:

compressing, for each image, pixels in the captured image data representative of the overlapping field of view; and

correlating the compressed pixels of each image.

4. The method of claim 2, further comprising:

responsive to a determination that the captured images include less than a threshold amount of image data representative of the overlapping field of view required to correlate the captured image data, capturing an additional pair of images having an overlapping field of view with the image sensors.

5. The method of claim 1, further comprising:

determining a time lag between the image sensors based on the identified pixel shift.

6. The method of claim 5, wherein determining the time lag between the image sensors comprises determining an amount of time necessary for one of the image sensors to capture a number of pixel lines corresponding to the pixel shift.

7. The method of claim 5, wherein a first image sensor of the pair of image sensors lags a second image sensor of the pair of image sensors during image capture, and wherein calibrating the at least one image sensor comprises:

initiating image capture at the first image sensor; and

after delaying for an amount of time equal to the time lag, initiating image capture at the second image sensor.

8. The method of claim 1, wherein the pair of image sensors are controlled by a master camera controlling a slave camera.

9. The method of claim 1, repeated to synchronize a plurality of pairs of image sensors.

10. A system comprising a pair of cameras, each camera having an image sensor, the system further comprising at least one processor and a non-transitory computer readable medium comprising instructions for synchronizing the image sensors, the instructions executable by the at least one processor, the instructions comprising instructions for:

capturing an image with each image sensor at a substantially same time, the image sensors having a rolling shutter direction and an overlapping field of view;

identifying, by one or more processors, a pixel shift between the captured images based on captured image data representative of the overlapping field of view; and

calibrating, based on the identified pixel shift, at least one image sensor to synchronize subsequent image capture by the image sensors by adjusting a read window of the at least one image sensor in the rolling shutter direction, the read window comprising a portion of the image sensor configured to capture light incident upon the portion to produce captured image data during an image capture.

11. The system of claim 10, wherein the instructions for identifying the pixel shift comprise instructions for:

correlating the captured image data representative of the overlapping field of view by shifting at least one image by a first number of pixels such that a measure of difference between pixels of the captured image data representative of the overlapping field of view is substantially minimized;

wherein the first number of pixels comprises the identified pixel shift.

12. The system of claim 10, further comprising instructions for:

determining a time lag between the image sensors based, at least in part, on the identified pixel shift and an amount of time necessary for one of the image sensors to capture a number of pixel lines corresponding to the pixel shift.

13. The system of claim 12, wherein a first image sensor of the pair of image sensors lags a second image sensor of the pair of image sensors during image capture, and wherein calibrating the at least one image sensor comprises:

initiating image capture at the first image sensor; and

after delaying for an amount of time equal to the time lag, initiating image capture at the second image sensor.

18. The system of claim 10, further comprising a housing encompassing the pair of cameras.

19. The system of claim 10, further comprising a third camera having an additional image sensor, the instructions further comprising instructions to synchronize the additional image sensor with the image sensors.

14. A non-transitory computer readable medium comprising instructions for synchronizing a pair of image sensors, the instructions executable by at least one processor, the instructions comprising instructions for:

capturing an image with each image sensor at a substantially same time, the image sensors having a rolling shutter direction and an overlapping field of view;

identifying, by one or more processors, a pixel shift between the captured images based on captured image data representative of the overlapping field of view; and

calibrating, based on the identified pixel shift, at least one image sensor to synchronize subsequent image capture by the image sensors by adjusting a read window of the at least one image sensor in the rolling shutter direction, the read window comprising a portion of the image sensor configured to capture light incident upon the portion to produce captured image data during an image capture.

15. The computer readable medium of claim 14, wherein the instructions for identifying the pixel shift comprise instructions for:

correlating the captured image data representative of the overlapping field of view by shifting at least one image by a first number of pixels such that a measure of difference between pixels of the captured image data representative of the overlapping field of view is substantially minimized;

wherein the first number of pixels comprises the identified pixel shift.

16. The computer readable medium of claim 14, further comprising instructions for:

determining a time lag between the image sensors based, at least in part, on the identified pixel shift and an amount of time necessary for one of the image sensors to capture a number of pixel lines corresponding to the pixel shift.

17. The computer readable medium of claim 16, wherein a first image sensor of the pair of image sensors lags a second image sensor of the pair of image sensors during image capture, and wherein calibrating the at least one image sensor comprises:

initiating image capture at the first image sensor; and

after delaying for an amount of time equal to the time lag, initiating image capture at the second image sensor.

20. The computer readable medium of claim 14, the instructions further comprising instructions to synchronize an additional image sensor with the pair of image sensors.