Auto-alignment of image sensors in a multi-camera system

• US 10,389,993 B2
• Filed: 03/23/2016
• Issued: 08/20/2019
• Est. Priority Date: 03/20/2014
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Macmillan, Timothy, Campbell, Scott Patrick, Newman, David A., Sun, Yajie
• Primary Examiner(s): Lee, Michael
• Application Number:
  US15/077,911
• Publication Number:
  US 20160205381A1
• Time in Patent Office: 1,245 Days
• Field of Search: 348 46
• CPC Class Codes: H04N 13/00 Stereoscopic video systems; Multi-view video systems; Details thereof
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 25/531 by controlling rolling shutters in CMOS SSIS
• CPC Class Codes: H04N 5/265 Mixing

Abstract

A pair of cameras having an overlapping field of view is aligned based on images captured by image sensors of the pair of cameras. A pixel shift is identified between the images. Based on the identified pixel shift, a calibration is applied to one or both of the pair of cameras. To determine the pixel shift, the camera applies correlation methods including edge matching. Calibrating the pair of cameras may include adjusting a read window on an image sensor. The pixel shift can also be used to determine a time lag, which can be used to synchronize subsequent image captures.

Claims

1. A method for synchronizing a pair of image sensors, the method comprising:

accessing a pair of images synchronously captured by the pair of image sensors, wherein the pair of image sensors include an image sensor of a first camera and an image sensor of a second camera, wherein the pair of image sensors are controlled by the first camera controlling the image sensor of the second camera;

identifying a pixel shift of a first of the pair of images relative to a second of the pair of images by matching a first edge of the first of the pair of images with a second edge of the second of the pair of images;

determining a time lag between the pair of image sensors based on the identified pixel shift, the time lag comprising an amount of time required to capture a number of pixel lines corresponding to the identified pixel shift;

generating a calibration measure based on the identified pixel shift for calibrating at least one image sensor to synchronize subsequent image capture by the pair of image sensors; and

capturing a subsequent pair of images based on the generated calibration measure.

2. The method of claim 1, wherein identifying the pixel shift comprises:

compressing, for each image, image pixels representative of an overlapping field of view of the pair of image sensors; and

correlating the compressed pixels of each image.

3. The method of claim 1, further comprising:

calibrating one or both of the pair of image sensors using the generated calibration measure.

4. The method of claim 3, wherein calibrating one or both of the pair of image sensors comprises:

adjusting a read window of one or both of the pair image sensors in a rolling shutter direction of the pair of image sensors based at least in part on the generated calibration measure, each read window comprising a portion of an image sensor configured to capture light incident upon the portion to produce captured image data during an image capture.

5. The method of claim 3, wherein each image sensor is located within a different camera system of a pair of communicatively coupled camera systems, and wherein one of the pair of communicatively coupled camera systems provides the generated calibration measure to the other of the pair of communicatively coupled camera systems.

6. The method of claim 3, wherein each image sensor is located within a different camera system of a pair of camera systems, wherein each of the pair of camera systems is communicatively coupled to an external controller, and wherein the external controller is configured to provide the generated calibration measure to one or both of the pair of camera systems.

7. The method of claim 6, wherein the pair of images are synchronously captured in response to an instruction provided by the external controller.

8. A multi-camera system, each camera having an image sensor, the multi-camera system configured to:

access a pair of images synchronously captured by the image sensors, wherein the image sensors include an image sensor of a first camera and an image sensor of a second camera, wherein the image sensors are controlled by the first camera controlling the image sensor of the second camera;

identify a pixel shift of a first of the pair of images relative to a second of the pair of images by matching a first edge of the first of the pair of images with a second edge of the second of the pair of images;

determine a time lag between the pair of image sensors based on the identified pixel shift, the time lag comprising an amount of time required to capture a number of pixel lines corresponding to the identified pixel shift;

generate a calibration measure based on the identified pixel shift for calibrating at least one image sensor to synchronize subsequent image capture by the image sensors; and

capture a subsequent pair of images based on the generated calibration measure.

9. The system of claim 8, wherein identifying the pixel shift comprises:

compressing, for each image, image pixels representative of an overlapping field of view of the image sensors; and

correlating the compressed pixels of each image.

10. The system of claim 8, further configured to calibrate one or more of the image sensors using the generated calibration measure.

11. The system of claim 10, wherein calibrating one or more of the image sensors comprises:

adjusting a read window of one or more of the image sensors in a rolling shutter direction of the image sensors based at least in part on the generated calibration measure, each read window comprising a portion of an image sensor configured to capture light incident upon the portion to produce captured image data during an image capture.

12. The system of claim 10, wherein one camera provides the generated calibration measure to another camera.

13. The system of claim 10, further comprising:

a controller configured to provide the generated calibration measure to each camera.

14. The system of claim 13, wherein the pair of images are synchronously captured in response to an instruction provided by the controller.

15. A non-transitory computer-readable medium comprising instructions for synchronizing a pair of image sensors, the instructions executable by at least one processor, the instructions comprising instructions for:

accessing a pair of images synchronously captured by the pair of image sensors, wherein the pair of image sensors include an image sensor of a first camera and an image sensor of a second camera, wherein the pair of image sensors are controlled by the first camera controlling the image sensor of the second camera;

identifying a pixel shift of a first of the pair of images relative to a second of the pair of images by matching a first edge of the first of the pair of images to a second edge of the second of the pair of images;

determining a time lag between the pair of image sensors based on the identified pixel shift, the time lag comprising an amount of time required to capture a number of pixel lines corresponding to the identified pixel shift;

generating a calibration measure based on the identified pixel shift for calibrating at least one image sensor to synchronize subsequent image capture by the pair of image sensors; and

capturing a subsequent pair of images based on the generated calibration measure.

16. The computer-readable medium of claim 15, wherein identifying the pixel shift comprises:

compressing, for each image, image pixels representative of an overlapping field of view of the pair of image sensors; and

correlating the compressed pixels of each image.

17. The computer-readable medium of claim 15, the instructions further comprising instructions for:

calibrating one or both of the pair of image sensors using the generated calibration measure.

18. The computer-readable medium of claim 17, wherein calibrating one or both of the pair of image sensors comprises:

adjusting a read window of one or both of the pair of image sensors in a rolling shutter direction of the pair of image sensors based at least in part on the generated calibration measure, each read window comprising a portion of an image sensor configured to capture light incident upon the portion to produce captured image data during an image capture.

19. The method of claim 17, wherein each image sensor is located within a different camera system of a pair of communicatively coupled camera systems, and wherein one of the pair of communicatively coupled camera systems provides the generated calibration measure to the other of the pair of communicatively coupled camera systems.

20. The method of claim 17, wherein each image sensor is located within a different camera system of a pair of camera systems, wherein each of the pair of camera systems is communicatively coupled to an external controller, and wherein the external controller is configured to provide the generated calibration measure to one or both of the pair of camera systems.