System and methods for calibration of an array camera

• US 10,560,684 B2
• Filed: 03/01/2019
• Issued: 02/11/2020
• Est. Priority Date: 03/10/2013
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Mullis, Robert
• Primary Examiner(s): Hsia, Sherrie
• Application Number:
  US16/290,672
• Publication Number:
  US 20190268586A1
• Time in Patent Office: 347 Days
• Field of Search: 348187, 348188, 348175, 3482181, 348135, 348 47, 348 48, 700258, 700259, 702104, 702116, 382103, 382106, 382107, 382153, 382154, 382284
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20048 Transform domain processing
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 17/02 for colour television signals
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

Systems and methods for calibrating an array camera are disclosed. Systems and methods for calibrating an array camera in accordance with embodiments of this invention include the capturing of an image of a test pattern with the array camera such that each imaging component in the array camera captures an image of the test pattern. The image of the test pattern captured by a reference imaging component is then used to derive calibration information for the reference component. A corrected image of the test pattern for the reference component is then generated from the calibration information and the image of the test pattern captured by the reference imaging component. The corrected image is then used with the images captured by each of the associate imaging components associated with the reference component to generate calibration information for the associate imaging components.

Claims

1. A method for manufacturing an array camera device, the method comprising:

assembling an array of cameras comprising a plurality of imaging components that capture images of a scene from different viewpoints;

configuring the array of cameras to communicate with at least one processor;

configuring the processor to communicate with at least one type of memory; and

performing a calibration process for the array of cameras, where the calibration process comprises;

capturing images of a test pattern using the array of cameras, where each of the plurality of imaging components captures an image from a particular viewpoint;

generating a first set of scene independent geometric corrections for image data captured by a first imaging component using a first set of test pattern image data captured by the first imaging component and data describing the test pattern using the processor;

generating a corrected test pattern image based on the first set of scene independent geometric corrections and the first set of test pattern image data captured by the first imaging component using the processor; and

generating a second set of scene independent geometric corrections for image data captured by a second imaging component of the plurality of imaging components using a second set of test pattern image data captured by the second imaging component and data for the corrected test pattern image using the processor; and

loading calibration information into the memory, wherein the calibration information is based on the first and second sets of generated scene independent geometric corrections.

2. The method of claim 1, wherein the first imaging component is a reference imaging component and the second imaging component is an associate imaging component, wherein the calibration information comprises:

reference calibration information for the reference imaging component comprising the scene independent geometric corrections for reference image data captured by the reference imaging component to account for distortions related to the mechanical construction of the reference imaging component and produce a corrected reference image; and

associate calibration information for the associate imaging component comprising the scene independent geometric corrections for associate image data captured by the associate imaging component that map locations of pixels in an image captured by the associate imaging component to corresponding pixel locations in the corrected reference image, where corresponding pixel locations represent the same point in a scene in the absence of disparity due to parallax.

3. The method of claim 2, wherein the calibration information further comprises colorimetric corrections or photometric corrections for image data captured by one or more imaging components of the plurality of imaging components.

4. The method of claim 2, further comprising loading a software application comprising machine readable instructions into the memory, where execution of the software application by the processor directs the processor to:

capture images of a scene using the plurality of imaging components in the array of cameras, wherein the captured images comprise;

an associate image captured by the associate imaging component; and

a reference image captured by the reference imaging component;

apply corrections to locations of pixels of the associate image using the associate calibration information;

generate a depth map by measuring disparity due to parallax between pixels in the reference image and corrected pixels in the associate image; and

synthesize an image using the generated depth map and at least some of the pixels from the captured images.

5. The method of claim 4, wherein the execution of the software application by the processor further directs the processor to apply corrections to locations of pixels of the reference image using the reference calibration information;

and

wherein the depth map is generated by measuring disparity due to parallax between the corrected pixels in the reference image and the corrected pixels in the associate image.

6. The method of claim 1, wherein the test pattern is placed at a defined distance away from the array of cameras when the images of the test pattern are captured, and the distance is at least 70 percent of a hyperfocal distance of the array of cameras.

7. The method of claim 1, wherein the test pattern is placed at a defined distance away from the array of cameras when the images of the test pattern are captured, and the distance is at least 50 percent of a hyperfocal distance of the array of cameras.

8. The method of claim 1, wherein the test pattern includes a low-contrast slanted edge pattern.

9. The method of claim 8, wherein the test pattern includes a plurality of Macbeth Color Chart type patterns inset at different positions in the low-contrast slanted pattern.

10. The method of claim 1, further comprising performing at least one pass/fail test of the array of cameras based on captured images of the test pattern to verify proper image capture by the plurality of imaging components.

11. The method of claim 1, wherein generating scene independent geometric corrections for reference image data comprises:

identifying reference intersection points in the image of the test pattern captured by the reference imaging component;

determining uniformity characteristics of the reference imaging component from reference intersection points and the test pattern; and

deriving parameters for the reference imaging component to compensate for low frequency aberrations in the image of the test pattern captured by the reference imaging component.

12. The method of claim 1, wherein the first imaging component is a reference imaging component and the second imaging component is an associate imaging component, wherein generating the second set of scene independent geometric corrections comprises:

identifying associate intersection points in images of the test pattern captured by the associate imaging component;

translating associate intersection points in accordance with an expected parallax shift for the associate imaging component relative to the reference imaging component; and

deriving parameters for the associate imaging component to compensate for low frequency aberrations in the image of the test pattern captured by the associate imaging component by comparing translated associate intersection points to corresponding intersection points in the corrected test pattern image for the reference imaging component.

13. The method of claim 12, wherein the expected parallax shift for the associate imaging component is based upon at least one of the physical offset of the associate imaging component to the reference imaging component, the behavior of sensor optics in the associate imaging component, and a distance of the test pattern from the array of cameras.

14. The method of claim 1, further comprising:

generating colorimetric corrections or photometric corrections for image data captured by each imaging component in the array of cameras using test pattern image data captured by each imaging component using the processor; and

storing the generated colorimetric corrections or photometric corrections in the memory.

15. The method of claim 1, wherein the array of cameras includes more than one reference imaging component.

16. The method of claim 1, wherein the processor includes a graphics processing unit.

17. The method of claim 1, wherein the second set of scene independent geometric corrections are represented by a grid that provides a geometric correction prescription for pixels of the second imaging component.

18. The method of claim 1, wherein the plurality of imaging components are configured in a 5×

5 array.

19. The method of claim 1, wherein at least one imaging component of the plurality of imaging components contains a lens stack array and at least one sensor element, where the sensor element is selected from the group of a traditional CIS (CMOS Image Sensor) pixel, a CCD (charge-coupled device) pixel, a high dynamic range sensor element, a multispectral sensor element, and a structure configured to generate an electrical signal indicative of light incident on the structure.

20. A non-transitory machine readable medium containing processor instructions, where execution of the instructions by a processor causes the processor to perform a calibration process comprising:

capturing images of a test pattern using the array of cameras, where each of the plurality of imaging components captures an image from a particular viewpoint;

generating a first set of scene independent geometric corrections for image data captured by a first imaging component using a first set of test pattern image data captured by the first imaging component and data describing the test pattern using a processor;

generating a corrected test pattern image based on the first set of scene independent geometric corrections and the first set of test pattern image data captured by the first imaging component using the processor; and

generating a second set of scene independent geometric corrections for image data captured by a second imaging component of the plurality of imaging components using a second set of test pattern image data captured by the second imaging component and data for the corrected test pattern image using the processor; and

loading calibration information into a memory, wherein the calibration information is based on the first and second sets of generated scene independent geometric corrections.