Systems and methods for measuring scene information while capturing images using array cameras

• US 9,917,998 B2
• Filed: 03/07/2014
• Issued: 03/13/2018
• Est. Priority Date: 03/08/2013
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: Fotonation Cayman Limited
• Inventors: Venkataraman, Kartik, Ciurea, Florian
• Primary Examiner(s): Tran, Nhan T
• Assistant Examiner(s): Yoder, III, Chriss
• Application Number:
  US14/773,748
• Publication Number:
  US 20160044257A1
• Time in Patent Office: 1,467 Days
• CPC Class Codes: H04N 23/13 with multiple sensors
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/743 Bracketing, i.e. taking a series of images with varying exposure conditions
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 25/581 acquired simultaneously
• CPC Class Codes: H04N 25/585 with pixels having different sensitivities within the sensor, e.g. fast or slow pixels or pixels having different sizes
• CPC Class Codes: H04N 25/70 SSIS architectures; Circuits associated therewith
• CPC Class Codes: H04N 5/265 Mixing
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

Systems and methods for measuring scene information while capturing images using array cameras in accordance with embodiments of the invention are disclosed. In one embodiment, a method of measuring scene information while capturing an image using an array camera includes defining at least two subsets of active cameras, configuring the active cameras using image capture settings, capturing image data using the active cameras, synthesizing at least one image using image data captured by a first subset of active cameras, measuring scene information using image data captured by a second subset of active cameras, and determining whether the image capture settings satisfy at least one predetermined criterion for at least one image capture parameter using the measured scene information, where new image capture settings are determined and utilized to configure the active cameras upon a determination that the image capture settings do not satisfy the at least one predetermined criterion.

Claims

1. A method of measuring scene information while capturing an image using an array camera, the method comprising:

defining a first subset and a second subset of active cameras in an array camera using a processor configured by software, wherein the first subset includes a first active camera, the second subset includes a second active camera different than the first active camera, and the first subset includes more than one active camera;

configuring the first subset and the second subset of active cameras in the array camera with image capture settings for each subset using the processor configured by software;

capturing image data using the active cameras in the array camera;

synthesizing at least one image from image data captured by the first subset of active cameras using the processor configured by software;

measuring scene information using the processor configured by software based upon image data captured by the second subset of active cameras, wherein measuring the scene information is performed independently of image data captured by the first subset of active cameras;

determining whether the image capture settings satisfy at least one predetermined criterion for at least one image capture parameter based upon the measured scene information obtained using the image data captured by the second subset of active cameras using the processor configured by software; and

when the image capture settings do not satisfy the at least one predetermined criterion for at least one image capture parameter, determine new image capture settings and configure the first subset of active cameras in the array camera, independently of the second subset of active cameras, with the new image capture settings using the processor configured by software.

2. The method of claim 1, wherein a subset of active cameras is configured using image capture settings specific to the entire subset of active cameras.

3. The method of claim 1, wherein a subset of active cameras is configured using image capture settings specific to each individual active camera within the subset.

4. The method of claim 1, wherein synthesizing at least one image using image data captured by the first subset of active cameras comprises fusing image data captured by the cameras in the first subset of active cameras to create a high resolution image.

5. The method of claim 4, wherein synthesizing at least one image using image data captured by the first subset of active cameras further comprises performing super resolution processing using the image data captured by the cameras in the first subset of active cameras to create a high resolution image.

6. The method of claim 1, wherein the at least one predetermined criterion is at least one predetermined criterion for a parameter selected from the group consisting of exposure, focus settings, shutter speed, aperture, and light sensitivity.

7. The method of claim 1, wherein the cameras in the second subset of active cameras are configured using different image capture settings and measure different scene information in parallel.

8. The method of claim 1, wherein the new image capture settings are used to configure the first subset of active cameras to capture an additional set of image data used to synthesize an image and the new image capture settings for the second subset of active cameras configure the second set of active cameras to capture additional image data in order to measure additional information concerning the scene.

9. The method of claim 8, wherein the new image capture settings for the second subset of active cameras are determined so that the first subset of cameras can smoothly transition to the image capture settings of any camera in the second subset of active cameras.

10. The method of claim 1, wherein the new image capture settings are used to configure a set of active cameras to capture an additional set of image data used to synthesize an image and the set of active cameras is larger than the first subset of active cameras and includes at least one camera from the second subset of cameras.

11. The method of claim 1, wherein:

the measured scene information comprises exposure measurements at each of the different image capture settings; and

the at least one predetermined criterion for at least one image capture parameter is at least one predetermined criterion with respect to auto-exposure.

12. The method of claim 1, wherein:

the measured scene information comprises measurements of the dynamic range of the scene;

the at least one predetermined criterion for at least one image capture parameter comprises determination of image capture settings for high dynamic range image capture based upon measurements of the dynamic range of the scene; and

when the image capture settings satisfy the at least one predetermined criterion for at least one image capture parameter;

configure a plurality of subsets of active cameras using the processor configured by software, where each subset of active cameras has different image capture settings;

capture image data using the plurality of active cameras; and

synthesize at least one high dynamic range image from image data captured by the plurality of active cameras using the processor configured by software.

13. The method of claim 1, wherein:

cameras in the array camera include autofocus mechanisms;

the measured scene information comprises focus measurements at each of the different image capture settings; and

the at least one predetermined criterion for at least one image capture parameter is at least one predetermined criterion with respect to autofocus.

14. The method of claim 1, wherein the first subset of active cameras comprises a 3×

3 array of active cameras patterned using a π

filter group, where the π

filter group includes;

a green camera at each corner,

a green reference camera in the center,

blue cameras on opposite sides of the reference camera, and

red cameras on opposite sides of the reference camera.

15. The method of claim 1, wherein the array camera includes a camera module comprising an array of individual cameras.

16. The method of claim 15, wherein the camera module comprises:

an imager array including an array of focal planes, where each focal plane comprises an array of light sensitive pixels;

an optic array including an array of lens stacks, where each lens stack creates an optical channel that forms an image on the array of light sensitive pixels within a corresponding focal plane;

wherein pairings of lens stacks and focal planes form multiple cameras including the plurality of active cameras.

17. The method of claim 16, wherein the lens stacks within the optical channels sample the same object space with sub-pixel offsets to provide sampling diversity.

18. A method of measuring scene information while capturing an image using an array camera, the method comprising:

configuring a plurality of active cameras within an array camera with different image capture settings using a processor configured by software, the plurality of active cameras including a first subset and a second subset of active cameras, wherein the first subset includes a first active camera, and the second subset includes a second active camera different than the first active camera;

capturing image data using the active cameras in the array camera;

measuring scene information using the processor configured by software based upon image data captured by the second subset of active cameras in parallel with the first subset of active cameras using different image capture settings, wherein measuring the scene information is performed independently of image data captured by the first subset of active cameras;

determining whether image capture settings satisfy at least one predetermined criterion for at least one image capture parameter based upon the measured scene information obtained using the image data captured by the second subset of active cameras using the processor configured by software;

when the image capture settings do not satisfy the at least one predetermined criterion for at least one image capture parameter, determine a first set of new image capture settings and a second set of new image capture settings, configure the first subset of active cameras with the first set of new image capture settings, and configure the second subset of active cameras in the array camera with the second set of new image capture settings to make additional measurements of scene information using the processor configured by software; and

when the image capture settings satisfy the at least one predetermined criterion for at least one image capture parameter;

configure a plurality of active cameras in the array camera using the image capture settings using the processor configured by software;

capture image data using the plurality of active cameras; and

synthesize at least one image from image data captured by the plurality of active cameras using the processor configured by software.

19. The method of claim 18, wherein:

the measured scene information comprises exposure measurements at each of the different image capture settings; and

the at least one predetermined criterion for at least one image capture parameter is at least one predetermined criterion with respect to auto-exposure.

20. The method of claim 18, wherein:

the measured scene information comprises measurements of the dynamic range of the scene;

the at least one predetermined criterion for at least one image capture parameter comprises determination of image capture settings for high dynamic range image capture based upon measurements of the dynamic range of the scene;

configuring a plurality of active cameras using the image capture settings further comprises configuring a plurality of subsets of active cameras using the processor configured by software, where each subset of active cameras has different image capture settings; and

synthesizing at least one image from image data captured by the plurality of active cameras further comprises synthesizing at least one high dynamic range image from image data captured by the plurality of active cameras using the processor configured by software.

21. The method of claim 18, wherein:

cameras in the array camera include autofocus mechanisms;

the measured scene information comprises focus measurements at each of the different image capture settings; and

the at least one predetermined criterion for at least one image capture parameter is at least one predetermined criterion with respect to autofocus.