Autofocus system for a conventional camera that uses depth information from an array camera

• US 10,122,993 B2
• Filed: 05/28/2015
• Issued: 11/06/2018
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Venkataraman, Kartik, Gallagher, Paul, Jain, Ankit K., Nisenzon, Semyon, Lelescu, Dan, Ciurea, Florian, Molina, Gabriel
• Primary Examiner(s): Le, Peter D
• Application Number:
  US14/724,447
• Publication Number:
  US 20150264337A1
• Time in Patent Office: 1,258 Days
• CPC Class Codes: G01P 3/38 using photographic means
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps

Abstract

Systems with an array camera augmented with a conventional camera in accordance with embodiments of the invention are disclosed. In some embodiments, the array camera is used to capture a first set of image data of a scene and a conventional camera is used to capture a second set of image data for the scene. An object of interest is identified in the first set of image data. A first depth measurement for the object of interest is determined and compared to a predetermined threshold. If the first depth measurement is above the threshold, a second set of image data captured using the conventional camera is obtained. The object of interest is identified in the second set of image data and a second depth measurement for the object of interest is determined using at least a portion of the first set of image data and at least a portion of the second set of image data.

Claims

1. An array camera system, comprising:

an array camera comprising a plurality of cameras and a separate camera, where;

the plurality of cameras capture images of a scene from different viewpoints;

the separate camera has a fixed geometric relationship with each of the plurality of cameras in the array camera; and

the separate camera captures an image of the scene from a different viewpoint to the viewpoints of the other cameras in the array camera;

a processor; and

memory in communication with the processor storing software;

wherein the software directs the processor to;

obtain a focus window based upon image data from the separate camera, where;

at least one object is detected within an image captured by the separate camera; and

the focus window is a defined region having a set of pixels, where the defined region is within an image captured by the separate camera based upon the location of the at least one object;

determine a focus window for the array camera based upon the focus window from the separate camera, where the focus window for the array camera comprises a corresponding set of pixels that corresponds to the set of pixels within the focus window from the separate camera;

obtain image data pertaining to the focus window of the array camera from at least two cameras in the plurality of cameras,

determine depth information for the focus window of the array camera from the image data from the at least two cameras,

colocate the depth information from the focus window of the array camera to the focus window of the separate camera to account for parallax between the separate camera and the plurality of cameras to generate depth information for the focus window of the separate camera,

determine a focus depth for the separate camera based upon the depth information for the focus window of the separate camera, and

adjust the focus of the separate camera to a desired focus based upon the determined focus depth for the separate camera.

2. The array camera system of claim 1 wherein the focus depth is determined using a histogram of depths in the depth information for the focus window of the separate camera.

3. The array camera system of claim 1 wherein the focus depth is the closest depth to the separate camera in the depth information of the focus window of the separate camera.

4. The array camera system of claim 3 wherein the focus depth is the closest depth in the focus depth information of the focus window of the separate camera that includes a number of pixels that is greater than a threshold of a predefined number of pixels.

5. The array camera system of claim 1 wherein the software further directs the processor to:

determine whether the focus depth for the separate camera can be determined from depth information; and

determine the focus depth for the separate camera using a conventional autofocus process to determine the focus depth in response to a determination that the focus depth cannot be determined from depth information.

6. The array camera system of claim 1 wherein the software directs the processor to obtain the focus window information for the separate camera by:

receiving image data from the separate camera; and

determining the focus window from the image data from the separate camera.

7. The array camera system of claim 6 wherein the focus window is determined by performing object detection on the image data and selecting one or more detected objects.

8. The array camera system of claim 7 where the object detection is performed in an area of the image data input by the user.

9. The array camera system of claim 6 wherein the focus window is determined by an input received from the user.

10. The array camera system of claim 1 wherein the focus window of the array camera is determined by mapping the focus window in the separate camera to the focus window in a reference camera of the array camera while accounting for the parallax between the separate camera and the array camera.

11. The array camera system of claim 1 wherein the colocating of the depth information from the array camera to the depth information for the separate camera occurs in the absence of depth regularization.

12. A method for autofocusing a separate camera in an array camera system including an array camera comprising a plurality of cameras that capture images of a scene from different viewpoints, a separate camera in a fixed geometric relationship with respect to each of the plurality of cameras in the array camera, where the separate camera captures an image of the scene from a different viewpoint to the viewpoints of the other cameras in the array camera;

a processor, and memory in communication with the processor storing software, comprising;

obtaining a focus window based upon image data from the separate camera using the processor, comprising;

detecting at least one object within an image captured by the separate camera; and

setting the focus window as a defined region having a set of pixels, where the defined region is within an image captured by the separate camera based upon the location of the at least one object;

determining a focus window for the array camera based upon the focus window from the separate camera using the processor, where the focus window for the array camera comprises a corresponding set of pixels that corresponds to the set of pixels within the focus window from the separate camera,

obtaining image data pertaining to the focus window of the array camera from at least two cameras in the plurality of cameras using the processor,

determining depth information for the focus window of the array camera from the image data from the at least two cameras using the processor,

colocating the depth information from the focus window of the array camera to the focus window of the separate camera to account for parallax between the separate camera and the plurality of cameras to generate depth information for the focus window of the separate camera using the processor,

determining a focus depth for the separate camera based upon the depth information for the focus window of the separate camera using the processor, and

adjusting the focus of the separate camera to a desired focus based upon the determined focus depth for the separate camera using the processor.

13. The method of claim 12 wherein the focus depth is determined using a histogram of depths in the depth information for the focus window of the separate camera.

14. The method of claim 12 wherein the focus depth is the closest depth to the separate camera in the depth information for the focus window of the separate camera.

15. The method of claim 14 wherein the focus depth is the closest depth in the depth information of the focus window of the separate camera that includes a number of pixels that is greater than a threshold of a predefined number of pixels.

16. The method of claim 12 further comprising:

determining whether the focus depth for the separate camera can be determined from depth information using the processor; and

determining the focus depth for the separate camera using a conventional autofocus process to determine the focus depth using the processor in response to a determination that the focus depth cannot be determined from depth information.

17. The method of claim 12 wherein obtaining the focus window information for the separate camera comprises:

receiving image data from the separate camera using the processor; and

determining the focus window from the image data from the separate camera using the processor.

18. The method of claim 17 wherein the focus window is determined by performing object detection on the image data and selecting one or more detected objects.

19. The method of claim 18 where the object detection is performed in an area of the image data input by the user.

20. The method of claim 17 wherein the focus window of the separate camera is determined by an input received from the user.

21. The method of claim 12 wherein the colocating of the depth information for the focus window of the array camera to the depth information for the focus window of the separate camera occurs in the absence of depth regularization.

22. The method of claim 12 wherein the focus window of the array camera is determined by mapping the focus window in the separate camera to the focus window in a reference camera of the array camera while accounting for the parallax between the separate camera and the array camera.