Systems and methods for estimating depth using ad hoc stereo array cameras

• US 9,602,805 B2
• Filed: 05/06/2015
• Issued: 03/21/2017
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: Fotonation Cayman Limited
• Inventors: Venkataraman, Kartik, Gallagher, Paul, Jain, Ankit, Nisenzon, Semyon
• Primary Examiner(s): Le, Peter D
• Application Number:
  US14/705,903
• Publication Number:
  US 20150237329A1
• Time in Patent Office: 685 Days
• US Class Current: 1/1
• CPC Class Codes: G01P 3/38 using photographic means
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors

Abstract

Systems and methods for stereo imaging with camera arrays in accordance with embodiments of the invention are disclosed. In one embodiment, a method of generating depth information for an object using two or more array cameras that each include a plurality of imagers includes obtaining a first set of image data captured from a first set of viewpoints, identifying an object in the first set of image data, determining a first depth measurement, determining whether the first depth measurement is above a threshold, and when the depth is above the threshold: obtaining a second set of image data of the same scene from a second set of viewpoints located known distances from one viewpoint in the first set of viewpoints, identifying the object in the second set of image data, and determining a second depth measurement using the first set of image data and the second set of image data.

Claims

1. An array camera, comprising:

a camera array comprising a plurality of cameras that capture images of a scene from different viewpoints;

a gyroscope and accelerometers;

a processor; and

memory in communication with the processor;

wherein software directs the processor to;

obtain a first set of images captured from different viewpoints at a first time using the camera array at a first location;

obtain a second set of images captured from different viewpoints at a second time using the camera array at a second location distant from the first location;

estimate extrinsic parameters for the camera array at the first time and at the second time based upon a combination of matching features in the first and second set of images and information from the gyroscope and accelerometers;

select a reference viewpoint;

determine depth estimates for pixel locations in an image from the reference viewpoint using the first set of images, wherein generating a depth estimate for a given pixel location in the image from the reference viewpoint comprises;

identifying pixels in the first set of images that correspond to the given pixel location in the image from the reference viewpoint based upon expected disparity at a plurality of depths;

comparing the similarity of the corresponding pixels identified at each of the plurality of depths; and

selecting the depth from the plurality of depths at which the identified corresponding pixels have the highest degree of similarity as a depth estimate for the given pixel location in the image from the reference viewpoint;

determine whether a depth estimate for pixel locations in an image from the reference viewpoint determined using the first set of images corresponds to an observed disparity below a threshold; and

when the depth estimate corresponds to an observed disparity below the threshold, refining the depth estimate using images in the second set of images by;

identifying pixels in the first and second sets of images that correspond to the given pixel location in the image from the reference viewpoint based upon expected disparity at a plurality of depths;

comparing the similarity of the corresponding pixels form the first and second sets of images identified at each of the plurality of depths; and

selecting the depth from the plurality of depths at which the identified corresponding pixels have the highest degree of similarity as a depth estimate for the given pixel location in the image from the reference viewpoint.

2. The array camera of claim 1, wherein software further directs the processor to select the plurality of depths at which pixels in images captured by the first array camera and by the second camera that correspond to the given pixel location in the image from the reference viewpoint are identified during refinement of the depth estimate based upon the depth estimate initially determined using the first set of images.

3. The array camera of claim 1, wherein software further directs the processor to:

identify an object of interest in the first set of images;

wherein determine depth estimates for pixel locations in an image from the reference viewpoint using the first set of images further comprises estimating a depth for the object of interest using the first set of images.

4. The array camera of claim 1, wherein software further directs the processor to calculate a confidence measure for the depth of an object of interest.

5. The array camera of claim 1, wherein software further directs the processor to generate a depth map using the depth estimates for pixel locations in an image from the reference viewpoint, where the depth map indicates distances of surfaces of scene objects from the reference viewpoint.

6. The array camera of claim 1, wherein the plurality of cameras in the camera array forms an M×

N array of cameras.

7. The array camera of claim 1, wherein the camera array comprises cameras that image portions of the spectral band selected from the group consisting of red, blue, green, infrared, and extended color.

8. The array camera of claim 1, wherein the cameras in the camera array form a π

filter group.

9. The array camera of claim 1, wherein:

software further configures the processor to synthesize an image from the first set of images using the depth map;

each image in the first set of images has a resolution; and

the image synthesized from the first set of images has a resolution that is greater than the resolution of each of the images in the first set of images.

10. The array camera of claim 1, wherein:

software further configures the processor to synthesize an image from the first and second sets of images using the depth map and the extrinsic parameters of the camera array at the first and second times;

each image in the first and second sets of images has a resolution; and

the image synthesized from the first set of images has a resolution that is greater than the resolution of each of the images in the first and second sets of images.

11. A method of estimating depth using an ad hoc stereo array camera, comprising:

obtaining a first set of images captured from different viewpoints at a first time using a camera array;

moving the camera array and obtaining a second set of images captured from different viewpoints at a second time using the camera array;

obtaining gyroscope and accelerometer measurements between the first time and the second time;

estimating extrinsic parameters for the camera array at the first time and at the second time based upon a combination of matching features in the first and second set of images and the gyroscope and accelerometers measurements;

using software to direct a processor to;

select a reference viewpoint;

determine depth estimates for pixel locations in an image from the reference viewpoint using the first set of images, wherein generating a depth estimate for a given pixel location in the image from the reference viewpoint comprises;

identifying pixels in the first set of images that correspond to the given pixel location in the image from the reference viewpoint based upon expected disparity at a plurality of depths;

comparing the similarity of the corresponding pixels identified at each of the plurality of depths; and

selecting the depth from the plurality of depths at which the identified corresponding pixels have the highest degree of similarity as a depth estimate for the given pixel location in the image from the reference viewpoint;

determine whether a depth estimate for pixel locations in an image from the reference viewpoint determined using the first set of images corresponds to an observed disparity below a threshold; and

when the depth estimate corresponds to an observed disparity below the threshold, refining the depth estimate using images in the second set of images by;

identifying pixels in the first and second sets of images that correspond to the given pixel location in the image from the reference viewpoint based upon expected disparity at a plurality of depths;

comparing the similarity of the corresponding pixels from the first and second sets of images identified at each of the plurality of depths; and

selecting the depth from the plurality of depths at which the identified corresponding pixels have the highest degree of similarity as a depth estimate for the given pixel location in the image from the reference viewpoint.

12. The method of claim 11, further comprising using software to direct the processor to select the plurality of depths at which pixels in images captured by the first array camera and by the second camera that correspond to the given pixel location in the image from the reference viewpoint are identified during refinement of the depth estimatebased upon the depth estimate initially determined using the first set of images.

13. The method of claim 11, further comprising using software to direct the processor to:

identify an object of interest in the first set of images;

wherein determine depth estimates for pixel locations in an image from the reference viewpoint using the first set of images further comprises estimating a depth for the object of interest using the first set of images.

14. The method of claim 11, further comprising using software to direct the processor to calculate a confidence measure for the depth of an object of interest.

15. The method of claim 11, further comprising using software to direct the processor to generate a depth map using the depth estimates for pixel locations in an image from the reference viewpoint, where the depth map indicates distances of surfaces of scene objects from the reference viewpoint.

16. The method of claim 11, wherein the plurality of cameras in the camera array forms an M×

N array of cameras.

17. The method of claim 11, wherein the camera array comprises cameras that image portions of the spectral band selected from the group consisting of red, blue, green, infrared, and extended color.

18. The method of claim 11, wherein the cameras in the camera array form a π

filter group.

19. The method of claim 11, further comprising using software to direct the processor to synthesize an image from the first set of images using the depth map;

wherein each image in the first set of images has a resolution; and

wherein the image synthesized from the first set of images has a resolution that is greater than the resolution of each of the images in the first set of images.

20. The method of claim 11, further comprising using software to direct the processor to synthesize an image from the first and second sets of images using the depth map and the extrinsic parameters of the camera array at the first and second times;

wherein each image in the first and second sets of images has a resolution; and

wherein the image synthesized from the first and second sets of images has a resolution that is greater than the resolution of each of the images in the first and second sets of images.