Systems and Methods for Stereo Imaging with Camera Arrays

• US 20140267633A1
• Filed: 03/17/2014
• Published: 09/18/2014
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: Pelican Imaging Corporation
• Inventors: Jain, Ankit, Venkataraman, Kartik, Gallagher, Paul, Nisenzon, Semyon
• Granted Patent: US 9,438,888 B2
• Time in Patent Office: Days
• US Class Current: 348/48
• CPC Class Codes: G01P 3/38 using photographic means
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors

Abstract

Systems and methods for stereo imaging with camera arrays in accordance with embodiments of the invention are disclosed. In one embodiment, a method of generating depth information for an object using two or more array cameras that each include a plurality of imagers includes obtaining a first set of image data captured from a first set of viewpoints, identifying an object in the first set of image data, determining a first depth measurement, determining whether the first depth measurement is above a threshold, and when the depth is above the threshold: obtaining a second set of image data of the same scene from a second set of viewpoints located known distances from one viewpoint in the first set of viewpoints, identifying the object in the second set of image data, and determining a second depth measurement using the first set of image data and the second set of image data.

Claims

1. A method of generating depth information for an object in a scene using two or more array cameras that each include a plurality of imagers where each imager captures an image of the scene, the method comprising:

obtaining a first set of image data comprising image data of a scene captured from a first set of different viewpoints;

identifying an object of interest in the first set of image data;

determining a first depth measurement for the object of interest using at least a portion of the first set of image data;

determining whether the first depth measurement for the object of interest is above a predetermined threshold; and

when the depth for the object of interest is above the predetermined threshold;

obtaining a second set of image data comprising image data of the same scene from a second set of different viewpoints located known distances from at least one viewpoint in the first set of different viewpoints;

identifying the object of interest in the second set of image data; and

determining a second depth measurement for the object of interest using at least a portion of the first set of image data and at least a portion of the second set of image data.

2. The method of claim 1, wherein obtaining a first set of image data comprising image data of a scene captured from a first set of different viewpoints comprises capturing a first set of image data using a first plurality of imagers in a first array.

3. The method of claim 2, further comprising determining image capture settings for active imagers in the first array.

4. The method of claim 3, wherein determining image capture settings for active imagers in the first array comprises calibrating for nonlinearities in the lenses of a plurality of the first plurality of imagers.

5. The method of claim 2, wherein one of the imagers is designated as a reference camera and captures image data from a reference viewpoint.

6. The method of claim 2, wherein identifying an object of interest in the first set of image data comprises generating a preview image, presenting the preview image via a screen, and capturing user input.

7. The method of claim 2, wherein identifying an object of interest in the first set of image data comprises detecting motion in an area of the scene.

8. The method of claim 2, wherein identifying an object of interest in the first set of image data comprises detecting an object in a designated region of interest.

9. The method of claim 2, wherein identifying an object of interest in the first set of image data comprises selecting one or more pixels and associating the one or more pixels with the object.

10. The method of claim 2, wherein obtaining a second set of image data comprising image data of the same scene from a second set of different viewpoints located known distances from at least one viewpoint in the first set of different viewpoints comprises capturing a second set of image data using a second plurality of imagers in a second array.

11. The method of claim 10, wherein the second array is the first array placed in a different location from the location in which the first array captured the first set of image data.

12. The method of claim 11, further comprising estimating the baseline distance between the two arrays by cross-correlating one or more sets of corresponding pixels from the first set of image data and the second set of image data.

13. The method of claim 1, wherein determining a first depth measurement for the object of interest using at least a portion of the first set of image data comprises determining the disparity between the location of a pixel in one image in the first set of image data and a corresponding pixel in a second image in the first set of image data.

14. The method of claim 1, further comprising calculating a confidence measure for the depth of the object of interest.

15. The method of claim 1, further comprising determining if the object of interest is visible in the second set of image data and identifying the object of interest in the second set of image data when the object of interest is visible in the second set of image data.

16. The method of claim 1, wherein determining a second depth measurement for the object of interest using at least a portion of the first set of image data and at least a portion of the second set of image data comprises determining the disparity between a first pixel associated with the object of interest in at least one image in the first set of image data and a corresponding second pixel in at least one image in the second set of image data.

17. The method of claim 1, wherein determining a second depth measurement for the object of interest using at least a portion of the first set of image data and at least a portion of the second set of image data comprises calculating the disparity between the location of a pixel in one image in the first set of image data and a corresponding pixel in a second image in the second set of image data.

18. The method of claim 1, wherein determining a second depth measurement for the object of interest using at least a portion of the first set of image data and at least a portion of the second set of image data comprises utilizing the first depth measurement for the object of interest.

19. A method for calculating the speed of an object in a scene using one or more array cameras that each include a plurality of imagers where each imager captures an image of the scene, the method comprising:

obtaining a first set of image data comprising image data of a scene captured from a first set of different viewpoints;

identifying an object of interest in the first set of image data;

determining a first depth measurement and a first angular measurement for the object of interest using at least a portion of the first set of image data;

determining a first location of the object of interest using at least the first depth measurement and first angular measurement;

obtaining a second set of image data comprising a second image data of a scene captured from a second set of different viewpoints at a time t after the first set of image data was captured;

identifying the object of interest in the second set of image data;

determining a second depth measurement and a second angular measurement for the object of interest using at least a portion of the second set of image data;

determining a second location of the object of interest using at least the second depth measurement and second angular measurement;

calculating a speed for the object of interest using at least the first location and the second location of the object of interest.

20. The method of claim 19, wherein:

obtaining a first set of image data comprising image data of a scene captured from a first set of different viewpoints comprises capturing a first set of image data using a first plurality of imagers in a first array; and

obtaining a second set of image data comprising a second image data of a scene captured from a second set of different viewpoints at a time t after the first set of image data was captured comprises capturing a second set of image data using a second plurality of imagers at a time t after the first set of image data was captured.

21. The method of claim 20, wherein the second plurality of imagers is the same as the first plurality of imagers and the second set of different viewpoints is the same as the first set of different viewpoints.

22. The method of claim 20, wherein the second plurality of imagers is selected from a second array that is different from the first array.

23. The method of claim 20, further comprising determining image capture settings for active imagers in the first array.

24. The method of claim 20, wherein identifying an object of interest in the first set of image data comprises generating a preview image, presenting the preview image via a screen, and capturing user input.

25. The method of claim 20, wherein identifying an object of interest in the first set of image data comprises detecting motion in an area of the scene.

26. The method of claim 20, wherein identifying an object of interest in the first set of image data comprises detecting an object in a designated region of interest.

27. The method of claim 20, wherein identifying an object of interest in the first set of image data comprises selecting one or more pixels and associating the one or more pixels with the object.

28. The method of claim 20, wherein:

capturing a first set of image data using a first plurality of imagers in a first array comprises capturing a first set of image data at a first location;

capturing a second set of image data using a second plurality of imagers at a time t after the first set of image data was captured comprises capturing a second set of image data in a second location at a time t after the first set of image data was captured; and

determining a second location of the object of interest using at least the second depth measurement and second angular measurement comprises determining the change from the first location to the second location.

29. The method of claim 20, wherein:

capturing a first set of image data using a first plurality of imagers in a first array comprises determining a first orientation of the first array;

capturing a second set of image data using a second plurality of imagers at a time t after the first set of image data was captured comprises determining a second orientation of the first array at a time t after the first set of image data was captured; and

determining a second location of the object of interest using at least the second depth measurement and second angular measurement comprises determining the change from the first orientation to the second orientation.

30. The method of claim 19, further comprising calculating a confidence measure of the speed of the object of interest.

31. The method of claim 19, wherein determining a second depth measurement and a second angular measurement for the object of interest using at least a portion of the second set of image data comprises determining a second depth measurement for the object of interest using at least a portion of the first set of image data and at least a portion of the second set of image data.