Determining a depth map from images of a scene

• US 9,836,855 B2
• Filed: 09/07/2012
• Issued: 12/05/2017
• Est. Priority Date: 09/14/2011
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Morgan-Mar, David, Pham, Tuan Quang, Arnison, Matthew R, Larkin, Kieran Gerard
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): Walsh, Kathleen
• Application Number:
  US13/606,822
• Publication Number:
  US 20130063566A1
• Time in Patent Office: 1,915 Days
• Field of Search: 348 46, 348 49, 348345, 348335, 34820799, 348 65, 348 77, 348E13074, 348234, 348241, 348103, 348 E5024, 382128, 382299, 375285, 375296, 375346
• CPC Class Codes: G06T 2207/10148 Varying focus
• CPC Class Codes: G06T 7/571 from focus

Abstract

A technique determines a depth measurement associated with a scene captured by an image capture device. The technique receives at least first and second images of the scene, in which the first image is captured using at least one different camera parameter than that of the second image. At least first and second image patches are selected from the first and second images, respectively, the selected patches corresponding to a common part of the scene. The selected image patches are used to determine which of the selected image patches provides a more focused representation of the common part. At least one value is calculated based on a combination of data in the first and second image patches, the combination being dependent on the more focused image patch. The depth measurement of the common part of the scene is determined from the at least one calculated value.

Claims

1. A method for determining a depth measurement associated with a scene captured by an image capture device, the method comprising:

receiving a first image and a second image of the scene captured by the image capture device, the first image being captured using at least one different camera parameter than that of the second image;

selecting a first patch from the first image;

selecting a second patch from the second image to correspond with the first patch such that the first and second patches represent a common part of the scene;

determining which of the first and second patches is more focused;

calculating Fourier transforms for each of the first and second patches; and

calculating a ratio of the Fourier transforms corresponding to the first and second patches to produce an estimate of a complex OTF ratio, with the Fourier transform of the more focused patch as denominator of the ratio, the estimate of the complex OTF ratio representing relative blur of the first and second patches and being a function of distance to the objects in the common part of the scene; and

determining the depth measurement of the common part of the scene from the estimate of the complex OTF ratio.

2. The method according to claim 1, wherein the estimate of the complex OTF ratio is a mean of the OTF ratio over an area.

3. The method according to claim 1, wherein the estimate of the complex OTF ratio is a median of the OTF ratio over an area.

4. The method according to claim 1, wherein the estimate of the complex OTF ratio is a weighted mean of the OTF ratio over an area defined by a weighting function.

5. The method according to claim 4, wherein the weighting function is zero at spatial frequencies greater than a Nyquist limit of the image capture device.

6. The method according to claim 4, wherein the weighting function is zero at spatial frequencies determined by selecting a predetermined fraction of pixels with a lower modulus of the Fourier transform of one of the first and second patches.

7. The method according to claim 1, wherein the ratio of the complex OTFs is determined by:

calculating an arc tangent of ratios of moduli of the Fourier transforms, multiplied by a complex phase equal to a difference of phases of the Fourier transforms, to produce an estimate of the ratio of the complex OTFs, with the complex OTF of the more focused patch as a denominator of the ratio.

8. The method according to claim 1, wherein determining which of the first and second patches is the more focused patch is done by calculating a variance of each of the first and second patches and selecting the more focused patch as one with a higher variance.

9. The method according to claim 1, wherein the selecting of the first and second patches comprises aligning the first image to the second image based on content of the images.

15. A depth measurement formed by the method according to claim 1.

16. A method according to claim 1, wherein the step of calculating the ratio of Fourier transforms comprises:

determining first and second zero mean (ZM) patches from the first and second patches, respectively; and

calculating the ratio of the Fourier transforms for the first and second ZM patches to produce an estimate of a complex OTF ratio, with the Fourier transform of the more focused patch as denominator of the ratio.

10. A method for determining a depth map associated with a scene captured by an image capture device, the method comprising:

receiving a first image and a second image of the scene captured by the image capture device, the first image being captured using at least one different camera parameter than that of the second image;

determining a depth measurement for a plurality of parts of the scene by;

(a) (i) selecting at least a first patch from the first image;

(ii) selecting a second patch from the second image to correspond with the first patch for a current part of the scene such that the first and second patches in the current part represent a common part of the scene;

(b) determining which of the first and second patches is a more focused patch;

(c0) calculating Fourier transforms for each of the first and second patches;

(c) calculating a ratio of the Fourier transforms corresponding to the first and second patches to produce an estimate of a complex OTF ratio, with the Fourier transform of the more focused patch as denominator of the ratio, the estimate of the complex OTF ratio representing relative blur of the first and second patches and being a function of distance to the objects in the common part of the scene;

(d) determining the depth measurement of the current part of the scene from the estimate of the complex OTF ratio; and

(e) repeating steps (a) to (d) using corresponding patches in at least one other part of the common part of the scene represented by the first and second images; and

assembling the determined depth measurements into a depth map of the common part of the scene.

11. A computer readable non-transitory storage medium having a program recorded thereon, the program being executable by a computerized apparatus to determine a depth measurement associated with a scene captured by an image capture device, the program comprising:

code for receiving a first image and a second image of the scene captured by the image capture device, the first image being captured using at least one different camera parameter than that of the second image;

code for selecting a first patch from the first image;

code for selecting a second patch corresponding to the first patch from the second image such that the first and second patches represent a common part of the scene;

code for determining which of the first and second patches is a more focused patch;

code for calculating Fourier transforms for each of the first and second patches;

code for calculating a ratio of the Fourier transforms corresponding to the first and second patches to produce an estimate of a complex OTF ratio, with the Fourier transform of the more focused patch as denominator of the ratio, the estimate of the complex OTF ratio representing relative blur of the first and second patches and being a function of distance to the objects in the common part of the scene; and

code for determining the depth measurement of the common part of the scene from the estimate of the complex OTF ratio.

12. An apparatus for determining a depth measurement associated with a scene captured by an image capture device, the apparatus comprising:

means for receiving a first image and a second image of the scene captured by the image capture device, the first image being captured using at least one different capture parameter than that of the second image;

means for selecting a first patch from the first image;

means for selecting a second patch corresponding to the first patch from the second image such that the first and second patches represent a common part of the scene;

means for determining which of the first and second patches is a more focused patch;

means for calculating Fourier transforms for each of the first and second patches;

means for calculating a ratio of the Fourier transforms corresponding to the first and second patches to produce an estimate of a complex OTF ratio, with the Fourier transform of the more focused patch as denominator of the ratio, the estimate of the complex OTF ratio representing relative blur of the first and second patches and being a function of distance to the objects in the common part of the scene;

means for determining the depth measurement of the common part of the scene from the estimate of the complex OTF ratio.

13. An image capture device, comprising:

a capture system for receiving a first image and a second image of the scene captured by the image capture device, the first image being captured using at least one different capture parameter than that of the second image;

a memory into which the first image and the second image are stored;

a processor configured for determining a depth measurement associated with a scene captured by the first image and the second image, the processor being configured to;

select a first patch from the first image;

select a second patch corresponding to the first patch from the second image such that the first and second patches represent a common part of the scene;

determine which of the first and second patches is a more focused patch;

calculate Fourier transforms for each of the first and second patches;

calculate a ratio of the Fourier transforms corresponding to the first and second patches to produce an estimate of a complex OTF ratio, with the Fourier transform of the more focused patch as denominator of the ratio, the estimate of the complex OTF ratio representing relative blur of the first and second patches and being a function of distance to the objects in the common part of the scene; and

determine the depth measurement of the common part of the scene from the estimate of the complex OTF ratio.

14. An image capture device comprising:

a capture system for receiving a first image and a second image of the scene captured by the image capture device, the first image being captured using at least one different capture parameter than that of the second image;

a memory into which the first image and the second image are stored;

a processor coupled to the memory and configured to determine a depth map associated with the scene captured with the images, the processor being operable to;

determine a depth measurement for a plurality of parts of the scene captured by the images by;

(a) (i) selecting a first patch from the first image;

(ii) selecting a second patch corresponding to the first patch from the second image for a current part of the scene such that the first and second patches in the current part represent a common part of the scene;

(b) determining which of the first and second patches is a more focused patch;

(c0) calculating Fourier transforms for each of the first and second patches;

(c) calculating a ratio of the Fourier transforms corresponding to the first and second patches to produce an estimate of a complex OTF ratio, with the Fourier transform of the more focused patch as denominator of the ratio, the estimate of the complex OTF ratio representing relative blur of the first and second patches and being a function of distance to the objects in the common part of the scene;

(d) determining the depth measurement of the current part of the scene from the estimate of the complex OTF ratio; and

(e) repeating steps (a) to (d) using corresponding patches in at least one other part of the common part of the scene; and

assemble the depth measurements into a depth map of the common part of the scene.