DETERMINING A DEPTH MAP FROM IMAGES OF A SCENE

• US 20130063566A1
• Filed: 09/07/2012
• Published: 03/14/2013
• Est. Priority Date: 09/14/2011
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Morgan-Mar, David, Pham, Tuan Quang, Larkin, Kieran Gerard, Arnison, Matthew R.
• Granted Patent: US 9,836,855 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: G06T 2207/10148 Varying focus
• CPC Class Codes: G06T 7/571 from focus

Abstract

A technique determines a depth measurement associated with a scene captured by an image capture device. The technique receives at least first and second images of the scene, in which the first image is captured using at least one different camera parameter than that of the second image. At least first and second image patches are selected from the first and second images, respectively, the selected patches corresponding to a common part of the scene. The selected image patches are used to determine which of the selected image patches provides a more focused representation of the common part. At least one value is calculated based on a combination of data in the first and second image patches, the combination being dependent on the more focused image patch. The depth measurement of the common part of the scene is determined from the at least one calculated value.

Claims

1. A method for determining a depth measurement associated with a scene captured by an image capture device, the method comprising:

receiving a first image and a second image of the scene captured by the image capture device, the first image being captured using at least one different camera parameter than that of the second image;

selecting a first image patch from the first image and a second image patch from the second image, said selected patches corresponding to a common part of the scene captured in each of the first and second images;

determining from the selected image patches which of the selected image patches is a more focused representation of the common part of the scene captured in the patches;

calculating at least one value based on a combination of the data in each of the first and second image patches, said combination being dependent on which of the image patches is determined to be the more focused image patch; and

determining the depth measurement of the common part of the scene from the at least one calculated value.

2. A method according to claim 1, wherein the calculated value is a correlation value based on the correlation between the patch of the first image and the patch of the second image, said correlation being normalised by at least an estimate of the variance in the brightness of the common part the scene represented by the patches in order to calculate the correlation value substantially independent of the scene.

3. A method according to claim 2, wherein the estimate of the variance in the brightness is determined from the patch that is the most focused representation of the scene.

4. A method according to claim 2, wherein the correlation value is calculated within a window function applied to the scene patches.

5. A method according to claim 1, wherein the calculated value is a correlation value based on the correlation between a Laplacian image of the patch of the first image and a Laplacian image of the patch of the second image, said correlation being normalised by at least an estimate of the variance in the brightness of the part the scene captured in the patches in order to calculate the correlation value substantially independent of the scene.

6. A method according to claim 1, wherein the calculated values is at least an estimate of a complex OTF ratio from the complex OTFs corresponding to the first and second images, said complex OTF ratio being a function of the distance to the objects in the common part of the scene.

7. A method according to claim 6, wherein the ratio of the complex OTFs is determined by:

calculating Fourier transforms for each of the patches; and

calculating the ratios of the Fourier transforms to produce an estimate of the ratio of the complex OTFs, with the complex OTF of the more focused patch as the denominator of the ratio.

8. A method according to claim 6, wherein the calculated value is the mean of the OTF ratio over an area.

9. A method according to claim 6, wherein the calculated value is the median of the OTF ratio over an area.

10. A method according to claim 6, wherein the calculated value is the weighted mean of the OTF ratio over an area defined by a weighting function.

11. A method according to claim 10, wherein the weighting function is zero at spatial frequencies greater than the Nyquist limit of the image capture device.

12. A method according to claim 10, wherein the weighting function is zero at spatial frequencies determined by selecting a predetermined fraction of pixels with the lowest modulus of the Fourier transform of one of the input image patches.

13. A method according to claim 6, wherein the ratio of the complex OTFs is determined by:

calculating a Fourier transform for each of the patches; and

calculating the arctangent of the ratios of the moduli of the Fourier transforms, multiplied by a complex phase equal to the difference of the phases of the Fourier transforms, to produce an estimate of the ratio of the complex OTFs, with the complex OTF of the more focused patch as the denominator of the ratio

14. A method according to claim 1, wherein determining which of the image patches is a more focused representation of the part of the scene captured in the patches is done by calculating the variance of the patches and selecting the patch with the highest variance.

15. A method according to claim 1, wherein the selecting of the patches comprises aligning the first image to the second image based on content of the images.

22. A depth measurement formed by the method according to claim 1.

16. A method for determining a depth map associated with a scene captured by an image capture device, the method comprising:

receiving a first image and a second image of the scene captured by the image capture device, the first image being captured using at least one different camera parameter than that of the second image;

determining a depth measurement for a plurality of parts of the scene captured by the images by;

(a) selecting at least a first image patch from the first image and a second image patch from the second image for a current part of the scene, said selected patches in the current part representing a common part of the scene captured in each of the first and second images;

(b) determining from the selected image patches which of the selected image patches is a more focused representation of the current part of the scene;

(c) calculating at least one value based on a combination of the data in each of the first and second image patches, said combination being dependent on which of the image patches is determined to be the more focused image patch;

(d) determining the depth measurement of the current part of the scene from the at least one calculated value; and

(e) repeating steps (a) to (d) using corresponding patches in at least one other part of the common part of then scene represent by the first and second images; and

assembling the determined depth measurements into a depth map of the common part of the scene.

17. A computer readable storage medium having a program recorded thereon, the program being executable by a computerized apparatus to determine a depth measurement associated with a scene captured by an image capture device, the program comprising:

code for receiving a first image and a second image of the scene captured by the image capture device, the first image being captured using at least one different camera parameter than that of the second image;

code for selecting a first image patch from the first image and a second image patch from the second image, said selected patches corresponding to a common part of the scene captured in each of the first and second images;

code for determining from the selected image patches which of the selected image patches is a more focused representation of the common part of the scene captured in the patches;

code for calculating at least one value based on a combination of the data in each of the first and second image patches, said combination being dependent on which of the image patches is determined to be the more focused image patch; and

code for determining the depth measurement of the common part of the scene from the at least one calculated value.

18. Computerized apparatus for determining a depth measurement associated with a scene captured by an image capture device, the apparatus comprising:

means for receiving a first image and a second image of the scene captured by the image capture device, the first image being captured using at least one different capture parameter than that of the second image;

means for selecting a first image patch from the first image and a second image patch from the second image, said selected patches corresponding to a common part of the scene captured in each of the first and second images;

means for determining from the selected image patches which of the selected image patches is a more focused representation of the common part of the scene captured in the patches;

means for calculating at least one value based on a combination of the data in each of the first and second image patches, said combination being dependent on which of the image patches is determined to be the more focused image patch;

means for determining the depth measurement of the common part of the scene from the at least one calculated value.

19. An image capture device, comprising:

a capture system for receiving a first image and a second image of the scene captured by the image capture device, the first image being captured using at least one different capture parameter than that of the second image;

a memory into which the first image and the second image are stored;

a processor configured for determining a depth measurement associated with a scene captured by the first image and the second image, the processor being configured to;

select a first image patch from the first image and a second image patch from the second image, said selected patches corresponding to a common part of the scene captured in each of the first and second images;

determine from the selected image patches which of the selected image patches is a more focused representation of the common part of the scene captured in the patches;

calculate at least one value based on a combination of the data in each of the first and second image patches, said combination being dependent on which of the image patches is determined to be the more focused image patch; and

determine the depth measurement of the common part of the scene from the at least one calculated value.

20. An image capture device comprising:

a capture system for receiving a first image and a second image of the scene captured by the image capture device, the first image being captured using at least one different capture parameter than that of the second image;

a memory into which the first image and the second image are stored;

a processor coupled to the memory and configured to determine a depth map associated with the scene captured with the images, said processor being operable to;

determine a depth measurement for a plurality of parts of the scene captured by the images by;

(a) selecting a first image patch from the first image and a second image patch from the second image for a current part of the scene, said selected patches in the current part representing a common part of the scene captured in each of the first and second images;

(b) determining from the selected image patches which of the selected image patches is a more focused representation of the current part of the scene;

(c) calculating at least one value based on a combination of the data in each of the first and second image patches, said combination being dependent on which of the image patches is determined to be the more focused image patch;

(d) determining the depth measurement of the current part of the scene from the at least one calculated value; and

(e) repeating steps (a) to (d) using corresponding patches in at least one other part of the common part of then scene represent by the first and second images; and

assemble the depth measurements into a depth map of the common part of the scene.

21. A method for determining a depth measurement associated with a scene captured by an image capture device, said method comprising:

receiving a first image and a second image of the scene captured by the image capture device, the first image being captured using at least one different camera parameter than that of the second image;

selecting a first image patch from the first image and a second image patch from the second image, said selected patches corresponding to a common part of the scene captured in each of the first and second images;

calculating at least one value based on a combination of the data in each of the first and second image patches, wherein the at least one value is an estimate of a complex OTF ratio from the complex OTFs corresponding to the first and second images, said complex OTF ratio being a function of the distance to the objects in the common part of the scene; and

determining the depth measurement of the common part of the scene from the at least one calculated value.