Refinement of depth maps by fusion of multiple estimates

• US 9,117,295 B2
• Filed: 12/20/2011
• Issued: 08/25/2015
• Est. Priority Date: 12/20/2011
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Rastogi, Anubha, Krishnamurthy, Balaji
• Primary Examiner(s): Koziol, Stephen R
• Assistant Examiner(s): HILL, SCHILLER D
• Application Number:
  US13/332,139
• Publication Number:
  US 20130155050A1
• Time in Patent Office: 1,344 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 15/40 Hidden part removal
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 25/63 applied to dark current

Abstract

A method, system, and computer-readable storage medium are disclosed for generating fused depth maps. A plurality of initial depth maps are generated for a first image of a stereo pair, using at least two separate and distinct techniques for depth estimation. A plurality of initial depth maps are generated for a second image of the stereo pair, using at least two separate and distinct techniques for depth estimation. A fused depth map is generated for the first image based on the plurality of initial depth maps for the first image and the plurality of initial depth maps for the second image. A fused depth map is generated for the second image based on the plurality of initial depth maps for the second image and the plurality of initial depth maps for the first image.

Claims

1. A computer-implemented method, comprising:

generating a plurality of initial depth maps for a first image of a stereo pair such that a given pixel p in the first image is associated with a first plurality of depth estimates, wherein the plurality of initial depth maps for the first image are generated using at least two separate and distinct techniques for depth estimation;

generating a plurality of initial depth maps for a second image of the stereo pair such that a given pixel in the second image is associated with a second plurality of depth estimates, wherein the plurality of initial depth maps for the second image are generated using at least two separate and distinct techniques for depth estimation;

generating a fused depth map for the first image that comprises a first plurality of fused depth estimates,

wherein a given one of the first plurality of fused depth estimates that corresponds to the given pixel p in the first image is derived from the first plurality of depth estimates,

wherein the given one of the first plurality of fused depth estimates is derived based on a smoothness parameter S(p) that represents a fraction of pixels in a neighborhood N(p) of the given pixel p in the first image that fall within (a) a predetermined depth proximity threshold δ and

(b) a predetermined color proximity threshold τ

, and

wherein the given one of the first plurality of fused depth estimates is also derived based on a cross-validation parameter that indicates whether any of the first plurality of depth estimates is cross-validated by any of the second plurality of depth estimates; and

generating a fused depth map for the second image that comprises the second plurality of fused depth estimates,

wherein a given one of the second plurality of fused depth estimates that corresponds to the given pixel in the second image is derived from the second plurality of depth estimates,

wherein the given one of the second plurality of fused depth estimates is derived based on a smoothness parameter that represents a fraction of pixels in a neighborhood of the given pixel in the second image that exhibit the relatively small variation in depth and the relatively small variation in color, and

wherein the given one of the second plurality of fused depth estimates is also derived based on a cross-validation parameter that indicates whether any of the second plurality of depth estimates is cross-validated by any of the first plurality of depth estimates.

2. The method as recited in claim 1,

wherein generating the fused depth map for the first image comprises:

generating a reliability value for each of the first plurality of depth estimates, wherein a reliability value for a particular one of the first plurality of depth estimates depends on smoothness and cross-validation parameters for that depth estimate; and

wherein generating the fused depth map for the second image comprises;

generating a reliability value for each of the second plurality of depth estimates, wherein a reliability value for a particular one of the second plurality of depth estimates depends on smoothness and cross-validation parameters for that depth estimate.

3. The method as recited in claim 1, further comprising:

assigning a weight to each of the first plurality of depth estimates for the first image; and

assigning a weight to each of the second plurality of depth estimates for the second image.

4. The method as recited in claim 3, wherein the weight for each depth estimate is determined based on the smoothness and cross-validation parameters.

5. The method as recited in claim 4,

wherein generating the fused depth map for the first image comprises:

for each pixel in the fused depth map for the first image, selecting a depth estimate having a highest weight; and

wherein generating the fused depth map for the second image comprises;

for each pixel in the fused depth map for the second image, selecting a depth estimate having a highest weight.

6. The method of claim 1, wherein:

the smoothness parameter S(p) of the given pixel p in the first image is given by

7. The method of claim 1, wherein:

the cross-validation parameter associated with an ith one of the first plurality of depth estimates d₁ⁱ associated with the given pixel p in the first image is given by

8. A system, comprising:

at least one processor; and

a memory coupled to the at least one processor, wherein the memory stores program instructions, wherein the program instructions are executable by the at least one processor to;

generate a plurality of initial depth maps for a first image of a stereo pair such that a given pixel p in the first image is associated with a first plurality of depth estimates, wherein the plurality of initial depth maps for the first image are generated using at least two separate and distinct techniques for depth estimation;

generate a plurality of initial depth maps for a second image of the stereo pair such that a given pixel in the second image is associated with a second plurality of depth estimates, wherein the plurality of initial depth maps for the second image are generated using at least two separate and distinct techniques for depth estimation;

generate a fused depth map for the first image that comprises a first plurality of fused depth estimates,

wherein a given one of the first plurality of fused depth estimates that corresponds to the given pixel p in the first image is derived from the first plurality of depth estimates,

wherein the given one of the first plurality of fused depth estimates is derived based on a smoothness parameter S(p) that represents a fraction of pixels in a neighborhood N(p) of the given pixel p in the first image that fall within (a) a predetermined depth proximity threshold δ and

(b) a predetermined color proximity threshold τ

, and

wherein the given one of the first plurality of fused depth estimates is also derived based on a cross-validation parameter that indicates whether any of the first plurality of depth estimates is cross-validated by any of the second plurality of depth estimates; and

generate a fused depth map for the second image that comprises the second plurality of fused depth estimates,

wherein a given one of the second plurality of fused depth estimates that corresponds to the given pixel in the second image is derived from the second plurality of depth estimates,

wherein the given one of the second plurality of fused depth estimates is derived based on a smoothness parameter that represents a fraction of pixels in a neighborhood of the given pixel in the second image that exhibit the relatively small variation in depth and the relatively small variation in color, and

wherein the given one of the second plurality of fused depth estimates is also derived based on a cross-validation parameter that indicates whether any of the second plurality of depth estimates is cross-validated by any of the first plurality of depth estimates.

9. The system as recited in claim 8,

wherein, in generating the fused depth map for the first image, the program instructions are executable by the at least one processor to:

generate a reliability value for each of the first plurality of depth estimates, wherein a reliability value for a particular one of the first plurality of depth estimates depends on smoothness and cross-validation parameters for that depth estimate; and

wherein, in generating the fused depth map for the second image, the program instructions are executable by the at least one processor to;

generate a reliability value for each of the second plurality of depth estimates, wherein a reliability value for a particular one of the second plurality of depth estimates depends on smoothness and cross-validation parameters for that depth estimate.

10. The system as recited in claim 8, wherein the program instructions are executable by the at least one processor to:

assign a weight to each of the first plurality of depth estimates for the first image; and

assign a weight to each of the second plurality of depth estimates for the second image.

11. The system as recited in claim 10, wherein the weight for each depth estimate is determined based on the smoothness and cross-validation parameters.

12. The system as recited in claim 11,

wherein, in generating the fused depth map for the first image, the program instructions are executable by the at least one processor to:

select a depth estimate having a highest weight for each pixel in the fused depth map for the first image; and

wherein, in generating the fused depth map for the second image, the program instructions are executable by the at least one processor to;

select a depth estimate having a highest weight for each pixel in the fused depth map for the second image.

13. The system of claim 8, wherein:

the smoothness parameter S(p) of the given pixel p in the first image is given by

14. The system of claim 8, wherein:

the cross-validation parameter associated with an ith one of the first plurality of depth estimates d₁ⁱ associated with the given pixel p in the first image is given by

15. A non-transitory computer-readable storage medium storing program instructions computer-executable to perform:

generating a plurality of initial depth maps for a first image of a stereo pair such that a given pixel p in the first image is associated with a first plurality of depth estimates, wherein the plurality of initial depth maps for the first image are generated using at least two separate and distinct techniques for depth estimation;

generating a plurality of initial depth maps for a second image of the stereo pair such that a given pixel in the second image is associated with a second plurality of depth estimates, wherein the plurality of initial depth maps for the second image are generated using at least two separate and distinct techniques for depth estimation;

generating a fused depth map for the first image that comprises a first plurality of fused depth estimates,

wherein a given one of the first plurality of fused depth estimates that corresponds to the given pixel p in the first image is derived from the first plurality of depth estimates,

wherein the given one of the first plurality of fused depth estimates is derived based on a smoothness parameter S(p) that represents a fraction of pixels in a neighborhood N(p) of the given pixel p in the first image that fall within (a) a predetermined depth proximity threshold δ and

(b) a predetermined color proximity threshold τ

, and

wherein the given one of the first plurality of fused depth estimates is also derived based on a cross-validation parameter that indicates

whether any of the first plurality of depth estimates is cross-validated by any of the second plurality of depth estimates; and

generating a fused depth map for the second image that comprises the second plurality of fused depth estimates,

wherein a given one of the second plurality of fused depth estimates that corresponds to the given pixel in the second image is derived from the second plurality of depth estimates,

wherein the given one of the second plurality of fused depth estimates is derived based on a smoothness parameter that represents a fraction of pixels in a neighborhood of the given pixel in the second image that exhibit the relatively small variation in depth and the relatively small variation in color, and

wherein the given one of the second plurality of fused depth estimates is also derived based on a cross-validation parameter that indicates whether any of the second plurality of depth estimates is cross-validated by any of the first plurality of depth estimates.

16. The non-transitory computer-readable storage medium as recited in claim 15,

wherein generating the fused depth map for the first image comprises:

generating a reliability value for each of the first plurality of depth estimates, wherein a reliability value for a particular one of the first plurality of depth estimates depends on smoothness and cross-validation parameters for that depth estimate; and

wherein generating the fused depth map for the second image comprises;

generating a reliability value for each of the second plurality of depth estimates, wherein a reliability value for a particular one of the second plurality of depth estimates depends on smoothness and cross-validation parameters for that depth estimate.

17. The non-transitory computer-readable storage medium as recited in claim 15, wherein the program instructions are computer-executable to further perform:

assigning a weight to each of the first plurality of depth estimates for the first image; and

assigning a weight to each of the second plurality of depth estimates for the second image;

wherein each of the weights is determined based on smoothness and cross-validation parameters.

18. The non-transitory computer-readable storage medium as recited in claim 17,

wherein generating the fused depth map for the first image comprises:

for each pixel in the fused depth map for the first image, selecting a depth estimate having a highest weight; and

wherein generating the fused depth map for the second image comprises;

for each pixel in the fused depth map for the second image, selecting a depth estimate having a highest weight.

19. The non-transitory computer-readable storage medium of claim 15, wherein

the smoothness parameter S(p) of the given pixel p in the first image is given by

20. The non-transitory computer-readable storage medium of claim 15, wherein

the cross-validation parameter associated with an ith one of the first plurality of depth estimates d₁ⁱ associated with the given pixel p in the first image is given by