DEPTH MAP CONFIDENCE FILTERING

• US 20110274366A1
• Filed: 05/07/2010
• Published: 11/10/2011
• Est. Priority Date: 05/07/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Tardif, John
• Granted Patent: US 8,885,890 B2
• Time in Patent Office: Days
• US Class Current: 382/260
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20012 Locally adaptive
• CPC Class Codes: G06T 2207/30168 Image quality inspection
• CPC Class Codes: G06T 5/70 Denoising; Smoothing

Abstract

An apparatus and method for filtering depth information received from a capture device. Depth information is filtered by using confidence information provided with the depth information based an adaptively created, optimal spatial filter on a per pixel basis. Input data including depth information is received on a scene. The depth information comprises a plurality of pixels, each pixel including a depth value and a confidence value. A confidence weight normalized filter for each pixel in the depth information is generated. The weight normalized filter is combined with the input data to provide filtered data to an application.

Claims

1. A method for filtering depth information the depth information including a confidence measure;

comprising;

receiving input data including depth information on a scene, the depth information comprising a plurality of pixels, each pixel including a depth value and a confidence value;

generating a confidence weight normalized filter for each pixel in the depth information, the confidence weight normalized filter including a filtering weight derived from confidence values relative to a distance of pixels surrounding each said pixel;

combining the normalized filter and the input data; and

outputting filtered data to an application.

2. The method of claim 1 wherein generating a confidence weight normalized filter includes:

selecting an input pixel in the depth information;

generating a spatial weighting matrix assigning a first filter weight based on the contributions a kernel of pixels surrounding the input pixel, taking into account the distance of the kernel pixels from the input pixel;

generating a confidence weighting matrix based on the confidence value of each pixel in the kernel;

multiplying the confidence weighting matrix and the spatial weighting matrix to provide a combined matrix;

normalizing the combined matrix; and

multiplying the weighted normalized matrix with the input data; and

summing the products to derive a final filtered value.

3. The method of claim 2 wherein the step of generating a spatial weighting matrix includes generating the matrix based on a confidence value of the input pixel.

4. The method of claim 3 wherein the step of generating a spatial matrix includes calculating a distance from the input pixel to every pixel in the spatial matrix and combining the distance with the confidence value of the input pixel.

5. The method of claim 4 wherein the step of generating a spatial matrix includes assigning a greater weight to the input pixel depth value than weights assigned to kernel pixel depth values when the input pixel confidence value is higher than a specified threshold.

6. The method of claim 4 wherein the step of generating a spatial matrix includes assigning a greater weight to kernel pixel depth values than a weight assigned to the input pixel depth values when the input pixel confidence value is lower than a specified threshold.

7. The method of claim 1 further including the step of generating an object correlation matrix and multiplying the combined matrix by an object correlation matrix.

8. A method for filtering depth information the depth information including a confidence measure;

comprising;

receiving depth information on a scene, the depth information including a depth value and a confidence value;

selecting an input pixel in the information;

generating a spatial weighting matrix assigning a first filter weight based on confidence value contributions from a kernel of pixels surrounding the input pixel, the confidence value contribution relative to a distance of each kernel pixel from the input pixel;

generating a confidence weighting matrix based on the confidence value of each pixel kernel;

generating an object correlation matrix based on the relative depth between an input pixel and pixels in the kernel;

combining the confidence weighting matrix, object correlation matrix, and the spatial weighting matrix to provide a combined matrix;

normalizing the combined matrix; and

multiplying the weighted normalized matrix with the input data and summing the products to derive a final filtered value.

9. The method of claim 8 wherein the step of generating a spatial weighting matrix includes generating the matrix based on a confidence value of the input pixel.

10. The method of claim 9 wherein the step of generating a spatial matrix includes calculating a distance from the input pixel to every pixel in the spatial matrix and combining the distance with the confidence value of the input pixel.

11. The method of claim 9 wherein the step of generating a spatial matrix includes assigning a greater weight to the input pixel depth value than weights assigned to kernel pixel depth values when the input pixel confidence value is higher than a specified threshold.

12. The method of claim 11 wherein the step of generating a spatial matrix includes assigning a greater weight to kernel pixel depth values than a weight assigned to the input pixel depth values when the input pixel confidence value is lower than a specified threshold.

13. The method of claim 10 wherein the step of generating a confidence weighting matrix includes assigning a weighting value based on the confidence value of each pixel in the kernel.

14. The method of claim 8 wherein the step of generating an object correlation matrix includes determining whether a kernel pixel is undefined or on a different depth plane than the input pixel, and if so assigning a value of zero to the kernel pixel, otherwise assigning a value of one to the kernel pixel.

15. An apparatus, comprising:

a depth information input;

a spatial weighting matrix generator;

a confidence weighting matrix generator;

an object correlation weighting matrix generator;

a first multiplier coupled to the spatial weighting matrix generator and the a confidence weighting matrix generator and outputting a combined matrix;

a combined matrix normalizer coupled to the first multiplier; and

a second multiplier coupled to the depth information input and the combined matrix normalizer, the multiplier providing a filtered depth information output.

16. The apparatus of claim 15 wherein the spatial weighting matrix generator includes an output comprising a first filter weight generated based on a kernel of pixels surrounding an input pixel, taking into account the distance of each kernel pixels from the input pixel.

17. The apparatus of claim 16 wherein the confidence weighting matrix generator outputs a confidence weighting matrix based on the confidence value of each pixel kernel.

18. The apparatus of claim 17 wherein the first multiplier provides the combined matrix by summing coefficients of each of the input pixel and kernel pixels in the confidence weighting matrix and the spatial weighting matrix.

19. The apparatus of claim 18 wherein the second multiplier combines depth information from the input with a normalized combined matrix.

20. The apparatus of claim 19 wherein spatial weighting matrix generator outputs a confidence weighting matrix based on a confidence value of the input pixel.