Depth map confidence filtering

• US 8,885,890 B2
• Filed: 05/07/2010
• Issued: 11/11/2014
• Est. Priority Date: 05/07/2010
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Tardif, John
• Primary Examiner(s): CONWAY, THOMAS A
• Application Number:
  US12/776,066
• Publication Number:
  US 20110274366A1
• Time in Patent Office: 1,649 Days
• Field of Search: None
• US Class Current: 382/106
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20012 Locally adaptive
• CPC Class Codes: G06T 2207/30168 Image quality inspection
• CPC Class Codes: G06T 5/70 Denoising; Smoothing

Abstract

An apparatus and method for filtering depth information received from a capture device. Depth information is filtered by using confidence information provided with the depth information based an adaptively created, optimal spatial filter on a per pixel basis. Input data including depth information is received on a scene. The depth information comprises a plurality of pixels, each pixel including a depth value and a confidence value. A confidence weight normalized filter for each pixel in the depth information is generated. The weight normalized filter is combined with the input data to provide filtered data to an application.

Claims

1. A method for filtering depth information the depth information including a confidence measure;

comprising;

receiving input data including depth information on a scene, the depth information comprising a plurality of pixels, each pixel including a depth value and a confidence value;

generating a spatial weighting matrix based on the confidence value of the input pixel, generating a confidence weight normalized filter for each pixel in the depth information, the confidence weight normalized filter based on the spatial weighting matrix including a filtering weight derived from pixel confidence values relative to a distance of pixels surrounding each said pixel and a confidence weighting matrix;

combining the confidence weight normalized filter and the input data; and

outputting filtered data to an application.

2. The method of claim 1 wherein generating a confidence weight normalized filter includes:

selecting an input pixel in the depth information;

generating the spatial weighting matrix assigning a first filter weight based on contributions of a kernel of pixels surrounding the input pixel, taking into account the distance of the kernel pixels from the input pixel;

generating the confidence weighting matrix based on the confidence value of each pixel in the kernel;

multiplying the confidence weighting matrix and the spatial weighting matrix to provide a combined matrix;

normalizing the combined matrix; and

multiplying the combined matrix with the input data; and

summing the combined matrix products to derive a final filtered value.

3. The method of claim 1 further including generating an object correlation matrix and multiplying the combined matrix by the object correlation matrix.

4. The method of claim 1 wherein the step of generating the spatial weighting matrix includes calculating a distance from the input pixel to every pixel in the spatial matrix and combining the distance with the confidence value of the input pixel.

5. The method of claim 4 wherein the step of generating the spatial matrix includes assigning a greater weight to the input pixel depth value than weights assigned to kernel pixel depth values when the input pixel confidence value is higher than a specified threshold.

6. The method of claim 4 wherein the step of generating the spatial matrix includes assigning a greater weight to kernel pixel depth values than a weight assigned to the input pixel depth values when the input pixel confidence value is lower than a specified threshold.

7. A method for filtering depth information the depth information including a confidence measure;

comprising;

receiving depth information on a scene, the depth information including a depth value and a confidence value;

selecting an input pixel in the information;

generating a spatial weighting matrix assigning a first filter weight based on confidence value contributions from a kernel of pixels surrounding the input pixel, the confidence value contribution relative to a distance of each kernel pixel from the input pixel;

generating a confidence weighting matrix based on the confidence value of each pixel kernel;

generating an object correlation matrix based on the relative depth between an input pixel and pixels in the kernel;

combining the confidence weighting matrix, object correlation matrix, and the spatial weighting matrix to provide a combined matrix;

normalizing the combined matrix; and

multiplying the combined matrix with the input data and summing the products to derive a final filtered value.

8. The method of claim 7 wherein the step of generating the object correlation matrix includes determining whether a kernel pixel is undefined or on a different depth plane than the input pixel, and if so, assigning a value of zero to the kernel pixel, otherwise assigning a value of one to the kernel pixel.

9. The method of claim 7 wherein the step of generating the spatial weighting matrix includes generating the spatial weighting matrix based on a confidence value of the input pixel.

10. The method of claim 9 wherein the step of generating the spatial matrix includes calculating a distance from the input pixel to every pixel in the kernel of pixels and combining the distance with the confidence value of the input pixel.

11. The method of claim 10 wherein the step of generating the confidence weighting matrix includes assigning a weighting value based on the confidence value of each pixel in the kernel.

12. The method of claim 9 wherein the step of generating the spatial matrix includes assigning a greater weight to the input pixel depth value than weights assigned to kernel pixel depth values when the input pixel confidence value is higher than a specified threshold.

13. The method of claim 12 wherein the step of generating the spatial matrix includes assigning a greater weight to kernel pixel depth values than a weight assigned to the input pixel depth values when the input pixel confidence value is lower than a specified threshold.

14. An apparatus, comprising:

a depth information input;

a spatial weighting matrix generator;

a depth confidence weighting matrix generator;

a first multiplier coupled to the spatial weighting matrix generator and the depth confidence weighting matrix generator and outputting a combined matrix;

a combined matrix normalizer coupled to the first multiplier, wherein the first multiplier provides the combined matrix by summing coefficients of each of the input pixel and kernel pixels in the confidence weighting matrix and the spatial weighting matrix; and

a second multiplier coupled to the depth information input and the combined matrix normalizer, the multiplier providing a filtered depth information output.

15. The apparatus of claim 14 wherein the spatial weighting matrix generator includes an output comprising a first filter weight generated based on a kernel of pixels surrounding an input pixel, taking into account the distance of each kernel pixels from the input pixel.

16. The apparatus of claim 15 wherein the depth confidence weighting matrix generator outputs a confidence weighting matrix based on the confidence value of each pixel kernel.

17. The apparatus of claim 14 wherein the second multiplier combines depth information from the input with a normalized combined matrix.

18. The apparatus of claim 17 wherein spatial weighting matrix generator outputs a spatial weighting matrix based on a confidence value of the input pixel.