COMBINED DEPTH FILTERING AND SUPER RESOLUTION

• US 20130129224A1
• Filed: 11/21/2011
• Published: 05/23/2013
• Est. Priority Date: 11/21/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Katz, Sagi, Adler, Avishai, Yahav, Giora
• Granted Patent: US 8,660,362 B2
• Time in Patent Office: Days
• US Class Current: 382/199
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

Systems and methods for increasing the resolution of a depth map by identifying and updating false depth pixels are described. In some embodiments, a depth pixel of the depth map is initially assigned a confidence value based on curvature values and localized contrast information. The curvature values may be generated by applying a Laplacian filter or other edge detection filter to the depth pixel and its neighboring pixels. The localized contrast information may be generated by determining a difference between the maximum and minimum depth values associated with the depth pixel and its neighboring pixels. A false depth pixel may be identified by comparing a confidence value associated with the false depth pixel with a particular threshold. The false depth pixel may be updated by assigning a new depth value based on an extrapolation of depth values associated with neighboring pixel locations.

Claims

1. A method for improving depth information, comprising:

acquiring a depth map of an environment, the depth map includes a plurality of pixels;

identifying one or more false depth pixels of the plurality of pixels, each of the one or more false depth pixels is located within a particular distance of a depth edge of the depth map;

updating the one or more false depth pixels, the updating the one or more false depth pixels includes updating a particular false depth pixel of the one or more false depth pixels based on one or more neighboring pixels of the particular false depth pixel; and

identifying one or more objects within the environment based on the depth map, the identifying one or more objects is performed subsequent to the updating the one or more false depth pixels.

2. The method of claim 1, further comprising:

generating a curvature map based on the depth map, the generating a curvature map includes applying a Laplacian filter to the depth map; and

calculating a plurality of confidence values based on the curvature map, each confidence value of the plurality of confidence values is associated with a different pixel of the plurality of pixels, the identifying one or more false depth pixels includes identifying the one or more false depth pixels based on the plurality of confidence values.

3. The method of claim 2, wherein:

the calculating a plurality of confidence values includes assigning to each confidence value of the plurality of confidence values an initial confidence value based on the curvature map and a localized image contrast.

4. The method of claim 2, wherein:

the identifying one or more false depth pixels includes determining whether the particular false depth pixel of the one or more false depth pixels is associated with a confidence value that is less than a particular confidence threshold.

5. The method of claim 2, wherein:

the updating a particular false depth pixel includes acquiring one or more confidence values associated with the one or more neighboring pixels, identifying a group of the one or more neighboring pixels based on the one or more confidence values, and determining a new depth value based on the group of the one or more neighboring pixels.

6. The method of claim 5, wherein:

the determining a new depth value includes extrapolating the new depth value based on one or more depth values associated with the group of the one or more neighboring pixels.

7. The method of claim 5, wherein:

the particular false depth pixel is associated with a particular curvature sign, each of the one or more neighboring pixels is associated with the particular curvature sign.

8. The method of claim 5, wherein:

the updating the one or more false depth pixels includes updating the one or more confidence values associated with the one or more false depth pixels.

9. The method of claim 2, further comprising:

invalidating one or more bad pixels of the one or more false depth pixels, the invalidating one or more bad pixels includes invalidating a particular bad pixel if a confidence value associated with the particular bad pixel is below a particular invalidating threshold.

10. The method of claim 1, further comprising:

executing a command on a computing system in response to the identifying one or more objects, the command is associated with controlling a display to modify the display of the one or more objects.

11. The method of claim 2, further comprising:

upscaling at least a portion of the depth map to a target resolution prior to performing the calculating a plurality of confidence values;

upscaling at least a portion of the curvature map to the target resolution prior to performing the calculating a plurality of confidence values;

12. A system for improving depth information, comprising:

a receiver, the receiver receives a depth map including a plurality of pixels; and

one or more processors, the one or more processors in communication with the receiver, the one or more processors generate a curvature map based on the depth map, the one or more processors initialize a plurality of confidence values based on the curvature map, each confidence value of the plurality of confidence values is associated with a different pixel of the plurality of pixels, the one or more processors identify one or more false depth pixels of the plurality of pixels based on the plurality of confidence values, the one or more processors update the one or more false depth pixels.

13. The electronic device of claim 12, wherein:

the one or more processors generate the curvature map by applying a Laplacian filter to the depth map.

14. The electronic device of claim 12, wherein:

the one or more processors assign to each confidence value of the plurality of confidence values an initial confidence value based on the curvature map and a localized image contrast.

15. One or more storage devices containing processor readable code for programming one or more processors to perform a method for improving depth information comprising the steps of:

acquiring a depth map of an environment;

generating a curvature map based on the depth map;

upscaling a portion of the depth map to a target resolution, the portion of the depth map includes a plurality of pixels;

upscaling a portion of the curvature map to the target resolution, the portion of the curvature map includes a plurality of curvature values;

initializing a plurality of confidence values based on the plurality of curvature values, each confidence value of the plurality of confidence values is associated with a different pixel of the plurality of pixels;

identifying one or more false depth pixels of the plurality of pixels based on the plurality of confidence values;

updating the one or more false depth pixels, the updating the one or more false depth pixels includes updating one or more depth values associated with the one or more false depth pixels; and

identifying one or more objects within the environment based on the portion of the depth map, the identifying one or more objects is performed subsequent to the updating the one or more false depth pixels.

16. The one or more storage devices of claim 15, wherein:

the generating a curvature map includes applying a Laplacian filter to the depth map.

17. The one or more storage devices of claim 16, wherein:

the initializing a plurality of confidence values includes assigning to each confidence value of the plurality of confidence values an initial confidence value based on the plurality of curvature values and a localized image contrast.

18. The one or more storage devices of claim 17, wherein:

the identifying one or more false depth pixels includes determining whether a particular false depth pixel of the one or more false depth pixels is associated with a confidence value that is less than a particular confidence threshold.

19. The one or more storage devices of claim 18, wherein:

the updating the one or more false depth pixels includes updating a particular false depth pixel of the one or more false depth pixels, the updating a particular false depth pixel includes identifying one or more neighboring pixels of the particular false depth pixel, acquiring one or more confidence values associated with the one or more neighboring pixels, and determining a new depth value based on the one or more neighboring pixels.

20. The one or more storage devices of claim 19, wherein:

the determining a new depth value includes extrapolating the new depth value based on one or more depth values associated with the one or more neighboring pixels.