Depth image processing

• US 9,092,657 B2
• Filed: 03/13/2013
• Issued: 07/28/2015
• Est. Priority Date: 03/13/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Ahmad, Anis, Hoof, Jonathan, Kennett, Daniel
• Primary Examiner(s): Patel, Jay
• Assistant Examiner(s): Geroleo, Francis G
• Application Number:
  US13/801,966
• Publication Number:
  US 20140267610A1
• Time in Patent Office: 867 Days
• Field of Search: 348/46
• US Class Current: 1/1
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

Embodiments described herein can be used to detect holes in a subset of pixels of a depth image that has been specified as corresponding to a user, and to fill such detected holes. Additionally, embodiments described herein can be used to produce a low resolution version of a subset of pixels that has been specified as corresponding to a user, so that when an image including a representation of the user is displayed, the image respects the shape of the user, yet is not a mirror image of the user. Further, embodiments described herein can be used to identify pixels, of a subset of pixels specified as corresponding to the user, that likely correspond to a floor supporting the user. This enables the removal of the pixels, identified as likely corresponding to the floor, from the subset of pixels specified as corresponding to the user.

Claims

1. A method for use with depth images, comprising:

(a) obtaining a depth image and information that specifies that a subset of pixels within the depth image correspond to a user;

(b) identifying one or more spans of pixels, within the subset of pixels specified as corresponding to the user, that are potentially part of a hole in the subset of pixels that resulted from a portion of the user occluding another portion of the user when a capture device was used to obtain the depth image or a higher resolution version of the depth image;

(c) analyzing span adjacent pixels to identify, as also potentially being part of a hole in the subset of pixels specified as corresponding to the user, each span adjacent pixel that both includes

(c1) at least a first threshold number of neighboring pixels identified as potentially being part of a hole, and

(c2) no more than a second threshold number of neighboring pixels identified as a potential hole boundary; and

(d) grouping together pixels that are adjacent to one another and have been identified as potentially being part of a hole in the subset of pixels specified as corresponding to the user, to thereby identify one or more islands of pixels that potentially correspond to one or more holes in the subset of pixels specified as corresponding to the user.

2. The method of claim 1, further comprising:

(e) classify each said island of pixels as either being a hole or not being a hole in the subset of pixels specified as corresponding to the user;

(f) filling each said island of pixels classified as being a hole; and

(g) using results of the hole filling when displaying an image including a representation of the user.

3. The method of claim 1, wherein the (b) identifying one or more spans of pixels that are potentially part of a hole in the subset of pixels corresponding to the user comprises:

(b1) identify each horizontal span of pixels, within the subset of pixels specified as corresponding to the user, where on both sides of the horizontal span there is a change in depth values from one pixel to its horizontal neighboring pixel that exceeds a depth discontinuity threshold; and

(b2) identify each vertical span of pixels, within the subset of pixels specified as corresponding to the user, where on both sides of the vertical span there is a change in depth values from one pixel to its vertical neighboring pixel that exceeds the depth discontinuity threshold.

4. The method of claim 1, further comprising:

(e) analyzing each of the identified one or more islands of pixels to thereby classify each said island of pixels as either being a hole or not being a hole in the subset of pixels specified as corresponding to the user.

5. The method of claim 4, wherein the (e) is analyzing each of the identified islands of pixels comprises, for each said island of pixels:

(e1) determining a ratio indicative of a height-to-width, or a width-to-height, of the island of pixels; and

(e2) comparing the determined ratio to a threshold ratio; and

(e3) classifying the island of pixels as either being a hole or not being a hole based on results of the comparing.

6. The method of claim 5, further comprising:

(f) hole filling each said island of pixels that was classified as being a hole in the subset of pixels specified as corresponding to the user; and

(g) storing information indicative of results of the hole filling so that said stored information is available for use when displaying an image corresponding to the user.

7. The method of claim 6, further comprising:

(b) using the information indicative of results of the hole filling when displaying an image corresponding to the user.

8. The method of claim 1, wherein the (a) obtaining the depth image and the information that specifies that a subset of pixels within the depth image correspond to a user comprises:

(a1) receiving an original version of a depth image obtained using a capture device located a distance from the user and original information that specifies that a subset of pixels within the depth image correspond to a user;

(a2) down-sampling the subset of pixels within the original depth image that are specified as corresponding to the user to produce a first low resolution subset of pixels that correspond to the user;

(a3) performing a morphological open on the first low resolution subset of pixels that correspond to the user; and

(a3) producing a second low resolution subset of pixels that correspond to the user by including, in the second low resolution subset of pixels that correspond to the user, pixels that are in both the original version of the subset of pixels that correspond to the user and in the first low resolution subset of pixels that correspond to the user;

wherein the second low resolution subset of pixels that correspond to the user, or a further down-sampled version thereof, comprises the subset of pixels specified as corresponding to the user that is used when (b) identifying one or more spans of pixels that are potentially part of a hole.

9. The method of claim 8, wherein the (a2) down-sampling the subset of pixels within the original depth image that are specified as corresponding to the user to produce a first low resolution subset of pixels that correspond to the user comprises, for each of a plurality of blocks of pixels within the original depth image:

selecting, from the block of pixels, one of the pixels that is specified as corresponding to the user in the original version of the depth image; and

replacing the selected one of the pixels with a weighted sum of its neighboring pixels.

10. The method of claim 9, wherein when replacing the selected one of the pixels with a weighted sum of its neighboring pixels, three weighting factors are used to produce the weighted sum including:

a spatial weighting factor indicative of a distance between the pixel and a neighboring pixel;

a depth weighting factor indicative of whether a difference between a depth value of the pixel and a depth value of a neighboring pixel is less than a threshold; and

a segmentation weighting factor indicative of whether a neighboring pixel is within the subset of pixels specified as corresponding to the user.

11. The method of claim 1, further comprising:

identifying one or more pixels, of the subset of pixels specified as corresponding to the user, that likely correspond to a floor that is supporting the user; and

removing, from the subset of pixels specified as corresponding to the user, the one or more pixels identified as likely corresponding to the floor that is supporting the user.

12. The method of claim 1, further comprising adjusting the depth image to reduce a sensitivity of the depth image to tilt of a capture device used to obtain the depth image, wherein the adjusting the depth image includes:

obtaining an up-vector associated with the capture device used to obtain the depth image;

transforming pixels in the depth image from depth image space to three-dimensional (3D) camera space to produce a 3D representation of the depth image which includes a subset of pixels specified as corresponding to the user;

using the up-vector to transform the subset of pixels specified as corresponding to the user to a plane having a specified attitude; and

transforming the 3D representation of the depth image from the 3D camera space back to the depth image space.

13. The method of claim 1, wherein each said span adjacent pixel comprises a pixel that is adjacent to at least one of the one or more spans of pixels identified at the (b) identifying one or more spans of pixels that are potentially part of a hole.

14. A system, comprising:

a capture device that obtains depth images;

a communication interface that receives depth images from the capture device;

one or more storage devices that store depth images;

a display interface; and

one or more processors in communication with the one or more storage devices and the display interface, wherein the one or more processors are configured to

obtain a depth image and information that specifies that a subset of pixels within the depth image correspond to a user;

identify one or more spans of pixels, within the subset of pixels specified as corresponding to the user, that are potentially part of a hole in the subset of pixels that resulted from a portion of the user occluding another portion of the user when a capture device was used to obtain the depth image or a higher resolution version of the depth image;

analyze span adjacent pixels to identify as also potentially being part of a hole in the subset of pixels specified as corresponding to the user, each span adjacent pixel that both includes

at least a first threshold number of neighboring pixels identified as potentially being part of a hole, and

no more than a second threshold number of neighboring pixels identified as a potential hole boundary; and

group together pixels that are adjacent to one another and have been identified as potentially being part of a hole in the subset of pixels specified as corresponding to the user, to thereby identify one or more islands of pixels that potentially correspond to one or more holes in the subset of pixels specified as corresponding to the user;

classify each identified island of pixels as either being a hole or not being a hole in the subset of pixels specified as corresponding to the user; and

hole fill each said island of pixels classified as being a hole.

15. The system of claim 14, wherein the one or more processors are also configured to use results of the hole filling to update an application running on the system and provide signals on the display interface that indicate an update to the application.

16. The system of claim 14, wherein the one or more processors are also configured to use results of the hole filling to update image data that includes a representation of the user.

17. The system of claim 14, wherein the one or more processors are also configured to:

identify one or more pixels, of the subset of pixels specified as corresponding to the user, that likely correspond to a floor that is supporting the user; and

remove, from the subset of pixels specified as corresponding to the user, the one or more pixels identified as likely corresponding to the floor that is supporting the user.

18. One or more processor readable hardware storage devices having instructions encoded thereon which when executed cause one or more processors to perform a method, the method comprising:

obtaining a depth image and information that specifies that a subset of pixels within the depth image correspond to a user;

identifying one or more spans of pixels, within the subset of pixels specified as corresponding to the user, that are potentially part of a hole in the subset of pixels that resulted from a portion of the user occluding another portion of the user when a capture device was used to obtain the depth image or a higher resolution version of the depth image;

analyzing span adjacent pixels to identify, as also potentially being part of a hole in the subset of pixels specified as corresponding to the user, each span adjacent pixel that both includes

at least a first threshold number of neighboring pixels identified as potentially being part of a hole, and

no more than a second threshold number of neighboring pixels identified as a potential hole boundary; and

grouping together pixels that are adjacent to one another and have been identified as potentially being part of a hole in the subset of pixels specified as corresponding to the user, to thereby identify one or more islands of pixels that potentially correspond to one or more holes in the subset of pixels specified as corresponding to the user;

analyzing each of the identified one or more islands of pixels to thereby classify each said island of pixels as either being a hole or not being a hole in the subset of pixels specified as corresponding to the user; and

hole filling each said island of pixels that was classified as being a hole in the subset of pixels specified as corresponding to the user.

19. The one or more processor readable hardware storage devices of claim 18, wherein the method further comprises at least one of:

using results of the hole filling when displaying an image including a representation of the user;

or

storing information indicative of results of the hole filling so that said stored information is available for use when displaying an image corresponding to the user.

20. The one or more processor readable hardware storage devices of claim 18, wherein the analyzing each of the identified islands of pixels comprises, for each said island of pixels:

determining a ratio indicative of a height-to-width, or a width-to-height, of the island of pixels;

comparing the determined ratio to a threshold ratio; and

classifying the island of pixels as either being a hole or not being a hole based on results of the comparing.