Depth image processing

• US 9,824,260 B2
• Filed: 07/07/2015
• Issued: 11/21/2017
• Est. Priority Date: 03/13/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Ahmad, Anis, Hoof, Jonathan, Kennett, Daniel
• Primary Examiner(s): Geroleo, Francis G
• Application Number:
  US14/793,337
• Publication Number:
  US 20150310256A1
• Time in Patent Office: 868 Days
• Field of Search: 348 43
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

Embodiments described herein can be used to detect holes in a subset of pixels of a depth image that has been specified as corresponding to a user, and to fill such detected holes. Additionally, embodiments described herein can be used to produce a low resolution version of a subset of pixels that has been specified as corresponding to a user, so that when an image including a representation of the user is displayed, the image respects the shape of the user, yet is not a mirror image of the user. Further, embodiments described herein can be used to identify pixels, of a subset of pixels specified as corresponding to the user, that likely correspond to a floor supporting the user. This enables the removal of the pixels, identified as likely corresponding to the floor, from the subset of pixels specified as corresponding to the user.

Claims

1. A method for use with depth images, comprising:

obtaining a depth image and information that specifies that a subset of pixels within the depth image correspond to a user;

transforming pixels in the depth image from depth image space to three-dimensional (3D) camera space to produce a 3D representation of the depth image in the 3D camera space which includes a subset of pixels in the 3D camera space specified as corresponding to the user;

after the transforming, identifying one or more pixels in the 3D camera space, of the subset of pixels in the 3D camera space specified as corresponding to the user, that likely correspond to a floor that is supporting the user;

removing, from the subset of pixels in the 3D camera space specified as corresponding to the user, the one or more pixels in the 3D camera space identified as likely corresponding to the floor that is supporting the user; and

after the removing, transforming the 3D representation of the depth image from the 3D camera space back to the depth image space.

2. The method of claim 1, wherein the identifying and removing comprise:

obtaining coefficients corresponding to the floor that satisfy a specified plane equation; and

for at least some pixels of the subset of pixels in the 3D camera space specified as corresponding to the user in the 3D representation of the depth image

calculating a floor relative value (FRV);

comparing the FRV to a specified value;

determining, based on results of the comparing, whether or not to remove the pixel from the subset of pixels in the 3D camera space specified as corresponding to the user; and

removing the pixel, from the subset of pixels in the 3D camera space specified as corresponding to the user, if it is determined that the pixel is to be removed.

3. The method of claim 2, wherein the specified plane equation is a*x +b*y +c*z +d =0, and wherein the obtaining coefficients corresponding to the floor that satisfy the specified plane equation comprises determining values for coefficients a, b, c and d in the specified plane equation.

4. The method of claim 3, wherein the calculating the FRV, which is calculated for the at least some of the pixels of the subset of pixels in the 3D camera space specified as corresponding to the user in the 3D representation of the depth image, comprises:

selecting a pixel in the 3D camera space specified as corresponding to the user from the 3D representation of the depth image; and

calculating the FRV for the selected pixel using the equation FRV =a*x +b*y +c*z +d, where the a, b, c and d coefficients corresponding to the floor are used, and the x, y and z values of the selected pixel are used.

5. The method of claim 4, further comprising repeating the selecting a pixel and the calculating the FRV for the selected pixel, for each of the at least some of the pixels of the subset of pixels in the 3D camera space specified as corresponding to the user in the 3D representation of the depth image.

6. The method of claim 2, wherein the at least some pixels, of the subset of pixels in the 3D camera space specified as corresponding to the user in the 3D representation of the depth image, comprises all pixels of the subset.

7. The method of claim 2, wherein the at least some pixels, of the subset of pixels in the 3D camera space specified as corresponding to the user in the 3D representation of the depth image, comprises only pixels of the subset that are within a specified distance of the floor.

8. The method of claim 2, wherein:

the specified value is zero;

the comparing the FRV to the specified value comprises comparing the FRV to zero to determine whether the FRV is less than or equal to zero; and

the determining, based on results of the comparing, whether or not to remove the pixel from the subset of pixels in the 3D camera space specified as corresponding to the user, comprises determining that the pixel is to be removed from the subset of pixels in the 3D camera space specified as corresponding to the user if the FRV for the pixel is determined to be less than or equal to zero.

9. The method of claim 2, wherein:

the specified value is zero;

the comparing the FRV to the specified value comprises comparing the FRV to zero to determine whether the FRV is less than zero; and

the determining, based on results of the comparing, whether or not to remove the pixel from the subset of pixels in the 3D camera space specified as corresponding to the user, comprises determining that the pixel is to be removed from the subset of pixels in the 3D camera space specified as corresponding to the user if the FRV for the pixel is determined to be less than zero.

10. The method of claim 1, further comprises:

adjusting the depth image to reduce a sensitivity of the depth image to tilt of a capture device used to obtain the depth image.

11. The method of claim 10, wherein the adjusting the depth image to reduce the sensitivity of the depth image to tilt of the capture device comprises:

obtaining an up-vector associated with the capture device used to obtain the depth image;

transforming pixels in the depth image from depth image space to three-dimensional (3D) camera space to produce a 3D representation of the depth image which includes a subset of pixels in the 3D camera space specified as corresponding to the user;

using the up-vector to transform the subset of pixels in the 3D camera space specified as corresponding to the user to a plane having a specified attitude; and

transforming the 3D representation of the depth image from the 3D camera space back to the depth image space.

12. The method of claim 1, further comprising:

after the removing, and after the transforming the 3D representation of the depth image from the 3D camera space back to the depth image space, using remaining pixels in the subset of pixels specified as corresponding to the user, when displaying an image corresponding to the user.

13. A system, comprising:

a capture device that obtains depth images;

a communication interface that receives depth images from the capture device; and

one or more storage devices that store depth images;

one or more processors in communication with the one or more storage devices, wherein the one or more processors are configured to

obtain a depth image and information that specifies that a subset of pixels within the depth image correspond to a user;

transform pixels in the depth image from depth image space to three-dimensional (3D) camera space to produce a 3D representation of the depth image in the 3D camera space which includes a subset of pixels in the 3D camera space specified as corresponding to the user;

identify one or more pixels in the 3D camera space, of the subset of pixels in the 3D camera space specified as corresponding to the user, that likely correspond to a floor that is supporting the user; and

remove, from the subset of pixels in the 3D camera space specified as corresponding to the user, the one or more pixels in the 3D camera space identified as likely corresponding to the floor that is supporting the user.

14. The system of claim 13, wherein the one or more processors, in order to identify and remove the one or more pixels in the 3D camera space that likely correspond to a floor that is supporting the user, from the subset of pixels in the 3D camera space specified as corresponding to the user, are configured to:

obtain coefficients corresponding to the floor that satisfy a specified plane equation; and

for at least some pixels of the subset of pixels in the 3D camera space specified as corresponding to the user in the 3D representation of the depth image

calculate a floor relative value (FRV);

compare the FRV to a specified value;

determine, based on results of the comparing, whether or not to remove the pixel from the subset of pixels in the 3D camera space specified as corresponding to the user; and

remove the pixel, from the subset of pixels in the 3D camera space specified as corresponding to the user, if it is determined that the pixel is to be removed.

15. The system of claim 14, wherein the one or more processors are configured to transform the 3D representation of the depth image from the 3D camera space back to the depth image space, after the pixels to be removed have been removed from the subset of pixels specified as corresponding to the user.

16. The system of claim 13, wherein the one or more processors are also configured to adjust the depth image to reduce a sensitivity of the depth image to tilt of the capture device.

17. The system of claim 16, wherein in order to adjust the depth image to reduce the sensitivity of the depth image to tilt of the capture device, the one or more processors are configured to:

obtain an up-vector associated with the capture device used to obtain the depth image; and

use the up-vector to transform the subset of pixels in the 3D camera space specified as corresponding to the user to a plane having a specified attitude.

18. One or more processor readable storage devices having instructions encoded thereon which when executed cause one or more processors to perform a method, the method comprising:

obtaining a depth image and information that specifies that a subset of pixels within the depth image correspond to a user;

transforming pixels in the depth image from depth image space to three-dimensional (3D) camera space to produce a 3D representation of the depth image in the 3D camera space which includes a subset of pixels in the 3D camera space specified as corresponding to the user;

after the transforming, identifying one or more pixels in the 3D camera space, of the subset of pixels in the 3D camera space specified as corresponding to the user, that likely correspond to a floor that is supporting the user;

removing, from the subset of pixels in the 3D camera space specified as corresponding to the user, the one or more pixels in the 3D camera space identified as likely corresponding to the floor that is supporting the user; and

after the removing, transforming the 3D representation of the depth image from the 3D camera space back to the depth image space.

19. The one or more processor readable storage devices of claim 18, wherein the identifying and removing comprise:

obtaining coefficients corresponding to the floor that satisfy a specified plane equation; and

for at least some pixels of the subset of pixels in the 3D camera space specified as corresponding to the user in the 3D representation of the depth image

calculating a floor relative value (FRV);

comparing the FRV to a specified value;

determining, based on results of the comparing, whether or not to remove the pixel from the subset of pixels in the 3D camera space specified as corresponding to the user; and

removing the pixel, from the subset of pixels in the 3D camera space specified as corresponding to the user, if it is determined that the pixel is to be removed.

20. The one or more processor readable storage devices of claim 18, wherein the method further comprises adjusting the depth image to reduce a sensitivity of the depth image to tilt of a capture device used to obtain the depth image, wherein the adjusting the depth image includes:

obtaining an up-vector associated with the capture device used to obtain the depth image; and

using the up-vector to transform the subset of pixels in the 3D camera space specified as corresponding to the user to a plane having a specified attitude.