User center-of-mass and mass distribution extraction using depth images

• US 9,052,746 B2
• Filed: 02/15/2013
• Issued: 06/09/2015
• Est. Priority Date: 02/15/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kennett, Daniel, Hoof, Jonathan, Ahmad, Anis
• Primary Examiner(s): Cheng, Joe H
• Application Number:
  US13/768,400
• Publication Number:
  US 20140232650A1
• Time in Patent Office: 844 Days
• Field of Search: 345/10, 345/24, 345/156, 345/158, 345/418, 345/949, 382/103, 382/169, 382/237, 463/36
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/66 of image moments or centre of gravity
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Embodiments described herein use depth images to extract user behavior, wherein each depth image specifies that a plurality of pixels correspond to a user. A depth-based center-of-mass position is determined for the plurality of pixels that correspond to the user. Additionally, a depth-based inertia tensor can also be determined for the plurality of pixels that correspond to the user. In certain embodiments, the plurality of pixels that correspond to the user are divided into quadrants and a depth-based quadrant center-of-mass position is determined for each of the quadrants. Additionally, a depth-based quadrant inertia tensor can be determined for each of the quadrants. Based on one or more of the depth-based center-of-mass position, the depth-based inertial tensor, the depth-based quadrant center-of-mass positions or the depth-based quadrant inertia tensors, an application is updated.

Claims

1. A method for using a depth image to extract user behavior, comprising:

receiving a depth image that specifies that a plurality of pixels correspond to a user,

wherein the depth image is obtained using a capture device located a distance from the user,

wherein the depth image also specifies, for each of the pixels corresponding to the user, a pixel location and a pixel depth, and

wherein the pixel depth, specified for each of the pixels corresponding to the user, is indicative of a distance between the capture device and a portion of the user represented by the pixel;

determining, for each of the pixels corresponding to the user, a pixel mass that accounts for a distance between the portion of the user represented by the pixel and the capture device used to obtain the depth image; and

determining, based on the pixel mass determined for each of the pixels corresponding to the user, a depth-based center-of-mass position for the plurality of pixels corresponding to the user that accounts for distances between the portions of the user represented by the pixels and the capture device used to obtain the depth image.

2. The method of claim 1, further comprising:

using the determined depth-based center-of-mass position to update an application.

3. The method of claim 1, wherein:

the pixel depth for each pixel is specified by a depth value; and

the pixel mass, determined for each pixel corresponding to the user, is determined by squaring the depth value that specifies the pixel depth for the pixel.

4. The method of claim 1, further comprising:

using the determined depth-based center-of-mass position to divide the plurality of pixels corresponding to the user into quadrants; and

determining a separate depth-based quadrant center-of-mass position for each of the quadrants.

5. The method of claim 4, further comprising:

using one or more of the determined depth-based quadrant center-of-mass positions to update an application.

6. The method of claim 1, further comprising:

determining a depth-based inertia tensor for the plurality of pixels corresponding to the user based on the determined depth-based center-of-mass position for the plurality of pixels corresponding to the user.

7. The method of claim 6, further comprising:

scaling the determined depth-based inertia tensor using an assumption that the plurality of pixels corresponding to the user has a predetermined mass; and

using the scaled depth-based inertia tensor to update an application.

8. The method of claim 6, further comprising:

using the determined depth-based center-of-mass position and the determined depth-based inertia tensor to divide the plurality of pixels corresponding to the user into quadrants;

determining a separate depth-based quadrant center-of-mass position for each of the quadrants; and

using one or more of the determined depth-based quadrant center-of-mass positions to update an application.

9. The method of claim 8, wherein the using the determined depth-based center-of-mass position and the determined depth-based inertia tensor to divide the plurality of pixels corresponding to the user into quadrants, comprises:

determining principal axes for the plurality of pixels corresponding to the user based on the determined depth-based inertia tensor, or a scaled version thereof; and

using one of the determined principal axes, and a line perpendicular to the one of the determined principal axes that intersects the determined depth-based center-of-mass position, to divide the plurality of pixels corresponding to the user into the quadrants.

10. The method of claim 6, further comprising:

using the determined depth-based center-of-mass position and the determined depth-based inertia tensor to divide the plurality of pixels corresponding to the user into quadrants;

determining a separate depth-based quadrant center-of-mass position for each of the quadrants;

determining a separate depth-based quadrant inertia tensor for each of the quadrants; and

using one or more of the determined depth-based quadrant inertia tensors to update an application.

11. A system that uses depth images to extract user behavior, comprising:

a capture device that obtains depth images;

a communication interface that receives depth images from the capture device;

one or more storage devices that store depth images;

a display interface; and

one or more processors in communication with the one or more storage devices and the display interface, wherein the one or more processors are configured to determine, for each of a plurality of depth images

a depth-based center-of-mass position for a plurality of pixels of the depth image that correspond to a user, and

a depth-based inertia tensor for the plurality of pixels of the depth image that correspond to the user.

13. The system of claim 11, wherein the one or more processors use one or more of the determined depth-based center-of mass position or the determined depth-based inertia tensor to update an application running on the system and provide signals on the display interface that indicate an update to the application.

14. The system of claim 11, wherein the one or more processors are also configured to:

use a determined depth-based center-of-mass position and a determined depth-based inertia tensor to divide a plurality of pixels corresponding to the user into quadrants;

determine a separate depth-based quadrant center-of-mass position for each of the quadrants; and

use one or more of the determined depth-based quadrant center-of-mass positions to update an application running on the system and provide signals on the display interface that indicate an update to the application.

15. The system of claim 11, wherein the one or more processors are also configured to:

use a determined depth-based center-of-mass position and a determined depth-based inertia tensor to divide a plurality of pixels corresponding to the user into quadrants;

determine a separate depth-based quadrant center-of-mass position for each of the quadrants; and

determine a separate depth-based quadrant inertia tensor for each of the quadrants; and

use one or more of the determined depth-based quadrant inertia tensors, or scaled versions thereof, to update an application running on the system and provide signals on the display interface that indicate an update to the application.

12. The system of 11, wherein the depth-based center-of mass position and the depth-based inertia tensor are determined in a manner that accounts for distances between portions of the user represented by the pixels and the capture device used to obtain the depth image.

16. One or more processor readable storage devices having instructions encoded thereon which when executed cause one or more processors to perform a method for using depth images to extract user behavior, the method comprising:

receiving a depth image that specifies that a plurality of pixels correspond to a user;

determining pixel masses for the pixels corresponding to the user;

determining, based on the determined pixel masses for the pixels corresponding to the user, a depth-based center-of-mass position for the plurality of pixels corresponding to the user; and

determining a depth-based inertia tensor for the plurality of pixels corresponding to the user based on the determined depth-based center-of-mass position for the plurality of pixels corresponding to the user.

17. The one or more processor readable storage devices of claim 16, wherein:

the determined pixel masses account for distances between the portions of the user represented by the pixels and a capture device used to obtain the depth image; and

the depth-based center-of-mass position accounts for distances between portions of the user represented by the pixels and the capture device used to obtain the depth image.

18. The one or more processor readable storage devices of claim 16, wherein the method further comprises:

using at least one of the depth-based center-of-mass position, the depth-based inertia tensor, or a scaled version of the depth-based inertia tensor to update an application.

19. The one or more processor readable storage device of claim 16, wherein the method further comprises:

using the determined depth-based center-of-mass position and the determined depth-based inertia tensor to divide the plurality of pixels corresponding to the user into quadrants;

determining a separate depth-based quadrant center-of-mass position for each of the quadrants; and

using one or more of the determined depth-based quadrant center-of-mass positions to update an application.