Extraction of user behavior from depth images

• US 9,959,459 B2
• Filed: 03/14/2016
• Issued: 05/01/2018
• Est. Priority Date: 03/08/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hoof, Jonathan, Ahmad, Anis, Kennett, Daniel
• Primary Examiner(s): GORADIA, SHEFALI DINESH
• Application Number:
  US15/069,558
• Publication Number:
  US 20160196468A1
• Time in Patent Office: 778 Days
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/44 : Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 40/103 : Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/23 : Recognition of whole body movements, e.g. for sport training

Abstract

Embodiments described herein use depth images to extract user behavior, wherein each depth image specifies that a plurality of pixels correspond to a user. In certain embodiments, one or more average extremity positions of a user, which can also be referred to as average positions of extremity blobs, are extracted from a depth image. An application is then updated based on the average positions of extremity blobs.

Claims

1. For use by a computing system including one or more processors that are running first software instructions, a method for extracting user behavior from a depth image that includes a plurality of pixels that correspond to a user that is interacting with the application running on the one or more processors, the method comprising:

identifying a pixel of the depth image that corresponds to an extremity of the user;

identifying an extremity blob that comprises pixels of the depth image that correspond to the user and are within a specified distance of the pixel identified as corresponding to the extremity of the user;

identifying an average position of the extremity blob by determining an average position of the pixels included in the extremity blob; and

using the average position of the extremity blob to provide input to second software instructions running on the one or more processors, wherein the second software instructions may comprise the first software instructions, wherein the input to the second software instructions running on the one or more processors comprises at least one of the following;

a command or action to control or activate a user interface element of the second software instructions;

a command or action to animate an avatar that is being displayed under control of the second software instructions;

an update to a position or movement of an avatar, character or other on-screen object or image that is being displayed under control of the second software instructions;

or

an update to tracking of the user performing an exercise or pose so that feedback or points can be provided by the application to the user.

2. The method of claim 1, wherein the average position of the extremity blob corresponds to one of a left side, a right side, a top, a bottom or a front of the user.

3. The method of claim 1, further comprising repeating the steps of the method of claim 1 for one or more further extremities of the user so that an average position of an extremity blob is determined for each of a plurality of extremities of the user.

4. The method of claim 3, wherein each said average position of an extremity blob corresponds to different one of a left side, a right side, a top, a bottom or a front of the user.

5. The method of claim 1, wherein the identifying a pixel of the depth image that corresponds to an extremity of the user comprises:

checking pixels in a line extending along a side of the depth image to determine if a pixel in the line corresponds to the user;

incrementally moving the line each time there is a determination that no pixel in the line corresponds to the user; and

in response to determining that a pixel in the line corresponds to the user, identifying the pixel in the line as corresponding to an extremity of the user.

6. The method of claim 1, wherein:

the identifying an extremity blob includes identifying pixels of the depth image that correspond to the user and are within specified blob boundaries; and

the identifying an average position of the extremity blob comprises;

determining an average x value of the pixels of the depth image that correspond to the user and are within the specified blob boundaries; and

determining an average y value of the pixels of the depth image that correspond to the user and are within the specified blob boundaries;

the average x value corresponds to an average horizontal position in the depth image of the extremity blob; and

the average y value corresponds to an average vertical position in the depth image of the extremity blob.

7. The method of claim 1, further comprising:

using a capture device, located a distance from the user, to obtain the depth image; and

performing a segmentation process to identify the plurality of pixels of the depth image that correspond to the user;

wherein the depth image specifies, for each of the pixels corresponding to the user, a pixel location and a pixel depth; and

wherein the pixel depth, specified for each of the pixels corresponding to the user, is indicative of a distance between the capture device and a portion of the user represented by the pixel.

8. A system that extracts user behavior from a depth image that includes a plurality of pixels that correspond to a user, the system comprising:

one or more storage devices that store depth images; and

one or more processors in communication with the one or more storage devices, wherein the one or more processors are configured to

identify a pixel of a said depth image that corresponds to an extremity of the user;

identify an extremity blob that comprises pixels of the said depth image that correspond to the user and are within a specified distance of the pixel identified as corresponding to the extremity of the user;

identify an average position of the extremity blob by determining an average position of the pixels included in the extremity blob; and

use the average position of the extremity blob to update software instructions running on the one or more processors, wherein the update to the software instructions running on the one or more processors comprises at least one of the following;

a command or action to control or activate a user interface element of the software instructions;

a command or action to animate an avatar that is being displayed under control of the software instructions;

an update to a position or movement of an avatar, character or other on-screen object that is being displayed under control of the software instructions;

or

an update to tracking of the user performing an exercise or pose so that feedback or points can be provided by the software instructions to the user.

9. The system of claim 8, wherein the average position of the extremity blob corresponds to one of a left side, a right side, a top, a bottom or a front of the user.

10. The system of claim 8, wherein the one or more processors are configured to identify an average position of an extremity blob for one or more further extremities of the user so that an average position of an extremity blob is identified for each of a plurality of extremities of the user;

and wherein the one or more processors are configured to use the average position of at least two of the extremity blobs to update the application.

11. The system of claim 10, wherein each said average position of an extremity blob corresponds to different one of a left side, a right side, a top, a bottom or a front of the user.

12. The system of claim 8, wherein in order to identify a pixel of a said depth image that corresponds to an extremity, the one or more processors are configured to:

check pixels in a line extending along a side of the depth image to determine if a pixel in the line corresponds to the user;

incrementally move the line each time there is a determination that no pixel in the line corresponds to the user; and

identify the pixel in the line as corresponding to an extremity of the user, in response to a determination that a pixel in the line corresponds to the user.

13. The system of claim 8, wherein:

in order to identify an extremity blob, the one or more processors are configured to identify pixels of a said depth image that correspond to the user and are within specified blob boundaries; and

in order to identify an average position of the extremity blob, the one or more processors are configured to;

determine an average x value of the pixels of the said depth image that correspond to the user and are within the specified blob boundaries; and

determine an average y value of the pixels of the said depth image that correspond to the user and are within the specified blob boundaries;

the average x value corresponding to an average horizontal position in the said depth image of the extremity blob; and

the average y value corresponding to an average vertical position in the said depth image of the extremity blob.

14. The system of claim 8, wherein:

the one or more processors are also configured to perform a segmentation process to identify the plurality of pixels of a said depth image that correspond to the user;

each of the depth images specifies, for each of the pixels corresponding to the user, a pixel location and a pixel depth; and

the pixel depth, specified for each of the pixels corresponding to the user, is indicative of a distance between a capture device and a portion of the user represented by the pixel.

15. One or more processor readable storage devices having instructions encoded thereon which when executed cause one or more processors to perform a method for using depth images to extract user behavior, the method comprising:

receiving a depth image that specifies that a plurality of pixels correspond to a user;

identifying a pixel of the depth image that corresponds to an extremity of the user;

identifying an extremity blob that comprises pixels of the depth image that correspond to the user and are within a specified distance of the pixel identified as corresponding to the extremity of the user; and

updating software instructions, which may comprise the instructions, running on one or more processors in dependence on the extremity blob, wherein the updating comprises at least one of the following;

a command or action to control or activate a user interface element of the software instructions;

a command or action to animate an avatar that is being displayed under control of the software instructions;

updating a position or movement of an avatar, character or other on-screen object that is being displayed under control of the software instructions;

or

updating tracking of the user performing an exercise or pose so that feedback or points can be provided by the software instructions to the user.

16. The one or more processor readable storage devices of claim 15, wherein:

the method further comprises identifying an average position of the extremity blob by determining an average position of the pixels included in the extremity blob; and

the updating an application in dependence on the extremity blob comprises using the average position of the extremity blob to update the application.

17. The one or more processor readable storage devices of claim 15, wherein the average position of the extremity blob corresponds to one of a left side, a right side, a top, a bottom or a front of the user.

18. The one or more processor readable storage devices of claim 15, wherein the identifying a pixel of the depth image that corresponds to an extremity of the user comprises:

checking pixels in a line extending along a side of the depth image to determine if a pixel in the line corresponds to the user;

incrementally moving the line each time there is a determination that no pixel in the line corresponds to the user; and

in response to determining that a pixel in the line corresponds to the user, identifying the pixel in the line as corresponding to an extremity of the user.

19. The one or more processor readable storage devices of claim 15, wherein:

the identifying an extremity blob includes identifying pixels of the depth image that correspond to the user and are within specified blob boundaries; and

the identifying an average position of the extremity blob comprises;

determining an average x value of the pixels of the depth image that correspond to the user and are within the specified blob boundaries; and

determining an average y value of the pixels of the depth image that correspond to the user and are within the specified blob boundaries;

the average x value corresponds to an average horizontal position in the depth image of the extremity blob; and

the average y value corresponds to an average vertical position in the depth image of the extremity blob.

20. The one or more processor readable storage devices of claim 15, wherein:

the method further comprises performing a segmentation process to identify the plurality of pixels of the depth image that correspond to the user;

the depth image specifies, for each of the pixels corresponding to the user, a pixel location and a pixel depth; and

the pixel depth, specified for each of the pixels corresponding to the user, is indicative of a distance between a capture device, used to capture the depth image, and a portion of the user represented by the pixel.