Extraction of user behavior from depth images

• US 9,311,560 B2
• Filed: 08/12/2015
• Issued: 04/12/2016
• Est. Priority Date: 03/08/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hoof, Jonathan, Ahmad, Anis, Kennett, Daniel
• Primary Examiner(s): GORADIA, SHEFALI DINESH
• Application Number:
  US14/824,869
• Publication Number:
  US 20150347864A1
• Time in Patent Office: 244 Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/44 : Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 40/103 : Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/23 : Recognition of whole body movements, e.g. for sport training

Abstract

Embodiments described herein use depth images to extract user behavior, wherein each depth image specifies that a plurality of pixels correspond to a user. In certain embodiments, information indicative of an angle and/or curvature of a user'"'"'s body is extracted from a depth image. This can be accomplished by fitting a curve to a portion of a plurality of pixels (of the depth image) that correspond to the user, and determining the information indicative of the angle and/or curvature of the user'"'"'s body based on the fitted curve. An application is then updated based on the information indicative of the angle and/or curvature of the user'"'"'s body. In certain embodiments, one or more average extremity positions of a user, which can also be referred to as average positions of extremity blobs, are extracted from a depth image. An application is then updated based on the average positions of extremity blobs.

Claims

1. A system that uses depth images to extract user behavior, comprising:

a communication interface that receives depth images;

one or more storage devices that store depth images;

a display interface; and

one or more processors in communication with the one or more storage devices and the display interface, wherein for each of a plurality of depth images the one or more processors are configured to

determine which pixels of the depth image correspond to a user;

identify a subset of pixels of interest from the pixels that correspond to the user;

fit a curve to the subset of pixels of interest to thereby produce a fitted curve comprising a plurality of straight line segments;

identify a first straight line that extends between endpoints of the fitted curve;

identify a second straight line that extends orthogonally from the first straight line to a point of the fitted curve that is farthest away from the first straight line; and

determine, based on the first and second straight lines, information indicative of a curvature of the user'"'"'s body;

wherein the one or more processors are also configured to use the information indicative of the curvature of the user'"'"'s body to update an application running on the system.

2. The system of claim 1, wherein the one or more processors are configured to:

determine a length of the first straight line that extends between endpoints of the fitted curve;

determine a length of the second straight line that extends orthogonally from the first straight line to the point of the fitted curve that is farthest away from the first straight line; and

determine a ratio of the length of the first straight line to the length of the second straight line;

wherein the ratio comprises the information indicative of the curvature of the user'"'"'s body.

3. The system of claim 1, wherein the subset of pixels of interest comprise pixels that correspond to a torso of the user.

4. The system of claim 1, wherein the subset of pixels of interest comprise pixels that correspond to a torso, legs and head of the user.

5. The system of claim 1, wherein the subset of pixels of interest comprise pixels that correspond to an upper peripheral portion of the user.

6. The system of claim 1, wherein the subset of pixels of interest comprise pixels that correspond to a lower peripheral portion of the user.

7. The system of claim 1, further comprising:

a capture device that obtains the depth images;

wherein the communication interface receives the depth images from the capture device.

8. The system of claim 7, wherein the capture device includes at least one camera.

9. The system of claim 7, wherein the capture device includes at least two cameras.

10. The system of claim 7, wherein the capture devices includes a three-dimensional camera.

11. The system of claim 1, wherein the one or more processors are configured to use the information indicative of the curvature of the user'"'"'s body to determine whether the user is in a position or pose that the user is instructed to position themselves, and to provide feedback about the position or pose of the user.

12. The system of claim 1, wherein the one or more processors are configured to use the information indicative of the curvature of the user'"'"'s body to determine whether the user is in a position or pose that the user is instructed to position themselves, and to provide feedback about the position or pose of the user.

13. The system of claim 1, wherein the one or more processors are configured to use the information indicative of the curvature of the user'"'"'s body to update a position or pose of an avatar of the user that is displayed using the display interface.

14. One or more processor readable storage devices having instructions encoded thereon which when executed cause one or more processors to perform a method for using depth images to extract user behavior, the method comprising:

receiving a depth image that that include a plurality of pixels that correspond to a user;

identifying a subset of pixels of interest from the pixels that correspond to the user;

fitting a curve to the subset of pixels of interest to thereby produce a fitted curve comprising a plurality of straight line segments;

identifying and determining a length of a first straight line that extends between endpoints of the fitted curve;

identifying and determining a length of a second straight line that extends orthogonally from the first straight line to a point of the fitted curve that is farthest away from the first straight line; and

determine a ratio of the length of the first straight line to the length of the second straight line, wherein the ratio is indicative of a curvature of the user'"'"'s body; and

using the information indicative of the curvature of the user'"'"'s body to update an application.

15. The one or more processor readable storage devices of claim 14, wherein the method further comprises performing a segmentation process to identify the pixels in the depth image that correspond to the user.

16. The one or more processor readable storage devices of claim 14, wherein the subset of pixels of interest comprise one of the following:

pixels that correspond to a torso of the user;

pixels that correspond to a torso, legs and head of the user;

pixels that correspond to an upper peripheral portion of the user;

or

pixels that correspond to a lower peripheral portion of the user.

17. The one or more processor readable storage devices of claim 14, wherein the method further comprises determining information indicative of an angle of the user'"'"'s body by determining an angle, relative to a plane, of the first straight line extending between endpoints of the fitted curve, wherein the determined angle is indicative of an angle of the user'"'"'s body.

18. The one or more processor readable storage devices of claim 14, wherein the method includes using the information indicative of the curvature of the user'"'"'s body to determine whether the user is in a position or pose that the user is instructed to position themselves, and to provide feedback about the position or pose of the user.

19. The one or more processor readable storage devices of claim 14, wherein the method include using the information indicative of the curvature of the user'"'"'s body to determine whether the user is in a position or pose that the user is instructed to position themselves, and to provide feedback about the position or pose of the user.

20. The one or more processor readable storage devices of claim 14, wherein the method includes using the information indicative of the curvature of the user'"'"'s body to update a position or pose of an avatar of the user that is displayed.