EXTRACTION OF USER BEHAVIOR FROM DEPTH IMAGES

• US 20150347864A1
• Filed: 08/12/2015
• Published: 12/03/2015
• Est. Priority Date: 03/08/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hoof, Jonathan, Ahmad, Anis, Kennett, Daniel
• Granted Patent: US 9,311,560 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/44 : Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 40/103 : Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/23 : Recognition of whole body movements, e.g. for sport training

Abstract

Embodiments described herein use depth images to extract user behavior, wherein each depth image specifies that a plurality of pixels correspond to a user. In certain embodiments, information indicative of an angle and/or curvature of a user'"'"'s body is extracted from a depth image. This can be accomplished by fitting a curve to a portion of a plurality of pixels (of the depth image) that correspond to the user, and determining the information indicative of the angle and/or curvature of the user'"'"'s body based on the fitted curve. An application is then updated based on the information indicative of the angle and/or curvature of the user'"'"'s body. In certain embodiments, one or more average extremity positions of a user, which can also be referred to as average positions of extremity blobs, are extracted from a depth image. An application is then updated based on the average positions of extremity blobs.

Claims

1. A system that uses depth images to extract user behavior, comprising:

a capture device that obtains depth images;

a communication interface that receives depth images from the capture device;

one or more storage devices that store depth images;

a display interface; and

one or more processors in communication with the one or more storage devices and the display interface, wherein for each of a plurality of depth images the one or more processors are configured to

determine which pixels of the depth image correspond to a user;

identify a subset of pixels of interest from the pixels that correspond to the user;

fit a curve to the subset of pixels of interest to thereby produce a fitted curve comprising a plurality of straight line segments;

identify a first straight line that extends between endpoints of the fitted curve;

identify a second straight line that extends orthogonally from the first straight line to a point of the fitted curve that is farthest away from the first straight line; and

determine, based on the first and second straight lines, information indicative of a curvature of the user'"'"'s body.

2. The system of claim 1, wherein the one or more processors are also configured to use the information indicative of the curvature of the user'"'"'s body to update an application running on the system.

3. The system of claim 1, wherein the one or more processors are configured to:

determine a length of the first straight line that extends between endpoints of the fitted curve;

determine a length of the second straight line that extends orthogonally from the first straight line to the point of the fitted curve that is farthest away from the first straight line; and

determine a ratio of the length of the first straight line to the length of the second straight line;

wherein the ratio comprises the information indicative of the curvature of the user'"'"'s body.

4. The system of claim 1, wherein the subset of pixels of interest comprise pixels that correspond to a torso of the user.

5. The system of claim 1, wherein the subset of pixels of interest comprise pixels that correspond to a torso, legs and head of the user.

6. The system of claim 1, wherein the subset of pixels of interest comprise pixels that correspond to an upper peripheral portion of the user.

7. The system of claim 1, wherein the subset of pixels of interest comprise pixels that correspond to a lower peripheral portion of the user.

8. One or more processor readable storage devices having instructions encoded thereon which when executed cause one or more processors to perform a method for using depth images to extract user behavior, the method comprising:

receiving a depth image that that include a plurality of pixels that correspond to a user;

identifying a subset of pixels of interest from the pixels that correspond to the user;

fitting a curve to the subset of pixels of interest to thereby produce a fitted curve comprising a plurality of straight line segments;

identifying and determining a length of a first straight line that extends between endpoints of the fitted curve;

identifying and determining a length of a second straight line that extends orthogonally from the first straight line to a point of the fitted curve that is farthest away from the first straight line; and

determine a ratio of the length of the first straight line to the length of the second straight line;

wherein the ratio is indicative of a curvature of the user'"'"'s body.

9. The one or more processor readable storage devices of claim 8, wherein the method further comprises performing a segmentation process to identify the pixels in the depth image that correspond to the user.

10. The one or more processor readable storage devices of claim 8, wherein the subset of pixels of interest comprise one of the following:

pixels that correspond to a torso of the user;

pixels that correspond to a torso, legs and head of the user;

pixels that correspond to an upper peripheral portion of the user;

or

pixels that correspond to a lower peripheral portion of the user.

11. The one or more processor readable storage devices of claim 8, wherein the method further comprises using the information indicative of the curvature of the user'"'"'s body to update an application.

12. The one or more processor readable storage devices of claim 8, wherein the method further comprises determining information indicative of an angle of the user'"'"'s body by determining an angle, relative to a plane, of the first straight line extending between endpoints of the fitted curve, wherein the determined angle is indicative of an angle of the user'"'"'s body.

13. A method for extracting user behavior from a depth image that includes a plurality of pixels that correspond to a user, the method comprising:

identifying a pixel of the depth image that corresponds to an extremity of the user;

identifying an extremity blob that comprises pixels of the depth image that correspond to the user and are within a specified distance of the pixel identified as corresponding to the extremity of the user; and

identifying an average position of the extremity blob by determining an average position of the pixels included in the extremity blob.

14. The method of claim 13, wherein that average position of the extremity blob corresponds to one of a left side, a right side, a top, a bottom or a front of the user.

15. The method of claim 13, further comprising repeating the steps of the method of claim 8 for one or more further extremities of the user so that an average position of an extremity blob is determined for each of a plurality of extremities of the user.

16. The method of claim 15, wherein each said average position of an extremity blob corresponds to different one of a left side, a right side, a top, a bottom or a front of the user.

17. The method of claim 13, wherein the identifying a pixel of the depth image that corresponds to an extremity of the user comprises:

checking pixels in a line extending along a side of the depth image to determine if a pixel in the line corresponds to the user;

incrementally moving the line each time there is a determination that no pixel in the line corresponds to the user; and

in response to determining that a pixel in the line corresponds to the user, identifying the pixel in the line as corresponding to an extremity of the user.

18. The method of claim 13, wherein the method further comprises using the average position of the extremity blob to update an application.

19. The method of claim 13, wherein:

the identifying an extremity blob includes identifying pixels of the depth image that correspond to the user and are within specified blob boundaries; and

the identifying an average position of the extremity blob comprises

determining an average x value of the pixels of the depth image that correspond to the user and are within the specified blob boundaries; and

determining an average y value of the pixels of the depth image that correspond to the user and are within the specified blob boundaries;

the average x value corresponds to an average horizontal position in the depth image of the extremity blob; and

the average y value corresponds to an average vertical position in the depth image of the extremity blob.

20. The method of claim 13, further comprising:

using a capture device, located a distance from the user, to obtain the depth image; and

performing a segmentation process to identify the plurality of pixels of the depth image that correspond to the user;

wherein the depth image specifies, for each of the pixels corresponding to the user, a pixel location and a pixel depth; and

wherein the pixel depth, specified for each of the pixels corresponding to the user, is indicative of a distance between the capture device and a portion of the user represented by the pixel.