Scalable real-time motion recognition

• US 8,968,091 B2
• Filed: 03/02/2012
• Issued: 03/03/2015
• Est. Priority Date: 09/07/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Raptis, Michalis, Noble, Chuck, Pritchett, Joel, Hoppe, Hugues, Kirovski, Darko
• Primary Examiner(s): McClellan, James S
• Application Number:
  US13/410,644
• Publication Number:
  US 20120165098A1
• Time in Patent Office: 1,096 Days
• Field of Search: 463/36, 463/39, 463/42, 463/43, 463/30, 463/31, 463/32
• US Class Current: 463/32
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/814 Musical performances, e.g. by evaluating the player's ability to follow a notation
• CPC Class Codes: A63F 13/833 Hand-to-hand fighting, e.g. martial arts competition A63F13/837 takes precedence
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/5553 user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics

Abstract

Human body motion is represented by a skeletal model derived from image data of a user. The model represents joints and bones and has a rigid body portion. The sets of body data are scaled to a predetermined number of sets for a number of periodic units. A body-based coordinate 3-D reference system having a frame of reference defined with respect to a position within the rigid body portion of the skeletal model is generated. The body-based coordinate 3-D reference system is independent of the camera'"'"'s field of view. The scaled data and representation of relative motion within an orthogonal body-based 3-D reference system decreases the data and simplifies the calculations for determining motion thus enhancing real-time performance for multimedia applications controlled by a user'"'"'s natural movements.

Claims

1. A method of scalable real-time motion recognition and/or similarity analysis of human body motion based on skeletal model data derived from image data of a user comprising:

(a) detecting an object from data received by a capture device;

(b) determining whether the object detected in said step (a) corresponds to a human user;

(c) directly detecting positions of body parts of the user where it is determined in said step (b) that the detected object comprises a human user and the data comprises skeletal data, the data represented in a 3-D reference system of the capture device;

(d) transforming the data into a body-based coordinate 3-D reference system having a frame of reference of the user, the body-based coordinate 3-D reference system including a plurality of joints described as a single rigid body so that the plurality of joints do not move relative to each other; and

(e) outputting a computer model of the user based in part on body-based coordinate 3-D reference system.

2. The method of claim 1, further comprising the step (f) of temporally scaling the received skeletal data received in said step (c) by synchronizing the skeletal data to a predetermined number of sets for a number of periodic units wherein the periodic unit is rhythmic.

3. The method of claim 2, wherein the periodic unit is a beat of music.

4. The method of claim 1, wherein said step (d) of translating the data into a body-based coordinate 3-D reference system having a frame of reference of the user comprises the step of obtaining three orthogonal axes for the body-based 3-D coordinate reference system by performing a principal component analysis on a set of data points representing the rigid body portion of the skeletal model.

5. The method of claim 1, further comprising the step of translating the skeletal data from a body-based coordinate 3-D reference system to a 3-D camera space system, independent of the camera based on a respective transformation angle defined to transform each of the axes of one system to an axis of the other system.

6. The method of claim 5, wherein the model of the human body comprises a skeleton model of joints and bones including:

a torso as the rigid body portion,

a set of first degree joints in which each respective joint is attached to a bone, another end of which is attached to a joint in the rigid body torso, and

a set of second degree joints in which each respective joint is attached to a bone, another end of which is attached to one of the joints in the set of first degree joints.

7. The method of claim 6, further comprising representing motion of a joint in the set of first degree joints attached to the rigid body portion in a skeletal data frame by:

translating the frame of reference for the 3-D body-based coordinate system from an origin of the system on the rigid body to a joint of the rigid body to which an end of a bone attached to the first degree joint is attached;

determining the position of the first degree joint in the data frame using spherical coordinate system angles with respect to the body-based 3-D reference system;

and storing the position of the first degree joint in the skeletal data frame as the spherical coordinate system angles.

8. The method of claim 1, further comprising the step of comparing the output in said step (e) to an oracle of a predefined motion to determine if the output matches the predefined motion above some confidence threshold.

9. The method of claim 8, further comprising the step of determining on a per joint basis a distance between a joint indicated in the output of said step (e) and a corresponding joint in the oracle.

10. A system for recognizing human motion from skeletal data derived from image data comprising:

a camera for repeatedly capturing image data sets of a human body;

one or more processors communicatively coupled to the camera for receiving the image data sets and having access to a memory for storing the body data sets;

the one or more processors executing software for representing the human body in the image data sets as a human skeleton model of joints;

the one or more processors executing software for generating a body-based coordinate 3-D reference system having a frame of reference defined with respect to a position of a single rigid body;

the one or more processors executing software for directly identifying positions of first degree joints, adjacent the single rigid body, positions of the first degree joints being defined in terms of positions of the first degree joints in relation to a position of the single rigid body;

the one or more processors executing software for directly identifying positions of second degree joints, adjacent the first degree joints, a position of a second degree joint of the second degree joints being defined in terms of a position of the second degree joint in relation to positions of the first degree joint adjacent the second degree joint; and

the one or more processors executing software for determining a motion of at least one body part using the body-based 3-D coordinate reference system and the determined positions of the first and second degree joints.

11. The system of claim 10, further comprising:

the one or more processors executing software for determining the relationship of the body-based 3-D coordinate system to the fixed camera-based 3-D coordinate system for each set of the scaled skeletal data.

12. The system of claim 11, further comprising the one or more processors executing software for scaling the received skeletal data by synchronizing the sets of skeletal data to a predetermined number of sets for a number of periodic units.

13. The system of claim 10, wherein the model of the human body comprises a skeleton model of joints and bones including:

a torso as the rigid body portion,

a set of first degree joints in which each respective joint is attached to a bone, another end of which is attached to a joint in the rigid body, and

a set of second degree joints in which each respective joint is attached to a bone, another end of which is attached to a joint in the set of first degree joints.

14. The system of claim 13, wherein determining a motion of at least one body part using the body-based 3-D coordinate reference system further comprises defining for each skeletal data frame a set of three transformation angles for transforming the rigid body portion from the rigid body-based 3-D coordinate reference system to the fixed camera-based 3-D coordinate reference system;

and spherical coordinate system angles determined for each joint to represent its position using the rigid body-based 3-D coordinate reference system.

15. One or more computer readable storage media not consisting of transitory signals, the one or more computer readable storage media having encoded thereon instructions for causing at least one processor to perform a method for recognizing a gesture from image data, the method comprising:

(a) receiving sets of skeletal data determined by the at least one processor to represent a human body in a fixed camera-based three-dimensional (3-D) coordinate reference system having a frame of reference defined with respect to a point in a camera'"'"'s field of view;

(b) conforming the sets of skeletal data received in said step (a) to a body-based coordinate 3-D reference system having a frame of reference defined with respect to a position of a plurality of joints grouped together as a single rigid body portion of a skeletal model so that the plurality of joints do not move relative to each other, the body-based coordinate 3-D reference system being independent of the camera'"'"'s field of view, the skeletal model includes a model of joints and bones, wherein the skeletal model includes a torso, a set of first degree joints having positions defined by the at least one processor in terms of their relation to the torso, and a set of second degree joints adjacent the first degree joints, a second degree joint of the second degree joints having a position defined by the at least one processor in terms of its relation to its adjacent first degree joint;

(c) representing the body-based coordinate 3-D reference system by one of;

i) a plurality of angles that rotate the skeletal data back into the fixed camera-based 3-D coordinate reference system and ii) an orientation matrix;

(d) determining a motion of at least one body part using the plurality of angles that rotate the skeletal data back into the fixed camera-based 3-D coordinate reference system; and

(e) determining whether a gesture has been made by at least one body part based on the determined motion of the at least one body part.

16. The one or more computer readable storage media of claim 15, wherein the human skeleton model of joints and bones includes:

the torso as the rigid body portion,

the set of first degree joints in which each respective joint is attached to a bone, another end of which is attached to a joint in the rigid body torso, and

the set of second degree joints in which each respective joint is attached to a bone, another end of which is attached to one of the joints in the set of first degree joints.

17. The one or more computer readable storage media of claim 16, wherein determining a motion of at least one body part using the body-based 3-D coordinate reference system further comprises:

representing motion of a joint in the set of first degree joints attached to the rigid body portion in a skeletal data frame by;

translating the frame of reference for the 3-D body-based coordinate system from an origin of the system on the rigid body to a joint of the rigid body to which an end of a bone attached to the first degree joint is attached;

determining the position of the first degree joint in the data frame using spherical coordinate system angles with respect to the body-based 3-D reference system; and

storing the position of the first degree joint in the skeletal data frame as the spherical coordinate system angles.

18. The one or more computer readable storage media of claim 17, wherein determining a motion of at least one body part using the body-based 3-D reference system further comprises:

representing motion of a joint in the set of second degree joints in a skeletal data frame by

translating the torso rigid body-based 3-D orthogonal coordinate reference system to a first degree joint to which an end of a bone attached to the second degree joint is attached;

generating a first axis of a 3-D orthogonal body-based coordinate reference system with its origin at the first degree joint, the first axis being a bone vector generated by extending a bone attached to a joint of the rigid body at one end and which is attached at the other end to the first degree joint;

generating the second 3-D orthogonal coordinate system including the bone vector;

defining the bone vector as spherical north;

determining the position of the second degree joint in terms of spherical coordinate system angles defined with respect to the body-based 3-D reference system centered at the first degree joint; and

storing the position of the second degree joint in the skeletal data frame as the spherical coordinate system angles in a motion determination data set for the second degree joint.

19. The one or more computer readable storage media of claim 18, wherein the motion determination data set for each set of scaled skeletal data includes:

a set of three transformation angles for transforming the rigid body portion from the rigid body-based 3-D coordinate reference system to the fixed camera-based 3-D coordinate reference system; and

the spherical coordinate system angles determined for each joint to represent its position.

20. The one or more computer readable storage media of claim 19, wherein determining whether a gesture has been made by the at least one body part based on the motion of the at least one body part further comprises:

generating a set of candidate gestures by determining correlations

between data items of the motion determination data sets with model poses representing different gestures;

selecting a match gesture based on a criteria; and

determining a score representing a deviation of the user'"'"'s gesture from a gesture standard.