Center of mass state vector for analyzing user motion in 3D images

• US 9,142,034 B2
• Filed: 03/14/2013
• Issued: 09/22/2015
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hoof, Jonathan R., Kennett, Daniel G.
• Primary Examiner(s): Harandi, Siamak
• Application Number:
  US13/804,541
• Publication Number:
  US 20140270351A1
• Time in Patent Office: 922 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/66 of image moments or centre of gravity

Abstract

Techniques described herein determine a center of mass state vector based on a body model. The body model may be formed by analyzing a depth image of a user who is performing some motion. The center of mass state vector may include, for example, center-of-mass position, center-of-mass velocity, center-of-mass acceleration, orientation, angular velocity, angular acceleration, inertia tensor, and angular momentum. A center of mass state vector may be determined for an individual body part or for the body as a whole. The center of mass state vector(s) may be used to analyze the user'"'"'s motion.

Claims

1. A method, comprising:

accessing, by a processor, image data that tracks a user;

forming, by the processor, a body model for the user that includes a plurality of body parts based on the image data;

computing, processor, a center of mass for different ones of the body parts;

determining, by the processor, a center of mass state vector for each of the plurality of body parts, the center of mass state vector for each of the body parts comprises position of the center of mass of the body part, velocity of the center of mass of the body part, orientation of the body part, and angular velocity of the body part; and

determining, by the processor, a center of mass for the user as a whole based on the center of mass of the different body parts.

2. The method of claim 1, further comprising:

determining, by the processor, an inertia tensor for each of the body parts; and

determining, by the processor, an inertia tensor for the user as a whole based on the inertia tensor for each of the body parts.

3. The method of claim 1, wherein the forming a body model for the user that includes a plurality of body parts based on the image data comprises:

forming, by the processor, a model having a plurality of joints; and

adding, by the processor, geometric shapes between pairs of the plurality of joints, each of the geometric shapes corresponds to one of the body parts, wherein the computing a center of mass for different ones of the body parts comprises computing a center of mass for the respective geometric shapes.

4. The method of claim 1, further comprising:

determining, by the processor, a center of mass state vector for the user for a plurality of point in time, the center of mass state vector for the user comprises position of the center of mass for the user, velocity of the center of mass for the user, orientation of a rigid body for the user, angular velocity for the user;

determining, by the processor, a change in the center of mass state vector for the user between two points in time; and

determining, by the processor, a force that one or more constraint points would need in order to result in the change in the center of mass state vector for the user.

5. The method of claim 4, wherein the determining a force that one or more constraint points would need in order to result in the change in the center of mass state vector for the user comprises:

determining, by the processor, what foot forces are required to change the center of mass state vector for the user, including treating the user as a rigid body having feet that are the one or more constraint points.

6. The method of claim 1, further comprising:

determining, by the processor, change in the center of mass state vectors for each of the body parts between two points in time;

adding, by the processor, a set of joint constraints to the body model; and

computing, processor, constraint forces and/or torques to move the body parts to result in the change in the center of mass state vectors for the body parts between the two points in time.

7. The method of claim 1, further comprising:

determining a center of mass state vector for the user for the plurality of points in time based on the center of mass state vector for each of the plurality of body parts for the plurality of points in time, the center of mass state vector for the user comprises position of the center of mass for the user, velocity of the center of mass for the user, orientation of a rigid body for the user, and angular velocity for the user.

8. The method of claim 1, further comprising:

computing, by the processor, the center of mass for the different ones of the body parts for each of a plurality of points of time; and

determining, by the processor, the center of mass state vector for each of the plurality of body parts for each of the plurality of points of time.

9. A system comprising:

a camera that captures 3D image data that tracks a user;

a processor in communication with the camera, the processor is configured to;

access the 3D image data from the camera;

form a body model for the user that includes a plurality of body parts and plurality of joints based on the image data;

compute a center of mass for different ones of the plurality of body parts;

determine a center of mass state vector for each of the plurality of body parts, the center of mass state vector for each of the body parts comprises position of a center of mass of the body part, velocity of a center of mass of the body part, orientation of the body part, and angular velocity of the body part; and

determine a center of mass for the user as a whole based on the center of mass of the body parts.

10. The system of claim 9, wherein the processor is further configured to:

determine an inertia tensor for each of the body parts; and

determine an inertia tensor for the user as a whole based on the inertia tensor for each of the body parts.

11. The system of claim 9, wherein the processor being configured to form a body model for the user that includes a plurality of body parts and plurality of joints based on the image data comprises the processor being configured to:

form a skeletal model having the plurality of joints; and

add geometric shapes between pairs of the plurality of joints, each of the geometric shapes corresponds to one of the body parts, wherein the processor being configured to compute a center of mass for different ones of the body parts comprises the processor being configured to compute a center of mass for the respective geometric shapes.

12. The system of claim 9, wherein the processor is further configured to:

determine a center of mass state vector for the user for a plurality of frames of the image data, the center of mass state vector for the user comprises position of the center of mass for the user, velocity of the center of mass for the user, orientation of a rigid body for the user, angular velocity for the user;

determine a change in the center of mass state vector for the user between two of the frames; and

determine a force that one or more constraint points would need in order to result in the change in the center of mass state vector for the user.

13. The system of claim 12, wherein the processor being configured to determine a force that one or more constraint points would need in order to result in the change in the center of mass state vector for the user comprises the processor being configured to:

treat the user as a rigid body;

treat feet of the user as constraint points between the rigid body and a surface below the user; and

determine what forces of the feet are required to change the center of mass state vector for the user.

14. The system of claim 9, wherein the processor is configured to:

determine the center of mass state vector for each of the body parts for a plurality of frames of the image data.

15. The system of claim 14, wherein the processor is further configured to:

determine change in the center of mass state vectors for each of the body parts between two of the plurality of frames of the image data;

treat the plurality of joints as constraints; and

compute constraint forces and/or torques to move the body parts to result in the change in the center of mass state vectors for the body parts between the two frames.

16. A computer readable storage medium comprising processor readable code for programming a processor to:

access skeletal data that tracks a user exercising, the skeletal data includes a plurality of joints;

add geometric shapes between pairs of the joints;

compute a center of mass for different ones of the geometric shapes;

determine a center of mass state vector for each of the geometric shapes for a plurality of points in time, the center of mass state vector for each of the geometric shapes comprises position of a center of mass of the geometric shape, velocity of a center of mass of the geometric shape, orientation of the geometric shape, and angular velocity of the geometric shape;

determine a center of mass for the user as a whole for each of the plurality of points in time based on a summation of the center of mass of the different geometric shapes for each point in time;

determine a center of mass state vector for the user as a whole for the plurality of points in time based on the center of mass state vector for each of the geometric shapes for the plurality of points in time, the center of mass state vector for the user comprises position of the center of mass for the user, velocity of the center of mass for the user, orientation of a rigid body for the user, and angular velocity for the user;

determine a change in the center of mass state vector for the user between two points in time and

determine a force that one or more constraint points on the rigid body for the user would need in order to result in the change in the center of mass state vector for the user.

17. The computer readable storage medium of claim 16, wherein the processor readable code is further for programming the processor to:

determine an inertia tensor for each of the geometric shapes; and

determine an inertia tensor for the user as a whole based on the inertia tensor for each of the geometric shapes.

18. The computer readable storage medium of claim 16, wherein the geometric shapes are symmetric about an axis of the respective geometric shape.

19. The computer readable storage medium of claim 16, wherein the processor readable code for programming the processor to determine a force that one or more constraint points on the rigid body for the user would need in order to result in the change in the center of mass state vector for the user comprises processor readable code for programming the processor to:

treat feet of the user having feet that as the one or more constraint points on the rigid body, the feet are constraints between the rigid body and ground; and

determine what foot forces are required to change the center of mass state vector for the user as a whole.

20. The computer readable storage medium of claim 16, wherein the processor readable code is further for programming the processor to:

determine change in the center of mass state vectors for each of the geometric shapes between two points in time;

treat the plurality of joints as constraints; and

compute constraint forces to move the geometric shapes to result in the change in the center of mass state vectors for the geometric shapes between the two points in time.