Motion Capture Apparatus and Method, and Motion Capture Program

• US 20080112592A1
• Filed: 06/18/2007
• Published: 05/15/2008
• Est. Priority Date: 06/19/2006
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: WU, Weiguo, YAMAOKA, Keisuke, LIU, Yuyu
• Granted Patent: US 8,355,529 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: A61B 5/1113 Local tracking of patients, e.g. in a hospital or private home
• CPC Class Codes: A61B 5/1116 Determining posture transitions
• CPC Class Codes: A61B 5/1121 Determining geometric values, e.g. centre of rotation or angular range of movement
• CPC Class Codes: A61B 5/4528 Joints A61B5/4533, A61B5/4538 take precedence
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

A motion capture apparatus includes: generating means; predicting means; projecting means; evaluating means; and estimating means. The generating means generates a kinematics model having joints connected to one another. The predicting means hierarchically calculates predicted positions at a current time of the joints of the objective person, and predicts three-dimensional positions and postures of the joints and the limbs. The projecting means projects the three-dimensional positions and postures of the joints and the limbs of the objective person on a two-dimensional image plane, the three-dimensional positions and the postures being predicted by predicting means. The evaluating means evaluates reliability about projection positions projected by the projecting means based on the observation image of the objective person. The estimating means estimates the three-dimensional positions and postures of the joints and the limbs of the objective person at the current time based on an evaluation result about the reliability.

Claims

1. A motion capture apparatus for tracking three-dimensional positions and postures of joints, and limbs of an objective person, said motion capture apparatus comprising:

generating means for generating a kinematics model having joints connected to one another from an observation image of said objective person;

predicting means for hierarchically calculating predicted positions at a current time of the joints of said objective person in accordance with a constraint of coupling among the joints and predicted positions at a preceding time of the joints in said kinematics model, and predicting three-dimensional positions and pastures of the joints and said limbs of said objective person at the current time;

projecting means for projecting the three-dimensional positions and postures of the joints and the limbs of objective person at the current time on a two-dimensional image plane, the three-dimensional positions and postures being predicted by said predicting means;

evaluating means for evaluating reliability about projection positions projected by said projecting means in accordance with the observation image of said objective person; and

estimating means for estimating the three-dimensional positions and postures of the joints and the limbs of objective person at said current time in accordance with an evaluation result about said reliability.

2. The motion capture apparatus according to claim 1, further comprising:

distance image generating means for generating a distance image from said observation image;

face detecting means for detecting a position and a distance to a face region from said observation image;

mask generating means for generating a mash image with said objective person as a foreground in accordance with said distance image and the distance to the face;

color distribution calculating means for calculating a color histogram about said face region;

hand detection window generating means for generating hand detection windows in predetermined positions, respectively, in accordance with the position and the distance to said face region;

hand detecting means for detecting positions and distances to respective hands in accordance with the color histogram about said face region, the positions of said hand detection windows, and said mask image; and

joint position calculating means for calculating positions of the joints of said kinematics model in accordance with the position and the distance to said face, and the positions and the distances to said respective hands.

3. The motion capture apparatus according to claim 2, wherein said joint position calculating means sets media on a Y axis of the joints of elbows in a foreground region of said mask image as Y coordinates of the joints of the elbows, obtains an approximate straight line for upper arm portions in a state in which media on the Y axis of the upper arm portions in the foreground region of said mask image are set as Y coordinates of the upper arms, and calculates intersections between an X axis and joints of shoulders, and intersections between said approximate straight line and the joints of the shoulders as positions of the joints of the shoulders.

4. The motion capture apparatus according to claim 1, wherein said generating means estimates a parallax image of objective person from said observation image, and inversely projects the parallax image on a three-dimensional space to calculate three-dimensional positions and postures of the joints and the limbs of said objective person, generating kinematics model.

5. The motion capture apparatus according to claim 1, wherein said projecting means projects the three-dimensional positions and the postures of the joints and the limbs at the current time predicted by said predicting means by obtaining projection positions on the two-dimensional image corresponding to the positions in the three-dimensional space in accordance with a projection matrix obtained through camera calibration.

6. The motion capture apparatus according to claim 1, wherein said evaluating means calculates a similarity between image information on the observation image corresponding to said projection positions, and previously preserved image information on the joints and the limbs, thereby evaluating said reliability.

7. The motion capture apparatus according to claim 1, wherein said estimating means estimates the three-dimensional positions and the postures of the joints and the limbs of objective person at said current time in accordance with an evaluation value for the projection positions projected by said projecting means.

8. The motion capture apparatus according to claim 1, further comprising image observing means including at least, two or more cameras disposed so as to photograph the same objective person, and previously having a positional relationship among said at least two or more cameras, and camera parameters.

9. A motion capture method of tracking three-dimensional positions and postures of joints and limbs of an objective person, said motion capture method comprising the steps of;

generating a kinematics model having joints connected to one another from an observation image of said objective person;

hierarchically calculating predicted positions at a current time of the joints of said objective person in accordance with a constraint of coupling among the joints and predicted positions at a preceding time of the joints in said kinematics model, and predicting three-dimensional positions and postures of the joints and the limbs of said objective person at the current time;

projecting the three-dimensional positions and postures of the joints and the limbs of objective person at the current time on a two-dimensional image plane, the three-dimensional, positions and postures being predicted, in said predicting step;

evaluating reliability about projection positions projected in said projecting step in accordance with the observation image ox said objective person; and

estimating the three-dimensional positions and postures of the joints and the limbs of said objective person at said current time in accordance with an evaluation result about said reliability.

10. A motion capture program for executing processing for tracking three-dimensional positions and postures of joints and limbs of an objective person, said motion capture program comprising the steps of:

generating a kinematics model having joints connected to one another from an observation image of said objective person;

hierarchically calculating predicted positions at a current time of the joints of said objective person in accordance with a constraint of coupling among the joints and predicted positions at a preceding time of the joints in said kinematics model, and predicting three-dimensional positions and postures of the joints and the limbs of said objective person at the current time;

projecting the three-dimensional positions and postures of the joints and the limbs of objective person at the current time on a two-dimensional image plane, the three-dimensional positions and postures being predicted in said predicting step;

evaluating reliability about projection positions projected in said projecting step in accordance with the observation image of said objective person; and

estimating the three-dimensional positions and postures of the joints and the limbs of said objective person at said current time in accordance with an evaluation result about said reliability.

11. A motion capture apparatus for tracking three-dimensional positions and postures of joints, and limbs of an objective person, said motion capture apparatus comprising:

a generating portion configured to generate a kinematics model having joints connected to one another from art observation image of said objective person;

a predicting portion configured to hierarchically calculate predicted positions at a current time of the joints of said objective person in accordance with a constraint of coupling among the joints and predicted positions at a preceding time of the joints in said kinematics model, and predict three-dimensional positions and postures of the joints and said limbs of said objective person at the current time;

a projecting portion configured to project the three-dimensional positions and postures of the joints and the limbs of objective person at the current time on a two-dimensional, image plane, the three-dimensional positions and postures being predicted by said predicting portion;

a evaluating portion configured to evaluate reliability about projection positions projected by said projecting portion in accordance with the observation image of said objective person; and

an estimating portion configured to estimate the three-dimensional positions and postures of the joints and the limbs of objective person at said current time in accordance with an evaluation result about said reliability.