System for reconstructing the 3-dimensional motions of a human figure from a monocularly-viewed image sequence

• US 6,115,052 A
• Filed: 02/12/1998
• Issued: 09/05/2000
• Est. Priority Date: 02/12/1998
• Status: Expired due to Fees

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Information Technology Center America, Inc. (Mitsubishi Electric Corporation)
• Inventors: Freeman, William T., Leventon, Michael E.
• Primary Examiner(s): Powell, Mark R.
• Assistant Examiner(s): NGUYEN, THU V
• Application Number:
  US09/023,148
• Time in Patent Office: 936 Days
• Field of Search: 345/419, 345/473, 345/474, 345/475, 382/107, 382/159, 382/165, 382/171, 382/174, 382/154, 382/209, 382/220
• US Class Current: 345/473
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings

Abstract

A system is provided for reconstructing the 3-dimensional motions of a hu figure from a monocularly viewed image sequence in which a statistical approach is used coupled with the use of a set of motion capture examples to build a gaussian probability model for short human motion sequences. In a simplified rendering domain, this yields an optimal 3-d estimate of human motion used to create realistic animations given a 2-d temporal sequence. The subject system is also useful to identify which motion modes are difficult to estimate. In one embodiment, a stick figure is overlaid on the input video image to allow manual correction of incorrectly tracked body parts. The utilization of stick figure correction permits manual correction for a more accurate 3-d reconstruction of the motion depicted in the input video sequence. An interactive tracking system processes real video sequences, and achieves good 3-d reconstructions of the human figure motion.

Claims

1. A system for reconstructing 3-dimensional motion of a human figure from a monocularly viewed image sequence of a person, comprising:

means for digitizing said monocularly viewed image sequence;

means for providing 3-d motion training data including representative motions and markers therefor as a training image sequence;

means for grouping said 3-d motion training data and said digitized image sequence into temporal segments, each temporal segment including a predetermined number of frames;

means, coupled to said means for digitizing and said means for grouping, for optimizing said temporal segments of said 3-d motion training data according to said markers so as to find a linear combination of said temporal segments of said 3-d motion training data corresponding to said monocularly viewed image sequence.

2. The system of claim 1, and further including

means for generating a 2-d stick figure corresponding to said estimated 3-d coordinates,

means for overlaying said 2-d stick figure on said digitized image sequence;

means for displaying a result thereof,

a user input device and

means, responsive to said user input device, for correcting the result, and wherein said means for optimizing includes means for correcting said estimated 3-d coordinates in accordance with said corrected result.

3. The system of claim 1, wherein said optimizing means includes means for implementing ##EQU5## where E(α

) is the energy function to be minimized by the optimal coefficients α

to be found, R is the vector of sensor responses over time from the image data, where the function f converts α

body motion coefficients to predicted sensor responses, where I_i is the ith point position specified by the user, and P_i projects the α

coefficients onto the corresponding ith stick figure part 2-d position, where Λ

^-1 is an inverse of a gaussian probability distribution of a covariance, and where λ

₁ and λ

₂ are constants which reflect the weights of the image data, the priors over human motions, and the interactively specified 2-d point matches.

4. The system of claim 3, wherein each motion segment has a number of frames and further including means for substituting ##EQU6## for (R-f(α

))², where t is an index of the frame time of each frame of the motion segment, I_i is a vector of the image intensities at time t at the 2-d position of its vector argument, and M(α

) is the 2-d positions of each marker for the human motion represented by the linear combination of training segments given by the combination coefficients α

, to minimize image intensities at the marker positions from frame to frame and to encourage said character model to track the motion of said person.

5. The system of claim 1 including means for estimating motion segments sequentially by overlapping selected motion segments to obtain multiple estimated 3-d coordinates for a given period of time, thus to provide 3-d reconstruction for a predetermined frame by computing the average of estimates over all overlapping motion segments.

6. The system of claim 1, wherein said temporal segments are divided into subsets, said optimizing means including means for selecting those subsets that are closest to the current best estimate of the segment to be estimated by said optimizing means, thereby to avoid overfitting and thus improve the estimation of said estimated 3-d coordinates.

7. The system of claim 1 further comprising:

means for outputting estimated 3-d coordinates of the corresponding markers;

means for fitting a character model to the estimated 3-d coordinates of said markers in said linear combination of said temporal segments of said 3-d motion training data; and

means for displaying said character model and motion thereof.

8. A method for estimating 3-d motion of a moving input figure in a sequence of input frames;

comprising the step of;

providing a sequence of training frames, the training frames including a moving training figure, the moving training figure including markers;

grouping the training frames into a plurality of training segments, each training segment including a predetermined number of temporally adjacent frames of the sequence;

determining 3-d motion of the training figure for each training segment from the markers;

grouping the input frames into a plurality of input segments, each input segment including a predetermined number of temporally adjacent input frames;

locating a linear combination of training segments that substantially corresponds to the sequence of input segments; and

extracting the 3-d motions of the training figure in the linear combination of training segments to estimate the 3-d motion of the moving input figure.

9. The method of claim 8 further comprising:

fitting a character model to the estimated motion of the moving input figure; and

displaying the moving character model.

10. The method of claim 8 further comprising:

fitting a stick figure to the estimated motion of the moving input figure; and

correcting the estimated motion of the input figure by manually manipulating the fit of the stick figure.