MONOCULAR TRACKING OF 3D HUMAN MOTION WITH A COORDINATED MIXTURE OF FACTOR ANALYZERS

• US 20070104351A1
• Filed: 10/26/2006
• Published: 05/10/2007
• Est. Priority Date: 10/28/2005
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Li, Rui, Yang, Ming-Hsuan
• Granted Patent: US 7,450,736 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 18/213 Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06F 18/21375 involving differential geometry, e.g. embedding of pattern manifold
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

Disclosed is a method and system for efficiently and accurately tracking three-dimensional (3D) human motion from a two-dimensional (2D) video sequence, even when self-occlusion, motion blur and large limb movements occur. In an offline learning stage, 3D motion capture data is acquired and a prediction model is generated based on the learned motions. A mixture of factor analyzers acts as local dimensionality reducers. Clusters of factor analyzers formed within a globally coordinated low-dimensional space makes it possible to perform multiple hypothesis tracking based on the distribution modes. In the online tracking stage, 3D tracking is performed without requiring any special equipment, clothing, or markers. Instead, motion is tracked in the dimensionality reduced state based on a monocular video sequence.

Claims

1. A method for tracking three-dimensional (3D) human motion comprising steps of:

receiving a two-dimensional (2D) image sequence, each image in the image sequence having first human motion data represented using a high dimensional space;

receiving a prediction model from an offline learning stage;

reducing the dimensionality of the first human motion data to generate second human motion data represented using a low dimensional space based at least in part on the prediction model; and

generating 3D tracking data based at least in part on the second human motion data and the 2D image sequence.

2. The method of claim 1 wherein the offline learning stage comprises steps of:

capturing 3D motion data;

processing the 3D motion data to extract pose information;

learning a dimensionality reduction model based on the extracted pose information adapted to bidirectionally map the pose information from the high dimensional space to the low dimensional space; and

generating a prediction model based at least in part on the dimensionality reduction model.

3. The method of claim 2 wherein learning the dimensionality reduction model includes forming clusters in the low dimensional space and modeling the clusters in the low dimensional space as Gaussian distributions.

4. The method of claim 2 wherein learning the dimensionality model comprises steps of:

computing local non-linear reconstruction weights to locally approximate a first pose state representation represented using the high dimensional space;

training a mixture of factor analyzers to locally reduce the dimensionality of the first pose state representation to form a second pose state representation represented using a locally coordinated low dimensional space;

determining a set of alignment parameters to map between the locally coordinated low dimensional space and a globally coordinated low dimensional space; and

determining a third pose state representation represented using a globally coordinated low dimensional space based on the second pose state representation and the alignment parameters.

5. The method of claim 2 wherein learning the dimensionality reduction model includes applying a Gaussian Process Latent Variable Model (GPLVM) algorithm.

6. The method of claim 2 wherein learning the dimensionality reduction model includes applying an Locally Linear Coordination (LLC) algorithm.

7. The method of claim 2 wherein the prediction model is further based on a learned dynamic model for at least one of the motions to be tracked.

8. The method of claim 1 wherein first human motion data includes a vector of joint angles.

9. The method of claim 1 wherein generating 3D tracking data comprises steps of:

generating a prediction including at least one predicted pose state at a future time frame based at least in part on the prediction model and at least one previous 2D image in the image sequence;

receiving a current 2D image from the 2D image sequence; and

updating the prediction based at least in part on the current 2D image.

10. The method of claim 9 wherein updating the prediction comprises selecting an optimal pose state of the at least one predicted pose states that best matches the current 2D image.

11. The method of claim 9 wherein generating 3D tracking data further comprises:

generating a prior probability density function based on the prediction model and at least one previous 2D image in the image sequence;

generating a likelihood function based on a matching cost between the at least one predicted pose state and the current 2D image; and

generating a posterior probability density function based on the prior probability density function and the likelihood function.

12. The method of claim 1 wherein the first human motion data includes motion having at least one of self-occlusion, motion blur, or large limb movements.

13. A system for tracking three-dimensional (3D) human motion comprising:

image receiving means for receiving a two-dimensional (2D) image sequence, each image in the image sequence having first human motion data represented using a high dimensional space;

prediction model receiving means for receiving a prediction model from an offline learning stage;

dimensionality reduction means for reducing the dimensionality of the first human motion data to generate second human motion data represented using a low dimensional space based at least in part on the prediction model; and

tracking means for generating 3D tracking data based at least in part on the second human motion data and the 2D image sequence.

14. The system of claim 13 wherein the offline learning stage comprises:

3D capture means for capturing 3D motion data;

processing means for processing the 3D motion data to extract pose information;

model learning means for learning a dimensionality reduction model based on the extracted pose information adapted to bidirectionally map the pose information from the high dimensional space to the low dimensional space; and

prediction model generating means for generating a prediction model based at least in part on the dimensionality reduction model.

15. The system of claim 14 wherein the model learning means for learning the dimensionality reduction model includes clustering means for forming clusters in the low dimensional space and modeling the clusters in the low dimensional space as Gaussian distributions.

16. The system of claim 14 wherein the model learning means comprises:

local approximation means for computing local non-linear reconstruction weights to locally approximate a first pose state representation represented using the high dimensional space;

local dimensionality reduction means for training a mixture of factor analyzers to locally reduce the dimensionality of the first pose state representation to form a second pose state representation represented using a locally coordinated low dimensional space;

alignment parameters determining means for determining a set of alignment parameters to map between the locally coordinated low dimensional space and a globally coordinated low dimensional space; and

global coordination means for determining a third pose state representation represented using a globally coordinated low dimensional space based on the second pose state representation and the alignment parameters.

17. The system of claim 14 wherein the model learning means for learning the dimensionality reduction model includes Gaussian Process Latent Variable Model (GPLVM) means for applying a GPLVM algorithm.

18. The system of claim 14 wherein the model learning means for learning the dimensionality reduction model includes Local Linear Coordination (LLC) means for applying a Locally Linear Coordination (LLC) algorithm.

19. The system of claim 14 wherein the prediction model is further based on a learned dynamic model for at least one of the motions to be tracked.

20. The system of claim 13 wherein first human motion data includes a vector of joint angles.

21. The system of claim 13 wherein the tracking means comprises:

prediction means for generating a prediction including at least one predicted pose state at a future time frame based at least in part on the prediction model and at least one previous 2D image in the image sequence;

current 2D image receiving means for receiving a current 2D image from the 2D image sequence; and

prediction updating means updating the prediction based at least in part on the current 2D image.

22. The system of claim 21 wherein the prediction updating means comprises selection means for selecting an optimal pose state of the at least one predicted pose states that best matches the current 2D image.

23. The system of claim 21 wherein the tracking means further comprises:

prior probability generating means for generating a prior probability density function based on the prediction model and at least one previous 2D image in the image sequence;

likelihood generating means for generating a likelihood function based on a matching cost between the at least one predicted pose state and the current 2D image; and

posterior probability generating means for generating a posterior probability density function based on the prior probability density function and the likelihood function.

24. The system of claim 13 wherein the first human motion data includes motion having at least one of self-occlusion, motion blur, or large limb movements.

25. A computer program product, comprising a computer readable medium storing computer executable code for tracking three-dimensional (3D) human motion, the computer executable code performing steps of:

receiving a two-dimensional (2D) image sequence, each image in the image sequence having first human motion data represented using a high dimensional space;

receiving a prediction model from an offline learning stage;

reducing the dimensionality of the first human motion data to generate second human motion data represented using a low dimensional space based at least in part on the prediction model; and

generating 3D tracking data based at least in part on the second human motion data and the 2D image sequence.