METHOD AND SYSTEM FOR MARKERLESS MOTION CAPTURE USING MULTIPLE CAMERAS

• US 20090232353A1
• Filed: 11/09/2007
• Published: 09/17/2009
• Est. Priority Date: 11/10/2006
• Status: Active Grant

• Current Assignee: University of Maryland, College Park
• Original Assignee: University of Maryland
• Inventors: SUNDARESAN, ARAVIND, CHELLAPPA, RAMALINGAM
• Granted Patent: US 8,023,726 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/20044 Skeletonization; Medial axis transform
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06T 7/596 from three or more stereo images
• CPC Class Codes: G06V 10/426 Graphical representations
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

Completely automated end-to-end method and system for markerless motion capture performs segmentation of articulating objects in Laplacian Eigenspace and is applicable to handling of the poses of some complexity. 3D voxel representation of acquired images are mapped to a higher dimensional space (^k), where k depends on the number of articulated chains of the subject body, so as to extract the 1-D representations of the articulating chains. A bottom-up approach is suggested in order to build a parametric (spline-based) representation of a general articulated body in the high dimensional space followed by a top-down probabilistic approach that registers the segments to an average human body model. The parameters of the model are further optimized using the segmented and registered voxels.

Claims

1. A method for markerless motion capture, comprises the steps of:

acquiring video sequences of a subject by a plurality of cameras positioned in a surrounding relationship with the subject;

coupling the acquired video sequences to a data processing unit to be processed therein;

in said data processing unit, for each frame of a set of frames of said acquired video sequences,

(a) constructing a three-dimensional (3-D) volumetric data (voxel) representation of the subject'"'"'s image, said 3-D voxel representation forming a kinematic structure including a plurality of articulated chains connected at respective joints and composed of said 3-D voxels,

(b) transforming said 3-D voxels of said kinematic structure to the Laplacian Eigenspace, each of said 3-D voxels being mapped to a respective one of a plurality of smooth 1-dimensional (1-D) curves in the Laplacian Eigenspace according to a position of said each 3-D voxel in said 3-D voxel representation,

(c) segmenting said 1-D smooth curves by 1-D spline fitting in the Laplacian Eigenspace,

(d) registering said segmented 1-D smooth curves to a graphical subject model,

(e) computing a skeleton curve of the subject from said segmented 1-D smooth curves,

(f) estimating a stature of the subject from said skeleton curves computed from said set of frames of said acquired video sequences,

(g) constructing an initial skeleton model having skeleton parameters corresponding to the stature of the subject,

(h) computing pose parameters of said initial skeleton model, and

(i) constructing an initial subject model based on said initial skeleton model parameters and said pose parameters thereof, and displaying said initial subject model at a display unit.

2. The method of claim 1, wherein said subject model is a super-quadric subject model.

3. The method of claim 2, further comprising the step of tracking said initial subject model performed after said step (i) to compute the pose of the subject at time (t+1) based on the pose parameters at time t.

4. The method of claim 1, wherein said step (a) comprises the steps of:

for each camera (j) of said plurality of cameras,

subtracting background data from said subject image to obtain the subject silhouette,

determining a set of 2-dimensional (2-D) points x_i^(j) for the camera (j) falling within said silhouette,

populate array s^j with said points x_i^(j),

adding said arrays s^j for said plurality of cameras to obtain a combined array, and

thresholding said combined array to obtain 3-D voxel representation of said subject image.

5. The method of claim 4, further comprising the steps of:

after said step (a), constructing an adjacency matrix based on the connectivity between said 3-D voxels of said subject image, and computing a connected graph of said kinematic structure based on said adjacency matrix.

6. The method of claim 1, wherein said step (b) further comprises the steps of:

mapping said each 3-D voxel of said kinematic structure to said respective 1-D curve in the Laplacian Eigenspace in accordance with a position of said each 3-D voxel along a respective one of said plurality of articulated chains.

7. The method of claim 1, wherein said step (c) further comprises the steps of:

segmenting said 3-D voxels into respective articulated chains by associating each said 3-D voxel with a respective one of said 1-D smooth curves in the Laplacian Eigenspace (LE) by analyzing mutual orientation of said 1-D smooth curves in the LE.

8. The method of claim 1, wherein said step (c) further comprises the steps of:

classifying said articulated chains into two distinct types depending on the number of connected ends thereof,

initiating a spline fit by identifying an initial mode y₀ on a respective articulated chain,

forming a set of nodes N₀ closest to said initial body y₀,

determining at least one principal axis for said articulated chains, a number of said at least one principal axes being a subject to the type of said articulated chains,

growing said 1-D spline in the direction of each of said at least one principal axis,

spline fitting by projecting node y_i on the respective principal axis, and

computing a 1-D spline.

9. The method of claim 8, wherein said step (c) further comprises the step of:

computing said 1-D spline by calculating a spline fit error at node y_i of said 1-D smooth curve in the LE to said y₀ to obtain proximity value of said node y_i to said y₀, and

minimizing said spline fit error.

10. The method of claim 9, further comprises the steps of:

in said step (c), propagating the computed said 1-D spline by adding nodes closest to a growing end of said computed 1-D spline,

defining outlier nodes for which the spline fit error exceeds a predetermined threshold, and

terminating the spline growing when the number of the outlier odes exceeds a predetermined value.

11. The method of claim 10, wherein in said step (e), a body graph is computed describing connections between the ends of said 1-D splines after said skeleton curve computation.

12. The method of claim 1, wherein in said step (d), said segmented 1-D smooth curves are registered by probabilistic registration.

13. The method of claim 1, wherein said subject is a human.

14. A system for markerless motion capture, comprising:

a plurality of cameras disposed in surrounding relationship with a subject,

a data processing unit coupled to said plurality of cameras to receive video sequences therefrom, said data processor unit processing said video sequences and building a subject model, said subject model being presented on a display unit,

wherein said data processor unit includes;

(a) a voxel constructing unit constructing a 3-dimensional (3-D) volumetric data (voxel) representation of the subject image from said video sequences acquired by said plurality of cameras, said 3-D voxel representation including a kinematic structure composed of a plurality of articulated chains connected at respective joints,

(b) a Laplacian Eigenspace (LE) transformation unit mapping the 3-D voxel representation to the LE to form a plurality of smooth 1-dimensional (1-D) curves in the LE,

(c) segmenting unit for segmenting the 1-D curves in the LE,

(d) model unit constructing a graphical model of the subject,

(e) registration unit registering said segmented 1-D curves to the graphical model,

(f) skeleton curve computing unit computing skeleton curve of the subject from said segmented 1-D curves, and

(g) estimation unit estimating parameters of said subject model and the subject pose parameters.

15. The system of claim 14, further comprising a pose tracking unit coupled to said estimation unit to compute the pose of the subject at a time (t+1) based on the pose parameters at a time t.

16. The system of claim 14, wherein said model unit further constructs said subject model as a super-quadric subject model.

17. The system of claim 14, wherein said voxel constructing unit for each camera (j) of said plurality of cameras,

subtracts background data from said subject image to obtain the subject silhouette,

determines a set of 2-dimensional (2-D) points x_i^(j) for the camera (j) falling within said silhouette,

populates an array s^j with said points x_i^(j),

adds said arrays s^j for said plurality of cameras to obtain a combined array of said subject image, and

thresholds said combined array to obtain said 3-D voxel representations.

18. The system of claim 14, wherein said LE transformation unit maps each voxel of said kinematic structure to said respective smooth 1-D curves in the LE in accordance with the position of said each voxel along a respective one of said plurality of articulated chains.

19. The system of claim 14, wherein said segmenting unit

classifies said articulated chains into two distinct types depending on the number of connected ends thereof,

initiates a spline fit by identifying an initial node y₀ on a respective articulated chain,

forms a set of nodes N₀ closest to said initial node y₀,

determines respective principal axes for said articulated chains,

grows each of said 1-D spline in the direction of the respective principal axis,

spline fits by projecting node y_i on the respective principal axis,

computes a 1-D spline by minimizing spline fit error at node y_i of said 1-D smooth curve in the LE to said y₀,

propagates the computed 1-D spline by adding nodes closest to a growing end of said computed 1-D spline,

defines outlier nodes for which the spline fit error exceeds a predetermined threshold, and

terminates the spline growing when the number of the outlier nodes exceeds a predetermined value.

20. The system of claim 14, wherein said estimation unit

estimates a stature of the subject from the skeleton curves computed from a set of frames of the acquired video sequences,

constructs an initial skeleton model having skeleton parameters corresponding to the stature of the subject,

computes pose parameters of said initial skeleton model,

constructs an initial subject model based on said initial skeleton model parameters and said pose parameters thereof, and

reiteratively refines the subject model.