Apparatus and method for high-speed marker-free motion capture

• US 20040119716A1
• Filed: 05/28/2003
• Published: 06/24/2004
• Est. Priority Date: 12/20/2002
• Status: Active Grant

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: Kim, Do-Hyung, Lee, Ji Hyung, Nam, Seung Woo, Lee, In Ho, Park, Chang Joon, Baek, Seong Min
• Granted Patent: US 7,239,718 B2
• Time in Patent Office: Days
• US Class Current: 345/473
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/579 from motion

Abstract

The present invention relates to an apparatus and method for a high-speed marker-free motion capture, which is capable of powerfully detecting a body'"'"'s feature points corresponding to a body'"'"'s end portions such as a head, hands, feet, trunk, arms and legs at a high speed in an illumination change or background or noises of cameras. The extracted feature points of the body can be directly tracked stably in a 3-dimensional space. The position errors of the feature points due to the change of the illumination conditions or a shadow can be automatically corrected and the feature points can be stably tracked with respect to overlapping and disappearance of the feature points. Further, when coordinates of the middle joints are estimated using 3-dimensional coordinates of the extracted feature points of the body, the apparatus and method of the present invention restores a human model by estimating the positions of the middle joints of the actor with high reality without using motion database, thereby securing the stability and reality of the 3-dimensional motion data required in the motion capture.

Claims

1. An apparatus for a high-speed marker-free motion capture comprising:

a 2-dimensional feature point extraction module for obtaining an image with respect to a background with no actor by using a plurality of cameras having different viewing angles from each other to store the obtained image as RGB color space having a characteristic which easily emphasizes color difference and HSI color space having a characteristic which easily emphasizes color similarity, obtaining an image with respect to an initial posture of the actor to make blob models with respect to a head, hands, feet, trunk, arms and legs, extracting an adjacent connection relationship between the blob models and storing the extracted relationship, receiving the photographed image of the movement of the actor to extract pixels having the color different from the background, detecting pixels having color and topology similar to the stored blob models with respect to the respective parts of the body to make the detected pixels into the blobs, distinguishing the blobs corresponding to the parts of the body from the correlation of the adjacent blobs with respect to the detected blob, and determining the distinguished blobs as 2-dimensional body feature points;

a 3-dimensional feature point tracking module for performing 3-dimensionla matching with respect to the 2-dimensional feature points corresponding to the body'"'"'s end portions extracted by the 2-dimensional feature point extraction module, restoring 3-dimensional coordinates, and performing a tracking operation by predicting topology changes of the feature points and the body'"'"'s center of gravity; and

a 3-dimensional motion restoration module for predicting a position of a middle joint using the 3-dimensional coordinates of the 2-dimensional feature points tracked by the 3-dimensional feature point tracking module, generating 3-dimensional motion data, and restoring the 3-dimensional motion data to a human model.

2. The apparatus of claim 1, wherein the 2-dimensional feature point extraction module constructs the blob models with respect to the head, the hands, the feet, the trunk using color component information in the HSI color space having a characteristic that easily emphasizes the color similarity, a magnitude and a topology information with respect to the respective parts of the body.

3. The apparatus of claim 1, wherein in case when the 2-dimensional feature point extraction module receives the photographed image of the movement of the actor and extracts the pixels having the color different from the background, the 2-dimensional feature point extraction module extracts candidate pixels having the color different from the background through a comparison of the color components in the RGB color space having a characteristic which easily emphasizes the color difference.

4. The apparatus of claim 1, wherein the 2-dimensional feature point extraction module detects the pixels having color and topology similar to the stored blob models with respect to the respective parts of the body using the HSI color space having a characteristic which easily emphasizes the color similarity, and makes the detected pixels into the blobs.

5. The apparatus of claim 3, wherein the 2-dimensional feature point extraction module first carries out a comparison sequence with respect to the RGB color space which easily emphasizes the color difference to select candidate pixels, and transforms the candidate pixels into the HSI color space which easily emphasizes the color similarity to find pixels having color similar to the models of the respective parts of the body.

6. The apparatus of claim 5, wherein the 2-dimensional feature point extraction module selects the pixels having the color difference to separate the actor from the background, and retrieves the pixels similar to the models of the respective parts of the body.

7. The apparatus of claim 1, wherein the 2-dimensional feature point extraction module distinguishes the blobs corresponding to the parts of the body and determines the distinguished blobs as 2-dimensional body feature points, based on the correlation that the hands and arms are always adjacent to each other, the feet and legs are always adjacent to each other and the head can be attached to the arms or legs and the hands or the feet while the head are always attached to the trunk.

8. The apparatus of claim 1, wherein, using a topology change prediction filter, the 3-dimensional feature point tracking module restores the 3-dimensional coordinates by predicting a length, a width and a variation of rotation angles of a minimum circumscribed quadrangle, which surrounds 3-dimensional matched feature point area, as topology information to be inputted in a next frame by using information on sequence frames.

9. The apparatus of claim 1, wherein the 3-dimensional feature point tracking module predicts and calculates the movement of the center of the gravity in the movement up to the previous frame, and predicts the topology change of the feature points and the body'"'"'s center of the gravity by correctly tracking the feet using a prediction of the center of the gravity, which limits the movement of the foot to which the center of the gravity is applied.

10. The apparatus of claim 1, wherein the 3-dimensional motion restoration module sets boundary values with respect to the actor'"'"'s body as a method for calculating the most proper unique solution among a plurality of solutions to the joints generated by an inverse kinematics, and estimates positions of the middle joint of the actor by applying a limitation condition that there is no collision with the actor'"'"'s own body based on the boundary values and it is possible to pass through the body.

11. A method for a high-speed marker-free motion capture, comprising the steps of:

obtaining an image with respect to a background with no actor and storing the obtained image as two color spaces whose characteristics are different from each other using a plurality of cameras having different viewing angles from each other by a 2-dimensional feature point extraction module;

obtaining an image with respect to an initial posture of the actor using the cameras to make blob models with respect to a head, hands, feet, trunk, arms and legs, extracting an adjacent connection relationship between the blob models, and storing the extracted relationship by the 2-dimensional feature point extraction module;

receiving the photographed image of the movement of the actor to extract pixels having the color different from the background, detecting pixels having color and topology similar to the stored blob models with respect to the respective parts of the body, and making the detected pixels into the blob by the 2-dimensional feature point extraction module;

distinguishing the blobs corresponding to the parts of the body from the correlation of the adjacent blobs with respect to the detected blobs and determining the distinguished blobs as the 2-dimensional body feature points by the 2-dimensional feature point extraction module;

performing 3-dimensionla matching with respect to the 2-dimensional feature points corresponding to the body'"'"'s end portions extracted by the 2-dimensional feature point extraction module to restore 3-dimensional coordinates, and performing a tracking operation by predicting topology changes of the feature points and the body'"'"'s center of gravity by a 3-dimensional feature point tracking module; and

estimating the position of a middle joint using the 3-dimensional coordinates of the 2-dimensional feature points tracked by the 3-dimensional feature point tracking module to generate 3-dimensional motion data, and restoring the 3-dimensional motion data to a human model by a 3-dimensional motion restoration module.