System and method for modeling three dimensional objects from a single image

• US 20050031195A1
• Filed: 08/08/2003
• Published: 02/10/2005
• Est. Priority Date: 08/08/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Liu, Zicheng
• Granted Patent: US 7,218,774 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06V 40/169 Holistic features and representations, i.e. based on the facial image taken as a whole

Abstract

A system and method for automatically modeling a three dimensional object, such as a face, from a single image. The system and method according to the invention can construct one or more three dimensional (3D) face models using a single image. Although use of a frontal image simplifies the processing, the system and method according to the invention can also generate a 3D model from a non-frontal image (e.g., an image where the object to be modeled has an out of plane rotation, such a face that is turned to one side to some extent). The system is fully automatic. It is fast compared to the other modeling systems. Furthermore, it is very robust. It can be used to construct personalized models for games, on-line chat, and so on. It can also be used as a tool to generate a database of faces with various poses which are needed to train most face recognition systems.

Claims

1. A computer-implemented process for generating a three dimensional model of an object from a single image, comprising the process actions of:

obtaining an image of an object;

identifying at least one object to be modeled in the image;

determining features of the object in the image; and

aligning the features of a generic model of the identified object with the features of the object in the image to obtain a tailored model.

2. The computer-implemented process of claim 1 further comprising the process action of:

applying texture of the object in the image to the tailored generic model.

3. The computer-implemented process of claim 1, further comprising the process actions of:

determining if the object in the image is rotated out of the plane of the image prior to using the features of the object in the image to align the features of a generic model of the object with the features of the object in the image to obtain a tailored model;

determining the amount of out of plane rotation of the object in the image; and

using the amount of rotation of the object in the image to rotate the generic model to match the input image.

4. The computer-implemented process of claim 1 wherein the process action of determining the features of the object comprises determining the features by at least one of the following:

using a pattern recognition technique to locate the features;

an edge detection technique to locate the features; and

using a generic model of the object to locate the features.

5. The computer-implemented process of claim 1 wherein the generic model of the object is represented as a linear combination of a neutral object, and some number of object metrics, wherein an object metric is a vector that linearly deforms the object in a certain way.

6. The computer-implemented process of claim 4 wherein the linear combination of a neutral object is represented by a mesh of vertices and triangles that represent the shape of the object.

7. The computer-implemented process of claim 4 wherein the process action of aligning the feature of a generic model of the object to the features of the object in the image to obtain a tailored model comprises modifying the object metrics so that the generic model matches the shape of the object in the image.

8. The computer-implemented process of claim 1 wherein the object modeled is a face.

9. The computer-implemented process of claim 1 wherein the object modeled is a human body.

10. The computer-implemented process of claim 1 wherein the object modeled is an animal.

11. The computer-implemented process of claim 1 wherein multiple objects are modeled by repeating the following process actions for more than one object:

identifying at least one object to be modeled in the image;

determining features of the object in the image; and

aligning the features of a generic model of the identified object with the features of the object in the image to obtain a tailored model.

12. A system for creating a face model from the input of a single image of a face, the system comprising:

a general purpose computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

input a single image of a face;

use a face detector to find the face to be modeled in the image;

determine features of the face to be modeled;

align features of a generic face model to features of the face in the image to obtain a tailored face model; and

apply the texture of the face in the image to the tailored generic face model.

13. The system of claim 12 further comprising modules to:

determine if the face in the image is rotated out of the plane of the image prior to using the features of the face in the image to align the features of a generic face model with the features of the face;

determine the amount of out of plane rotation of the face in the image; and

use the amount of rotation of the face in the image to rotate the generic face model to match the face in the input image.

14. The system of claim 12 wherein the features of the face to be modeled comprise two eyebrows, two eyes, a nose, a mouth and two sides of a face.

15. The system of claim 12 wherein the generic model of a face is represented as a linear combination of a neutral face, and some number of face metrics, wherein a face metric is a vector that can linearly deform the neutral face model in a certain way.

16. The system of claim 15 wherein a generic model of a face is represented by a vector S=(v₁^T, . . . ,v_n^T)^T where v_i=(X_i,Y_i,Z_i)^T (i=1, . . . , n) are the vertices, and a metric is represented by a vector M=(δ

v₁, . . . , δ

v_n)^T, where δ

v₁=(δ

X_i,δ

Y_i,δ

Z_i)^T. and wherein a neutral face S^o=(v₁^0T, . . . , v_n^0T)^T, and a set of m metrics M^j=(δ

v₁^jT, . . . , δ

v_n^jT)^T, and wherein the linear space of face geometries spanned by these metrics is

$S=S^0+\underset{j=1}{\overset{m}{\sum <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>c_j<br><br>M^j<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{subject}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{to}<br><br>\text{\hspace{1em}<br><br>}<br><br>c_j\in <br><br>\left[l_j,u_j\right]$

where the c_j'"'"'s are the metric coefficients, and l_j and u_j are the valid range of c_j.

17. The system of claim 15 wherein the linear combination of a neutral face is represented by a mesh of vertices and triangles that represent the shape of the face.

18. The system of claim 15 wherein using the features of the object to be modeled to align the features of a generic face model to the feature points of the face in the image comprises modifying the face metrics so that the generic face model matches the shape of the face in the image.

19. The system of claim 15 wherein using the features of the face to be modeled to align the features of a generic face model to the features of the face in the image comprises modifying the face metrics so that the generic face model matches the shape of the face in the image.

20. The system of claim 19 wherein modifying the metrics comprises:

calculating the differences between the generic face model and the face in the image; and

adjusting the generic face model to represent the face in the image by using said calculated differences to adjust the generic face model.

21. The system of claim 17 wherein the module for applying texture comprises:

associating each vertex on the generic face model with its coordinates in the image; and

using said associated coordinates of the vertices with the image to apply said texture of the original image.

22. A computer-readable medium having computer-executable instructions for creating a three-dimensional model of an object in a single image, said computer executable instructions comprising:

inputting a single image of a three-dimensional object;

creating a generic linear space representation of the three-dimensional object depicted in the image;

estimating object pose and linear coefficients describing the three-dimensional object in the image in the linear space using an orthogonal projection;

using the estimated pose and linear coefficients to obtain a linear space model of the object in the image; and

applying texture information of the three-dimensional object in the image to the linear space model of the object.

23. The computer-readable medium of claim 22 wherein said computer-executable instruction for creating a generic linear space representation of the three-dimensional object depicted in the image, comprises sub-instructions for:

using an object detector to find the object to be modeled in the image;

determining features of the object; and

using the features of the object to be modeled to align the features of a generic linear space representation of the object to the shape of the object in the image.

24. The computer-readable medium of claim 22 further comprising a computer-executable instruction for changing the characteristics of the linear space model of the object by changing the shape of the linear space model of the object in the image underlying the texture.