3D face reconstruction from 2D images

• US 7,856,125 B2
• Filed: 01/30/2007
• Issued: 12/21/2010
• Est. Priority Date: 01/31/2006
• Status: Expired due to Fees

• Current Assignee: University of Southern California
• Original Assignee: University of Southern California
• Inventors: Fidaleo, Douglas, Medioni, Gerard
• Primary Examiner(s): Ahmed; Samir A.
• Assistant Examiner(s): PATEL, JAYESH A
• Application Number:
  US11/669,099
• Publication Number:
  US 20070183653A1
• Time in Patent Office: 1,421 Days
• Field of Search: 382/206, 382/294, 382/118, 382/15
• US Class Current: 382/118
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions

Abstract

A 3D face reconstruction technique using 2D images, such as photographs of a face, is described. Prior face knowledge or a generic face is used to extract sparse 3D information from the images and to identify image pairs. Bundle adjustment is carried out to determine more accurate 3D camera positions, image pairs are rectified, and dense 3D face information is extracted without using the prior face knowledge. Outliers are removed, e.g., by using tensor voting. A 3D surface is extracted from the dense 3D information and surface detail is extracted from the images.

Claims

1. A method for automatically reconstructing a 3D face from a plurality of 2D images of a human face, comprising:

deriving initial camera position estimates using prior face knowledge of a generic face;

selecting image pairs representing corresponding images from the plurality of 2D images that define similar poses and extracting sparse feature points representing perimeters of a face or locations of facial features for each of said image pairs;

refining said initial camera position estimates and said sparse feature points;

extracting dense 3D point clouds from said image pairs by performing dense feature detection and matching across said image pairs using a purely data-driven approach that does not use prior face knowledge or generic faces, wherein said dense feature matching identifies additional features beyond the sparse feature points;

merging said dense 3D point clouds into a single 3D cloud;

removing outliers from said single 3D cloud to form a cleaned 3D point cloud without using prior face knowledge or generic faces;

fitting a connected surface to the cleaned 3D point cloud without using prior face knowledge or generic faces; and

texture mapping surface detail and color information of a subject'"'"'s face onto said connected surface.

2. The method of claim 1, wherein said prior face knowledge comprises locations of facial features.

3. The method of claim 1, wherein said prior face knowledge indicates a perimeter of said generic face.

4. The method of claim 1, wherein said selecting image pairs comprises tracking feature points across multiple images.

5. The method of claim 1, wherein said refining said initial camera position estimates comprises performing a bundle adjustment optimization interpolation.

6. The method of claim 1, wherein said extracting dense 3D point clouds comprises performing an epipolar geometry process.

7. The method of claim 1, wherein said removing outliers from said single 3D cloud comprises rejecting outlier portions that are more than a predetermined distance from other features.

8. The method of claim 1, wherein said removing outlier portions comprises:

converting data to voxels; and

rejecting data that is more than a predetermined distance from said voxels.

9. The method of claim 1, wherein said removing outlier portions comprises performing a tensor voting process that does not use prior face knowledge or generic faces.

10. The method of claim 1, further comprising storing the reconstructed 3D face in a memory.

11. The method of claim 1, wherein refining said initial camera position estimates and said sparse feature points is performed using data-driven optimization.

12. The method of claim 1, wherein selecting image pairs representing corresponding images from the plurality of 2D images that define similar poses is based, at least in part, on a comparison of an angular baseline between two of the plurality of 2D images.

13. The method of claim 1, wherein said extracting sparse feature points is performed without any user intervention.

14. A system for automatically reconstructing a 3D face from a plurality of 2D images of a human face, comprising:

means for deriving initial camera position estimates using prior face knowledge of a generic face;

means for selecting image pairs representing corresponding images from the plurality of 2D images that define similar poses and extracting sparse feature points representing perimeters of a face or locations of facial features for each of said image pairs;

means for refining said initial camera position estimates and said sparse feature points;

means for extracting dense 3D point clouds from said image pairs by performing dense feature detection and matching across said image pairs using a purely data-driven approach that does not use prior face knowledge or generic faces, wherein said dense feature matching identifies additional features beyond the sparse feature points;

means for merging said dense 3D point clouds into a single 3D cloud;

means for removing outliers from said single 3D cloud to form a cleaned 3D point cloud without using prior face knowledge or generic faces;

means for fitting a connected surface to the cleaned 3D point cloud without using prior face knowledge or generic faces; and

means for texture mapping surface detail and color information of a subject'"'"'s face onto said connected surface.

15. The system of claim 14, further comprising means for storing the reconstructed 3D face in a memory.