3D face reconstruction from 2D images

• US 8,126,261 B2
• Filed: 07/25/2007
• Issued: 02/28/2012
• Est. Priority Date: 01/31/2006
• Status: Active Grant

• Current Assignee: University of Southern California
• Original Assignee: University of Southern California
• Inventors: Medioni, Gerard, Fidaleo, Douglas
• Primary Examiner(s): Patel, Jayesh A
• Application Number:
  US11/828,214
• Publication Number:
  US 20080152213A1
• Time in Patent Office: 1,679 Days
• Field of Search: 382/118, 382/154, 382/294
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions

Abstract

A 3D face reconstruction technique using 2D images, such as photographs of a face, is described. Prior face knowledge or a generic face is used to extract sparse 3D information from the images and to identify image pairs. Bundle adjustment is carried out to determine more accurate 3D camera positions, image pairs are rectified, and dense 3D face information is extracted without using the prior face knowledge. Outliers are removed, e.g., by using tensor voting. A 3D surface is extracted from the dense 3D information and surface detail is extracted from the images.

Claims

1. A face reconstruction method, comprising:

analyzing a plurality of images of a face to find sparse three-dimensional face features using prior knowledge of a face; and

using said sparse three-dimensional face features to analyze said plurality of images, finding dense three-dimensional features using a data driven approach without using said prior knowledge of the face.

2. The method of claim 1, wherein said prior knowledge of the face comprises facial features of a generic face.

3. The method of claim 1, further comprising rectifying between pairs of images to find likely image pairs.

4. The method of claim 3, wherein said likely image pairs are image pairs which include an angular baseline that is sufficient to identify three-dimensional information.

5. The method of claim 3, further comprising testing said image pairs to require an angular baseline greater than a first specified amount and to require correspondence between features in the image pairs, wherein said correspondence is greater than a second specified amount.

6. The method of claim 1, further comprising:

analyzing said plurality of images to find image clusters that have feature point matches between the images greater than a specified amount; and

using said image clusters to refine said analyzing carried out using said prior knowledge of the face to form a set of tracked feature points.

7. The method of claim 6, further comprising using said set of tracked feature points to find position and motion of said tracked feature points.

8. The method of claim 1, wherein said finding dense three-dimensional features comprises rejecting outlier portions that are more than a predetermined distance from other features.

9. The method of claim 8, wherein said rejecting outlier portions comprises:

converting data to voxels; and

rejecting data that is more than a predetermined distance from the voxels.

10. A face reconstruction system, the system comprising:

a camera configured to obtain a plurality of images of a face; and

a processing part configured to analyze said plurality of images to find sparse three-dimensional face features using prior knowledge of a face, to use said sparse three-dimensional face features to analyze said plurality of images, and to find dense three-dimensional features using a data driven approach, without using said prior knowledge of the face.

11. The system of claim 10, wherein said camera is a still camera.

12. The system of claim 10, wherein said processing part operates using the prior knowledge of the face to identify features in the images and to rectify between pairs of images to find likely image pairs.

13. The system of claim 12, wherein said processing part uses said prior knowledge of the face to identify a face mask representing a generic face.

14. The system of claim 10, wherein said processing part operates to test the likely image pairs to require an angular baseline greater than a first specified amount, and to require correspondence between features in the image pairs which is no greater than a second specified amount.

15. The system of claim 10, wherein said processing part finds dense features by rejecting outlier portions which are more than a predetermined distance from other features.

16. The system of claim 15, wherein said processing part carries out said rejecting using tensor voting.

17. A method comprising:

analyzing a plurality of images of a face to find sparse three-dimensional face features;

using the sparse three-dimensional face features to find dense three-dimensional features using a data driven approach without using any prior face knowledge;

rectifying between pairs of images to find likely image pairs; and

testing said image pairs to require an angular baseline greater than a first specified amount and to require correspondence between features in the image pairs, wherein said correspondence is greater than a second specified amount.

18. The method of claim 17, wherein said analyzing comprises use of prior face knowledge or a generic face.

19. The method of claim 17, further comprising rejecting outlier portions that are more than a particular distance from other features.

20. The method of claim 17, further comprising extracting a facial surface from said dense three-dimensional features and extracting surface details from said plurality of images.