3D Face Reconstruction from 2D Images

• US 20070183653A1
• Filed: 01/30/2007
• Published: 08/09/2007
• Est. Priority Date: 01/31/2006
• Status: Active Grant

• Current Assignee: University of Southern California
• Original Assignee: University of Southern California
• Inventors: Medioni, Gerard, Fidaleo, Douglas
• Granted Patent: US 7,856,125 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions

Abstract

A 3D face reconstruction technique using 2D images, such as photographs of a face, is described. Prior face knowledge or a generic face is used to extract spars(e 3D information from the images and to identify image pairs. Bundle adjustment is carried out to determine more accurate 3D camera positions, image pairs are rectified, and dense 3D face information is extracted without using the prior face knowledge. Outliers are removed, e.g., by using tensor voting. A 3D surface is extracted from the dense 3D information and surface detail is extracted from the images.

Claims

1. A face reconstruction method, comprising:

analyzing a plurality of images of a face to find sparse, three-dimensional face features using prior knowledge of a face; and

using said sparse three-dimensional face features to analyze said plurality of images, to find dense three-dimensional features using a data driven approach, without using any prior knowledge.

2. A method as in claim 1, further comprising using the prior knowledge to identify features in the images, and rectifying between pairs of images to find likely image pairs.

3. A method as in claim 1, wherein said prior knowledge is used to identify portions of the face.

4. A method as in claim 1, wherein said prior knowledge is used to identify a face mask representing a generic face.

5. A method as in claim 1, wherein said using said prior knowledge comprises using the prior knowledge to constrain face features that form a set of sparse three-dimensional features.

6. A method as in claim 2, wherein said likely image pairs are image pairs which include an angular baseline that is sufficient to identify three-dimensional information, but is not so large as to undesirably increase measurement uncertainty beyond a specified amount.

7. A method as in claim 2, further comprising testing the image pairs to require an angular baseline greater than a first specified amount, and to require correspondence between features in the pair which is greater than a second specified amount.

8. A method as in claim 1, further comprising second analyzing the plurality of images to find image clusters which have feature point matches between the images greater than a specified amount, and using said image clusters to refine the first analyzing carried out using said prior knowledge, to form a set of tracked feature points.

9. A method as in claim 8, wherein said image clusters include pairs of images.

10. A method as in claim 8, further comprising using said set of tracked feature points to find position and motion of said tracked feature points.

11. A method as in claim 10, further comprising using the position of said tracked feature points to refine said sparse three-dimensional features.

12. A method as in claim 1, wherein said finding dense features comprises constraining the scope of a search for the dense features.

13. A method as in claim 1, wherein said finding dense features comprises rejecting outlier portions which are more than a predetermined distance from other features.

14. A method as in claim 13, wherein said rejecting outlier portions comprises converting data to voxels, and rejecting data that is more than a predetermined distance from the voxels.

15. A method as in claim 1, further comprising rejecting portions which deviate from surface saliency by more than a specified amount.

16. A method as in claim 15, wherein said rejecting comprises detecting said portions by using tensor voting.

17. A face reconstruction system, comprising:

a camera, obtaining a plurality of images of a face; and

a processing part, analyzing said plurality of images, to find sparse, three-dimensional face features using prior knowledge of a face, and using said sparse three-dimensional face features to analyze said plurality of images, to find dense features using a data driven approach, without using any prior knowledge.

18. A system as in claim 17, wherein said camera is a still camera.

19. A system as in claim 17, wherein said camera is a video camera.

20. A system as in claim 17, wherein said processing part operates using the prior knowledge to identify features in the images, and rectify between pairs of images to find likely image pairs.

21. A system as in claim 20, wherein said processing part uses said prior knowledge to identify a face mask representing a generic face.

22. A system as in claim 17, wherein said processing part operates to test the image pairs to require an angular baseline greater than a first specified amount, and to require correspondence between features in the pair which is o greater than a second specified amount.

23. A system as in claim 22, wherein said image clusters include pairs of images.

24. A system as in claim 17, wherein said processing part finds dense features by rejecting outlier portions which are more than a predetermined distance from other features.

25. A system as in claim 24, wherein said processing part carries out said rejecting using tensor voting.

26. A face reconstruction method, comprising:

analyzing a plurality of images from a single camera, to rectify said plurality of images and find three-dimensional information indicative of at least one face from said plurality of images, said analyzing comprising an initial analysis which uses prior knowledge of a face to determine initial features within the images, and a subsequent analysis which uses said initial features of the face to find additional information without using any prior knowledge.

27. A method as in claim 26, further comprising using the prior knowledge to identify features in the images, and rectifying between pairs of images to find likely image pairs.

28. A method as in claim 26, wherein said prior knowledge is used to identify portions of the face.

29. A method as in claim 28, wherein said prior knowledge is used to identify a face mask representing a generic face.

30. A method as in claim 27, wherein said likely image pairs are image pairs which include an angular baseline that is sufficient to identify three-dimensional information, but is not so large as to undesirably increase measurement uncertainty beyond a specified amount.

31. A method as in claim 26, wherein said subsequent analysis comprises constraining the scope of a search for the dense features.

32. A method as in claim 26, further comprising rejecting outlier portions which are more than a predetermined distance from other features.

33. A method as in claim 32, wherein said rejecting outlier portions comprises converting data to voxels, and rejecting data that is more than a predetermined distance from the voxels.

34. A method as in claim 26, further comprising rejecting portions which deviate from surface saliency by more than a specified amount.

35. A method as in claim 34, wherein said rejecting comprises detecting said portions by using tensor voting.

36. A method comprising:

analyzing a plurality of images of a face to find sparse information about the face;

using the sparse information to find dense information using a data driven approach, said using comprising constraining a scope of the search for dense information using a tensor voting technique.

37. A face processing method, comprising:

analyzing a plurality of images of a face to find matches between said images using prior knowledge of a generic face, said matches used to form sparse information; and

using said matches to form pairs of images;

analyzing said pairs to find a set of dense features using a data driven approach without any prior knowledge of the generic face, said analyzing comprising removing outlier portions from the set of dense features.

38. A method for automatically reconstructing a 3D face from a plurality of 2D images of a human face, comprising:

using prior face knowledge of a generic face in deriving initial camera position estimates;

selecting image pairs and extracting sparse feature points for each of said image pairs;

refining said initial camera position estimates and said sparse feature points;

using a purely data-driven approach in extracting dense 3D point clouds from said image pairs;

merging said dense 3D point clouds into a single 3D cloud;

removing outliers from said single 3D point cloud to form a cleaned 3D point cloud;

fitting a connected surface to the cleaned 3D point cloud, and texture mapping surface detail and color information of a subject'"'"'s face onto said connected surface.

39. A method for automatically reconstructing a 3D face from a plurality of 2D images of a human face, comprising:

using prior face knowledge of a generic face in deriving initial camera position estimates;

selecting image pairs and extracting sparse feature points for each of said image pairs;

refining said initial camera position estimates and said sparse feature points;

using a purely data-driven approach in extracting dense 3D point clouds from said image pairs;

merging said dense 3D point clouds into a single 3D cloud;

using prior face knowledge of a generic face in removing outliers from said single 3D cloud to form a cleaned 3D cloud;

fitting a connected surface to the cleaned 3D point cloud; and

texture mapping surface detail and color information of a subject'"'"'s face onto said surface.