Method and apparatus for generating 3D face models from one camera

• US 7,103,211 B1
• Filed: 09/04/2002
• Issued: 09/05/2006
• Est. Priority Date: 09/04/2001
• Status: Active Grant

• Current Assignee: Image Metrics, Inc.
• Original Assignee: Geometrix Inc. (Alive Tech Inc.)
• Inventors: Pesenti, Bastien, Medioni, Gérard G.
• Primary Examiner(s): Mancuso, Joseph
• Assistant Examiner(s): SCHAFFER, JONATHAN C
• Application Number:
  US10/236,020
• Time in Patent Office: 1,462 Days
• Field of Search: 382/118, 382/154
• US Class Current: 382/154
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions

Abstract

Techniques for generating a fully-textured 3D model of a human face are disclosed. According to one aspect of the present invention, a fully automatic system is configured to generate a 3-D model from a sequence of images that are produced from a single camera when a user is moving his/her head steadily from one direction to another, for example, from the left side of the camera to the right side of the camera. The images are processed in a two-stage process. First, the camera pose information is estimated for each of the valid image frames. Second, a mesh model is generated with respect to the camera pose information, and subsequently is textured with reference to one or more of the image frames.

Claims

1. A method for generating a fully-textured 3D model of a human face without depending on a deformable generic face, the method comprising:

generating a sequence of images from a camera while the human face rotates from one direction to another direction in front of the camera;

determining a foreground and a background in each of the image, the foreground occupied by the human face and thus changing from one to another in the sequence of the images;

extracting features located in the foreground of each of the images;

determining camera pose information in each of the images with respect to the features being tracked across the sequence of images;

generating a 3D mesh model in accordance with the camera pose information; and

texturing the 3D mesh model with respect to one or more of the images.

2. The method of claim 1 further comprising:

rectifying the images in accordance with calibration information of the camera; and

discarding some of the Images that demonstrate no motion of the human face with reference to an adjacent image thereof in the sequence of the images.

3. The method of claim 1, wherein the features are of sufficient number and largely distributed across the foreground;

and the method further comprising;

subdividing each of the images into a plurality of small regions;

analyzing each of the regions for brightness and contrast information; and

applying a filter to detect a pre-defined number of the features in the foreground.

4. The method of claim 3, wherein the features detected in one of the images are respectively confirmed with an adjacent image thereof.

5. The method of claim 4, wherein the features detected in one of the images are respectively matched in the adjacent image.

6. The method of claim 4, wherein the features detected In one of the Images are respectively tracked in the adjacent image.

7. The method of claim 4, wherein the determining of the camera pose information in each of the images comprises:

computing, based on the confirmed features between the one of the images and the adjacent image thereof, the camera pose information with respect to the human face in each of the images; and

refining the camera pose information using a non-linear bundle adjuster.

8. The method of claim 7 wherein the generating of the 3D mesh model in accordance with the camera pose information comprises:

building dense stereo maps based on the pose information from each pair of two adjacent images in the sequence of the images,

determining a respective dense disparity between the each pair of two adjacent images in accordance with the related features; and

integrating the dense stereo maps into a 3D mesh by triangulation In reference to the respective dense disparity.

9. The method of claim 7, wherein the generating of the 3D mesh model in accordance with the camera pose information of the camera comprises:

triangulating the confirmed features into a coarse mesh;

adding incrementally the confirmed features and refining respective locations of the features in the images using the computed coarse mesh and the camera pose information; and

triangulating the refined features to generate the 3D mesh model.

10. The method of claim 1, wherein the texturing of the 3D mesh model with respect to one or more of the images comprises:

selecting a polygon In the 3D mesh model;

determining one of the image frames whose associated camera plane is closest to that of the polygon; and

assigning corresponding texture from the one of the image frames to the polygon.

11. A system for generating a fully-textured 3D model of a human face without depending on a deformable generic face, the system comprising:

a computing device;

a single camera coupled to the computing device and generating a sequence of Images while the human face rotates from one direction to another direction before the single camera; and

wherein the computing device is configured to perform operations of;

receiving the images from the camera;

determining camera pose information in each of the images with respect to detected features being tracked across the sequence of images;

generating a mesh model of the human face in accordance with the camera pose information; and

texturing each of polygons in the mesh model with respect to one or more of the images.

12. The system of claim 11, wherein the determining of the camera pose information in each of the images comprises:

determining a foreground and a background in each of the image, the foreground occupied by the human face and thus changing from one to another in the sequence of the images

extracting the detected features located in the foreground of each of the images;

confirming the features in one of the images with an adjacent image thereof; and

computing the camera pose information using the confirmed features.

13. The system of claim 12, wherein the generating of the mesh model of the human face comprises:

building dense stereo maps, based on the camera pose information, from each pair of two adjacent images in the sequence of the images,

determining a respective dense disparity between the each pair of two adjacent images in accordance with the confirmed features; and

integrating the dense stereo maps into the mesh model by triangulation in reference to the respective dense disparity.

14. The system of claim 12, wherein the generating of the mesh model of the human face comprises:

triangulating the confirmed features into a coarse mesh

incrementally adding the confirmed features and refining respective locations of the confirmed features in the images using the computed coarse mesh and the camera pose information; and

triangulating the refined points to generate the mesh model.

15. A software product encoded on a computer readable medium and to be executed in a computing device for generating a fully-textured 3D model of a human face without depending on a deformable generic face, the software product comprising:

program code for receiving a sequence of images from a camera, the image being produced while the human face rotates from one direction to another direction in front of the camera;

program code for determining a foreground and a background in each of the image, the foreground occupied by the human face and thus changing from one to another in the sequence of the images;

program code for extracting features located in the foreground of each of the images;

program code for determining pose information of the camera in each of the images with respect to the features being tracked across the sequence of images;

program code for generating a 3D mesh model in accordance with the camera pose information with respect to the features; and

program code for texturing the 3D mesh model with respect to one or more of the images.

16. The software product of claim 15 further comprising:

program code for rectifying the images in accordance with calibration information of the camera; and

program code for discarding some of the images that demonstrate no motion of the human face with reference to an adjacent image thereof in the sequence of the images.

17. The software product of claim 15, wherein the features are of sufficient number and largely distributed across the foreground;

and the software product further comprising;

program code for subdividing each of the images into a plurality of small regions;

program code for analyzing each of the regions for brightness and contrast information; and

program code for applying a filter to detect a pre-defined number of the features in the foreground.

18. The software product of claim 17, wherein the features detected in one of the images are respectively related to an adjacent image thereof.

19. The software product of claim 18, wherein the program code for determining the camera pose information in each of the images comprises:

program code for determining, based on the related features between the one of the images and the adjacent image thereof, pose information of the human face with respect to the camera In each of the images; and

program code for refining the camera pose information using a non-linear bundle adjuster.

20. The software product of claim 19, wherein the program code for generating a 3D mesh model in accordance with the camera pose information comprises:

program code for building dense stereo maps based on the pose information;

program code for integrating the dense stereo maps into a 3D mesh by triangulation in reference to the respective dense disparity.

21. The software product of claim 19, wherein the program code for generating a 3D mesh model in accordance with the camera pose information comprises:

program code for refining Incrementally points obtained from the camera pose information; and

program code for triangulating the refined points to generate the mesh model.

22. The software product of claim 15, wherein the program code for texturing the 3D mesh model with respect to one or more of the images comprises:

program code for selecting a polygon in the mesh model;

program code for determining one of the images whose associated camera plane is closest to that of the polygon; and

program code for assigning corresponding texture from the image to the polygon.