Methods and apparatus for face fitting and editing applications

• US 8,923,392 B2
• Filed: 02/29/2012
• Issued: 12/30/2014
• Est. Priority Date: 09/09/2011
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Bourdev, Lubomir D., Shechtman, Elya, Wang, Jue, Yang, Fei
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): MESSMORE, JONATHAN R
• Application Number:
  US13/408,873
• Publication Number:
  US 20130121409A1
• Time in Patent Office: 1,035 Days
• Field of Search: 375/240.08
• US Class Current: 375/240.08
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 18/211 Selection of the most significant subset of features
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/174 Facial expression recognition

Abstract

Various embodiments of methods and apparatus for face fitting are disclosed. In one embodiment, sets of two-dimensional local feature points on a face in each image of a set of images are identified. The set of images includes a sequence of frames a video stream. A three-dimensional face model for the face in the each image is generated as a combination of a set of predefined three-dimensional face models. In some embodiments, the generating includes reducing an error between a projection of vertices of the set of predefined three-dimensional face models and the two-dimensional local feature points of the each image, and constraining facial expression of the three-dimensional face model to change smoothly from image to image in the sequence of video frames.

Claims

1. A method, comprising:

causing one or more processors to perform

identifying sets of two-dimensional local feature points on a face in each image of a set of images, wherein

the set of images comprises a sequence of frames a video stream; and

generating a three-dimensional face model for the face in the each image as a combination of a set of predefined three-dimensional face models, wherein said generating further comprises

reducing an error between a projection of vertices of the set of predefined three-dimensional face models and the two-dimensional local feature points of the each image,

constraining facial expression of the three-dimensional face model to change smoothly from image to image in the sequence of video frames, and

representing each of the set of predefined three-dimensional face models using a three-dimensional tensor decomposition defined by

a vector of coefficients representing an expression dimension that varies for each frame, and

a vector of identity coefficients that is fixed across frames.

2. The method of claim 1, wherein

the generating the three-dimensional face model for the face in the each image further comprises

optimizing an energy minimization function including a fidelity term and a temporal coherence term.

3. The method of claim 1, wherein

the generating the three-dimensional face model for the face in the each image further comprises

optimizing an energy minimization function including a temporal coherence term.

4. The method of claim 1, wherein

the generating the three-dimensional face model for the face in the each image further comprises

optimizing an energy minimization function including a fidelity term.

5. The method of claim 1, further comprising

generating a new image representing an adjustment of a respective one or more of the combinations to adjust an expression of the face.

6. The method of claim 1, further comprising

compressing an image stream, wherein the compressing comprises

generating a new image representing a key frame and a series of deformations representing each of the set of images.

7. The method of claim 1, further comprising

generating a new image representing an adjustment of a respective one or more of the combinations to replace a missing frame.

8. A system, comprising:

at least one processor; and

a memory comprising program instructions, wherein the program instructions are executable by the at least one processor to;

identify sets of two-dimensional local feature points on a face in each image of a set of images, wherein

the set of images comprises a sequence of frames a video stream; and

generate a three-dimensional face model for the face in the each image as a combination of a set of predefined three-dimensional face models, wherein the program instructions executable by the at least one processor to generate the three-dimensional face model for the face in the each image further comprise program instructions executable by the at least one processor to

represent each of the set of predefined three-dimensional face models using a three-dimensional tensor decomposition defined by

a vector of coefficients representing an expression dimension that varies for each frame, and

a vector of identity coefficients that is fixed across frames, and

constrain facial expression of the three-dimensional face model.

9. The system of claim 8, wherein

the program instructions executable by the at least one processor to generate the three-dimensional face model for the face in the each image further comprise program instructions executable by the at least one processor to optimize an energy minimization function including a fidelity term and a temporal coherence term.

10. The system of claim 8, wherein

the program instructions executable by the at least one processor to generate the three-dimensional face model for the face in the each image further comprise program instructions executable by the at least one processor to

reduce an error between a projection of vertices of the set of predefined three-dimensional face models and the two-dimensional local feature points of the each image.

11. The system of claim 10, wherein

the program instructions executable by the at least one processor to constrain facial expression of the three-dimensional face model further comprise the program instructions executable by the at least one processor to constrain facial expression of the three-dimensional face model change smoothly from image to image in the sequence of video frames.

12. The system of claim 8, further comprising

program instructions executable by the at least one processor to generate a new image representing an adjustment of a respective one or more of the deformations to adjust an expression of the face.

13. The system of claim 8, further comprising

program instructions executable by the at least one processor to compress an image stream, wherein the program instructions executable by the at least one processor to compress the image stream comprise program instructions executable by the at least one processor to generate a new image representing a key frame and a series of deformations representing each of the set of images.

14. The system of claim 8, further comprising

program instructions executable by the at least one processor to generate a new image representing an adjustment of a respective one or more of the combinations to replace a missing frame.

15. A non-transitory computer-readable storage medium storing program instructions, wherein the program instructions are computer-executable to implement:

identifying sets of two-dimensional local feature points on a face in each image of a set of images, wherein

the set of images comprises a sequence of frames a video stream; and

generating a three-dimensional face model for the face in the each image as a combination of a set of predefined three-dimensional face models, wherein the program instructions computer-executable to implement said generating further comprise program instructions computer-executable to implement

reducing an error between a projection of vertices of the set of predefined three-dimensional face models and the two-dimensional local feature points of the each image,

representing each of the set of predefined three-dimensional face models using a three-dimensional tensor decomposition defined by

a vector of coefficients representing an expression dimension that varies for each frame, and

a vector of identity coefficients that is fixed across frames.

16. The non-transitory computer-readable storage medium of claim 15, wherein

the program instructions computer-executable to implement generating the three-dimensional face model for the face in the each image further comprise program instructions computer-executable to implement optimizing an energy minimization function including a fidelity term and a temporal coherence term.

17. The non-transitory computer-readable storage medium of claim 15, wherein

the program instructions computer-executable to implement generating the three-dimensional face model for the face in the each image further comprise program instructions computer-executable to implement optimizing an energy minimization function including a temporal coherence term.

18. The non-transitory computer-readable storage medium of claim 17, further comprising

program instructions computer-executable to implement constraining facial expression of the three-dimensional face model to change smoothly from image to image in the sequence of video frames.

19. The non-transitory computer-readable storage medium of claim 15, further comprising

program instructions computer-executable to implement generating a new image representing an adjustment of a respective one or more of the deformations to adjust an expression of the face.

20. The non-transitory computer-readable storage medium of claim 15, further comprising

program instructions computer-executable to implement compressing an image stream, wherein the program instructions computer-executable to implement compressing the image stream comprise the program instructions computer-executable to implement generating a new image representing a key frame and a series of deformations representing each of the set of images.