Methods and systems for animating facial features and methods and systems for expression transformation

• US 7,098,920 B2
• Filed: 07/27/2004
• Issued: 08/29/2006
• Est. Priority Date: 08/30/2000
• Status: Expired due to Term

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Kang, Sing Bing, Olynyk, Kirk, Guenter, Brian K., Raghupathy, Sashi, Marschner, Stephen
• Primary Examiner(s): Chauhan, Ulka
• Assistant Examiner(s): HAJNIK, DANIEL F
• Application Number:
  US10/900,252
• Publication Number:
  US 20040263510A1
• Time in Patent Office: 763 Days
• Field of Search: None
• US Class Current: 345/473
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 17/205 Re-meshing
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: Y10S 345/956 Language driven animation

Abstract

The illustrated and described embodiments describe techniques for capturing data that describes 3-dimensional (3-D) aspects of a face, transforming facial motion from one individual to another in a realistic manner, and modeling skin reflectance.

Claims

1. One or more computer-readable media having computer-readable instructions thereon which, when executed by a computing device, perform a facial expression transformation method comprising:

defining a code book containing data defining a first set of facial expressions of a first person;

providing data defining a second set of facial expressions, the second set of facial expressions providing a training set of expressions of a second person who is different from the first person;

deriving a transformation function from the training set of expressions and corresponding expressions from the first set of expressions; and

applying the transformation function to the first set of expressions to provide a synthetic set of expressions.

2. The one or more computer-readable media of claim 1, wherein the training set of expressions contains fewer expressions than the code book.

3. The one or more computer-readable media of claim 1, wherein the transformation function compensates for differences in the size and shape of the faces of the first and second persons.

4. The one or more computer-readable media of claim 1, wherein said deriving of the transformation function comprises computing a linear transformation from one set of expressions to another.

5. The one or more computer-readable media of claim 1, wherein the deriving of the transformation function comprises:

representing each expression as a 3m-vector that contains x, y, z displacements at m standard sample positions; and

computing a set of linear predictors a_j, one for each coordinate of g_a, given a set of n expression vectors for a face to be transformed, g_{a1 . . . n}, and a corresponding set of vectors for a target face, g_{b1 . . . n}, by solving 3m linear least squares systems of the following form;

a_j·

g_ai=g_bi[j], i=1 . . . n

6. The one or more computer-readable media of claim 5, wherein said computing comprises using only a subset of points for each g_aj.

7. The one or more computer-readable media of claim 6, wherein said using comprises using only points that share edges with a standard sample point under consideration.

8. The one or more computer-readable media of claim 5 further comprising controlling the spread of singular values when computing a pseudoinverse to solve for the a_j.

9. The one or more computer-readable media of claim 8, wherein said controlling the spread comprises zeroing out all singular values less than α

σ

₁, where σ

₁ is the largest singular value of the matrix.

10. The one or more computer-readable media of claim 1, wherein said providing data defining a second set of facial expressions comprises:

illuminating the second person'"'"'s face with illumination; and

contemporaneously capturing structure data describing the face'"'"'s structure and reflectance data describing reflectance properties of the face from the illumination.

11. The one or more computer-readable media of claim 10, wherein said illuminating comprises:

using multiple light sources, one of which projecting a pattern on the second person'"'"'s face from which the structure data can be ascertained;

at least one of the light sources comprising an infrared light source;

at least one of the light sources being polarized; and

said capturing comprising using a camera having a polarizer that suppresses specularly-reflected light so that diffuse component reflection data is captured.

12. The one or more computer-readable media of claim 1, wherein said providing data defining a second set of facial expressions comprises:

illuminating the second person'"'"'s face with a first polarized light source that is selected so that specularly-suppressed reflective properties of the face can be ascertained;

illuminating the second person'"'"'s face with a second structured light source that projects a pattern onto the face, while simultaneously illuminating the face with the first polarized light source; and

capturing both specularly-suppressed reflection data and structure data from the simultaneous illumination.

13. The one or more computer-readable media of claim 12, wherein the light sources provide light at different frequencies.

14. The one or more computer-readable media of claim 12, wherein the light sources provide infrared light.

15. The one or more computer-readable media of claim 12, further comprising processing the captured data to provide both (a) data that describes dimensional aspects of the face and (b) data that describes diffuse reflective properties of the face.

16. The one or more computer-readable media of claim 1, wherein said providing data defining a second set of facial expressions comprises:

illuminating the second person'"'"'s face with multiple different light sources;

measuring range map data from said illuminating;

measuring image data from said illuminating;

deriving a 3-dimensional surface from the range map data;

computing surface normals to the 3-dimensional surface; and

processing the surface normals and the image data to derive an albedo map.

17. The one or more computer-readable media of claim 16, wherein at least one of the light sources is polarized.

18. The one or more computer-readable media of claim 16, wherein all of the light sources are polarized.

19. A computing device embodying the one or more computer-readable media of claim 1.

20. A computing system comprising:

one or more computer-readable media having computer-readable instructions thereon which, when executed by a computer, cause the computer to;

operate on a training set of expressions from one person and corresponding expressions from a code book of another person to compute a linear transformation function from the training set and their corresponding expressions; and

apply the transformation function to a plurality of expressions from the code book to provide a synthetic set of expressions; and

one or more processors configured to execute said computer-readable instructions.

21. The computing system of claim 20, wherein the instructions cause the one or more processors to use the synthetic set of expressions to transform expressions from the one person into expressions of the other person.

22. The computing system of claim 21, wherein the instructions cause the one or more processors to transform expressions from the one person that are different from those expressions comprising the code book expressions.

23. The computing system of claim 21, wherein the instructions cause the one or more processors to transform expressions by transmitting at least one index of a synthetic expression to a receiver that can reconstruct the expression.

24. The computing system of claim 21, wherein the instructions cause the one or more processors to transform facial expressions.

25. A facial expression transformation system comprising:

a code book embodied on a computer-readable medium, the code book containing data defining a first set of facial expressions of a first person;

data embodied on a computer-readable medium, the data defining a second set of facial expressions, the second set of facial expressions providing a training set of expressions of a second person who is different from the first person; and

one or more computer-readable media embodying instructions that implement a transformation processor configured to derive a transformation function from the training set of expressions and corresponding expressions from the first set of expressions.

26. The expression transformation system of claim 25, wherein the transformation processor comprises a linear transformation processor.

27. The expression transformation system of claim 25 further comprising a synthetic set of expressions embodied on a computer-readable medium, the synthetic set of expressions being derived by applying the transformation function to the code book expressions.

28. The expression transformation system of claim 25, wherein the transformation function compensates for differences in the size and shape of the faces of the first and second persons.

29. The expression transformation system of claim 25, wherein the transformation processor derives the transformation function by:

representing each expression as a 3m-vector that contains x, y, z displacements at m standard sample positions; and

computing a set of linear predictors a_j, one for each coordinate of g_a, given a set of n expression vectors for a face to be transformed, g_{a1 . . . n}, and a corresponding set of vectors for a target face, g_{b1 . . . n}, by solving 3m linear least squares systems of the following form;

a_j·

g_ai=g_bi[j], i=1 . . . n