Method and system for generating facial animation values based on a combination of visual and audio information

• US 6,940,454 B2
• Filed: 08/13/2001
• Issued: 09/06/2005
• Est. Priority Date: 04/13/1998
• Status: Expired due to Term

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Nevengineering Incorporated
• Inventors: Dzhurinskiy, Yevgeniy V., Paetzold, Frank, Derlich, Karin M., Neven, Hartmut, Buddemeier, Ulrich F.
• Primary Examiner(s): Chen, Shih-Chao
• Assistant Examiner(s): MANCUSO, HUEDUNG XUAN CAO
• Application Number:
  US09/929,516
• Publication Number:
  US 20020118195A1
• Time in Patent Office: 1,485 Days
• Field of Search: 345/473, 345/474, 345/475, 345/619, 704/236, 704/235
• US Class Current: 345/473
• CPC Class Codes: G06T 2207/20064 Wavelet transform [DWT]
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/262 using transform domain methods, e.g. Fourier domain methods
• CPC Class Codes: G06V 10/426 Graphical representations
• CPC Class Codes: G06V 10/449 Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships

Abstract

Facial animation values are generated using a sequence of facial image frames and synchronously captured audio data of a speaking actor. In the technique, a plurality of visual-facial-animation values are provided based on tracking of facial features in the sequence of facial image frames of the speaking actor, and a plurality of audio-facial-animation values are provided based on visemes detected using the synchronously captured audio voice data of the speaking actor. The plurality of visual facial animation values and the plurality of audio facial animation values are combined to generate output facial animation values for use in facial animation.

Claims

1. Method for generating facial animation values using a sequence of facial image frames and synchronously captured audio data of a speaking actor, comprising the steps for:

providing a plurality of visual-facial-animation values based on tracking of facial features in the sequence of facial image frames of the speaking actor;

providing a plurality of audio-facial-animation values based on visemes detected using the synchronously captured audio voice data of the speaking actor; and

combining the plurality of visual facial animation values and the plurality of audio facial animation values to generate output facial animation values for use in facial animation.

2. Method for generating facial animation values as defined in claim 1, wherein the output facial animation values associated with a mouth for a facial animation are based only on the respective mouth-associated values of the plurality of audio facial animation values.

3. Method for generating facial animation values as defined in claim 1, wherein the output facial animation values associated with a mouth for a facial animation are based on a weighted average of the respective mouth-associated values of the plurality of visual facial animation values and the respective mouth-associated values of the plurality of audio facial animation values.

4. Method for generating facial animation values as defined in claim 3, wherein the output facial animation values are calculated using the following equation:

${\left({\underset{\_}{f}}_n=\frac{{\underset{\_}{\mathrm{\sigma <br><br>}}}_n^a}{{\underset{\_}{\mathrm{\sigma <br><br>}}}_n^a+{\underset{\_}{\mathrm{\sigma <br><br>}}}_n^v}·<br><br>{\underset{\_}{a}}_n+\frac{{\underset{\_}{\mathrm{\sigma <br><br>}}}_n^v}{{\underset{\_}{\mathrm{\sigma <br><br>}}}_n^v+{\underset{\_}{\mathrm{\sigma <br><br>}}}_n^a}·<br><br>{\underset{\_}{v}}_n\right)}_i$

where;

f_n are the output facial animation values;

v_n are the visual facial animation values;

a_n are the respective mouth-associated values of the audio facial animation values;

σ

_n^a are the weights for the audio facial animation values; and

σ

_n^v are the weights for the visual facial animation values.

5. Method for generating facial animation values as defined in claim 1, wherein the output facial animation values associated with a mouth for a facial animation are based on Kalman filtering of the respective mouth-associated values of the plurality of visual facial animation values and the respective mouth-associated values of the plurality of audio facial animation values.

6. Method for generating facial animation values as defined in claim 1, wherein the step of combining the plurality of visual facial animation values and the plurality of audio facial animation values to generate output facial animation values includes detecting whether speech is occurring in the synchronously captured audio voice data of the speaking actor and, while speech is detected as occurring, generating the output facial animation values associated with a mouth based only on the respective mouth-associated values of the plurality of audio facial animation values and, while speech is not detected as occurring, generating the output facial animation values associated with a mouth based only on the respective mouth-associated values of the plurality of visual facial animation values.

7. Method for generating facial animation values as defined in claim 1, wherein the tracking of facial features in the sequence of facial image frames of the speaking actor is performed using bunch graph matching.

8. Method for generating facial animation values as defined in claim 1, wherein the tracking of facial features in the sequence of facial image frames of the speaking actor is performed using transformed facial image frames generated based on wavelet transformations.

9. Method for generating facial animation values as defined in claim 1, wherein the tracking of facial features in the sequence of facial image frames of the speaking actor is performed using transformed facial image frames generated based on Gabor wavelet transformations.

10. Method for generating facial animation values as defined in claim 1, wherein the tracking of facial features in the sequence of facial image frames of the speaking actor is performed without using markers attached to the speaking actor'"'"'s face.

11. Apparatus for generating facial animation values using a sequence of facial image frames and synchronously captured audio data of a speaking actor, comprising:

means for providing a plurality of visual-facial-animation values based on tracking of facial features in the sequence of facial image frames of the speaking actor;

means for providing a plurality of audio-facial-animation values based on visemes detected using the synchronously captured audio voice data of the speaking actor; and

means for providing a plurality of visual-facial-animation values based on tracking of facial features in the sequence of facial image frames of the speaking actor;

means for combining the plurality of visual facial animation values and the plurality of audio facial animation values to generate output facial animation values for use in facial animation.

12. Apparatus for generating facial animation values as defined in claim 11, wherein the output facial animation values associated with a mouth for a facial animation are based only on the respective mouth-associated values of the plurality of audio facial animation values.

13. Apparatus for generating facial animation values as defined in claim 11, wherein the output facial animation values associated with a mouth for a facial animation are based on a weighted average of the respective mouth-associated values of the plurality of visual facial animation values and the respective mouth-associated values of the plurality of audio facial animation values.

14. Apparatus for generating facial animation values as defined in claim 13, wherein the output facial animation values are calculated using the following equation:

${\left({\underset{\_}{f}}_n=\frac{{\underset{\_}{\mathrm{\sigma <br><br>}}}_n^a}{{\underset{\_}{\mathrm{\sigma <br><br>}}}_n^a+{\underset{\_}{\mathrm{\sigma <br><br>}}}_n^v}·<br><br>{\underset{\_}{a}}_n+\frac{{\underset{\_}{\mathrm{\sigma <br><br>}}}_n^v}{{\underset{\_}{\mathrm{\sigma <br><br>}}}_n^v+{\underset{\_}{\mathrm{\sigma <br><br>}}}_n^a}·<br><br>{\underset{\_}{v}}_n\right)}_i$

where;

f_n are the output facial animation values;

v_n are the visual facial animation values;

a_n are the respective mouth-associated values of the audio facial animation values;

σ

_n^a are the weights for the audio facial animation values; and

σ

_n^v are the weights for the visual facial animation values.

15. Apparatus for generating facial animation values as defined in claim 11, wherein the output facial animation values associated with a mouth for a facial animation are based on Kalman filtering of the respective mouth-associated values of the plurality of visual facial animation values and the respective mouth-associated values of the plurality of audio facial animation values.

16. Apparatus for generating facial animation values as defined in claim 11, wherein the means for combining the plurality of visual facial animation values and the plurality of audio facial animation values to generate output facial animation values includes means for detecting whether speech is occurring in the synchronously captured audio voice data of the speaking actor and, while speech is detected as occurring, generating the output facial animation values associated with a mouth based only on the respective mouth-associated values of the plurality of audio facial animation values and, while speech is not detected as occurring, generating the output facial animation values associated with a mouth based only on the respective mouth-associated values of the plurality of visual facial animation values.

17. Apparatus for generating facial animation values as defined in claim 11, wherein the tracking of facial features in the sequence of facial image frames of the speaking actor is performed using bunch graph matching.

18. Apparatus for generating facial animation values as defined in claim 11, wherein the tracking of facial features in the sequence of facial image frames of the speaking actor is performed using transformed facial image frames generated based on wavelet transformations.

19. Apparatus for generating facial animation values as defined in claim 11, wherein the tracking of facial features in the sequence of facial image frames of the speaking actor is performed using transformed facial image frames generated based on Gabor wavelet transformations.

20. Apparatus for generating facial animation values as defined in claim 11, wherein the tracking of facial features in the sequence of facial image frames of the speaking actor is performed without using markers attached to the speaking actor'"'"'s face.