Wavelet-based facial motion capture for avatar animation

• US 6,580,811 B2
• Filed: 05/31/2001
• Issued: 06/17/2003
• Est. Priority Date: 04/13/1998
• Status: Expired due to Term

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Neven Vision, Inc. (Alphabet Inc.)
• Inventors: Nocera, Luciano Pasquale Agostino, Elagin, Egor Valerievich, Steffens, Johannes Bernhard, Maurer, Thomas, Neven, Hartmut
• Primary Examiner(s): Patel, Jayanti K.
• Assistant Examiner(s): Choobin, Barry
• Application Number:
  US09/871,370
• Publication Number:
  US 20010033675A1
• Time in Patent Office: 747 Days
• Field of Search: 382/103, 348/169
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/20064 Wavelet transform [DWT]
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/262 using transform domain methods, e.g. Fourier domain methods
• CPC Class Codes: G06V 10/426 Graphical representations
• CPC Class Codes: G06V 10/449 Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships

Abstract

The present invention is embodied in an apparatus, and related method, for sensing a person'"'"'s facial movements, features and characteristics and the like to generate and animate an avatar image based on facial sensing. The avatar apparatus uses an image processing technique based on model graphs and bunch graphs that efficiently represent image features as jets. The jets are composed of wavelet transforms processed at node or landmark locations on an image corresponding to readily identifiable features. The nodes are acquired and tracked to animate an avatar image in accordance with the person'"'"'s facial movements. Also, the facial sensing may use jet similarity to determine the person'"'"'s facial features and characteristic thus allows tracking of a person'"'"'s natural characteristics without any unnatural elements that may interfere or inhibit the person'"'"'s natural characteristics.

Claims

1. A method for generating a head model specific to a person for use in animating a display image having the person'"'"'s features, the method comprising the steps for:

providing a frontal facial image of the person, the frontal facial image having an array of pixels;

providing a profile facial image of the person, the profile facial image having an array of pixels;

transforming the frontal facial image using wavelet transformations to generate a transformed frontal image having an array of pixels, each pixel of the transformed frontal image being associated with a respective pixel of the frontal facial image and being represented by a frontal transform jet associated with a predetermined number of wavelet component values;

transforming the profile facial image using wavelet transformations to generate a transformed profile image having an array of pixels, each pixel of the transformed profile image being associated with a respective pixel of the profile facial image and being represented by a profile transform jet associated with a predetermined number of wavelet component values;

locating predetermined facial features in the transformed frontal image based on a comparison between the frontal transform jets of the transformed frontal facial image and frontal graph jets at sensing nodes of a frontal graph;

locating predetermined facial features in the transformed profile image based on a comparison between the profile transform jets of the transformed profile facial image and sensing nodes at profile graph jets of a profile graph;

producing a three-dimensional model of the person'"'"'s head using the frontal facial image and the profile facial image and based on the positions of the predetermined features located in the transformed frontal facial image and the predetermined features located in the transformed profile facial image;

displaying the head model as a display image.

2. A method for generating a head model as defined in claim 1, wherein the positions of the predetermined features are associated with respective nodes for animating the head model.

3. A method for generating a head model as defined in claim 1, wherein the predetermined facial features associated with the frontal facial image include the person'"'"'s eyes, nose, mouth, and chin.

4. A method for generating a head model as defined in claim 1, wherein the predetermined facial features associated with the profile facial image include the person'"'"'s eyes, nose, mouth, and chin.

5. A method for generating a head model as defined in claim 1, wherein the wavelet transformations use Gabor wavelets.

6. A method for generating a head model as defined in claim 1, wherein each wavelet component value is generated based on a Gabor wavelet having a particular orientation and frequency.

7. A method for providing a display avatar having a specific person'"'"'s features, the method comprising the steps for:

providing a frontal facial image of the person, the frontal facial image having an array of pixels;

providing a profile facial image of the person, the profile facial image having an array of pixels;

transforming the frontal facial image using wavelet transformations to generate a transformed frontal image having an array of pixels, each pixel of the transformed frontal image being associated with a respective pixel of the frontal facial image and being represented by a frontal transform jet associated with a predetermined number of wavelet component values;

transforming the profile facial image using wavelet transformations to generate a transformed profile image having an array of pixels, each pixel of the transformed profile image being associated with a respective pixel of the profile facial image and being represented by a profile transform jet associated with a predetermined number of wavelet component values;

locating predetermined facial features in the transformed frontal facial image based on a comparison between the frontal transform jets of the transformed frontal facial image and frontal graph jets at sensing nodes of a frontal graph;

locating predetermined facial features in the transformed profile facial image based on a comparison between the profile transform jets of the transformed profile facial image and sensing nodes at profile graph jets of a profile graph;

producing the display avatar using the frontal facial image and the profile facial image and based on the positions of the predetermined features located in the transformed frontal facial image and the predetermined features located in the transformed profile facial image;

displaying the display avatar.

8. A method for providing a display avatar as defined in claim 7, wherein the positions of the predetermined features are associated with respective nodes for animating the avatar.

9. A method for providing a display avatar as defined in claim 7, wherein the predetermined facial features associated with the frontal facial image include the person'"'"'s eyes, nose, mouth, and chin.

10. A method for providing a display avatar as defined in claim 7, wherein the predetermined facial features associated with the profile facial image include the person'"'"'s eyes, nose, mouth, and chin.

11. A method for providing a display avatar as defined in claim 7, wherein the wavelet transformations use Gabor wavelets.

12. A method for providing a display avatar as defined in claim 7, wherein each wavelet component value is generated based on a Gabor wavelet having a particular orientation and frequency.