Wavelet-based facial motion capture for avatar animation

• US 6,272,231 B1
• Filed: 11/06/1998
• Issued: 08/07/2001
• Est. Priority Date: 11/06/1998
• Status: Expired due to Term

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Neven Vision, Inc. (Alphabet Inc.)
• Inventors: Maurer, Thomas, Neven, Hartmut, Nocera, Luciano Pasquale Agostino, Elagin, Egor Valerievich, Steffens, Johannes Bernhard
• Primary Examiner(s): Boudreau, Leo
• Assistant Examiner(s): CHOOBIN, BARRY
• Application Number:
  US09/188,079
• Time in Patent Office: 1,005 Days
• Field of Search: 345/473, 345/419, 382/276, 382/118, 382/103, 382/209, 379/52
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/20064 Wavelet transform [DWT]
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/262 using transform domain methods, e.g. Fourier domain methods
• CPC Class Codes: G06V 10/426 Graphical representations
• CPC Class Codes: G06V 10/449 Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships

Abstract

The present invention is embodied in an apparatus, and related method, for sensing a person'"'"'s facial movements, features and characteristics and the like to generate and animate an avatar image based on facial sensing. The avatar apparatus uses an image processing technique based on model graphs and bunch graphs that efficiently represent image features as jets. The jets are composed of wavelet transforms processed at node or landmark locations on an image corresponding to readily identifiable features. The nodes are acquired and tracked to animate an avatar image in accordance with the person'"'"'s facial movements. Also, the facial sensing may use jet similarity to determine the person'"'"'s facial features and characteristic thus allows tracking of a person'"'"'s natural characteristics without any unnatural elements that may interfere or inhibit the person'"'"'s natural characteristics.

Claims

1. A method for feature sensing on a sequence of image frames, comprising:

a step for transforming each image frame using a wavelet transformation to generate a transformed image frame;

a step for initializing nodes of a model graph, each node associated with a wavelet jet specific to a feature, to locations on the transformed image frame by moving the model graph across the transformed image frame and placing the model graph at a location in the transformed image frame of maximum jet similarity between the wavelet jets of the nodes and locations on the transformed image frame determined as the model graph is moved across the transformed image frame;

a step for tracking the location of one or more nodes of the model graph between image frames; and

a step for reinitializing the location of a tracked node if the tracked node'"'"'s location deviates beyond a predetermined position constraint between image frames.

2. A method for feature sensing as defined in claim 1, wherein the model graph used in the initializing step is based on a predetermined pose.

3. A method for feature sensing as defined in claim 1, wherein the tracking step tracks the node locations using elastic bunch graph matching.

4. A method for feature sensing as defined in claim 1, wherein the tracking step uses linear position prediction to predict node locations in a subsequent image frame and the reinitializing step reinitializes a node location based on a deviation from the predicted node location that is greater than a predetermined threshold deviation.

5. A method for feature sensing as defined in claim 1, wherein the predetermined position constraint is based on a geometrical position constraint associated with relative positions between the node locations.

6. A method for feature sensing as defined in claim 1, wherein the node locations are transmitted to a remote site for animating an avatar image.

7. A method for feature sensing as defined in claim 1, wherein the tracking step includes determining a facial characteristic.

8. A method for feature sensing as defined in claim 7, further comprising transmitting the node locations and facial characteristics to a remote site for animating an avatar image having facial characteristics which are based upon the facial characteristics determined by the tracking step.

9. A method for feature sensing as defined in claim 7, wherein the facial characteristic determined by the tracking step is whether mouth is open or closed.

10. A method for feature sensing as defined in claim 7, wherein the facial characteristic determined by the tracking step is whether eyes are open or closed.

11. A method for feature sensing as defined in claim 7, wherein the facial characteristic determined by the tracking step is whether a tongue is visble in the mouth.

12. A method for feature sensing as defined in claim 7, wherein the facial characteristic determined by the tracking step is based on facial wrinkles detected in the image.

13. A method for feature sensing as defined in claim 7, wherein the facial characteristic determined by the tracking step is based on hair type.

14. A method for feature sensing as defined in claim 7, wherein each facial characteristic is associated by training with an image tag that identifies an image segment of the image frame that is associated with the facial characteristic.

15. A method for feature sensing as defined in claim 14, wherein the image segments identified by the associated image tag are morphed into an avatar image.

16. A method for feature sensing as defined in claim 14, wherein the node locations and feature tags are used for volume morphing the corresponding image segments into a three-dimensional image.

17. A method for feature sensing as defined in claim 7, wherein the model graph comprises 18 location nodes associated with distinguishing features on a human face.

18. A method for feature sensing as defined in claim 17, wherein the 18 node locations of the face are associated with, respectively,

a right eye pupil;

a left eye pupil;

a top of a nose;

a right corner of a right eyebrow;

a left corner of the right eyebrow;

a left corner of a left eyebrow;

a right nostril;

a tip of the nose;

a left nostril;

a right corner of a mouth;

a center of an upper lip;

a left corner of the mouth;

a center of a lower lip;

a bottom of a right ear;

a top of the right ear;

a top of a left ear; and

a bottom of the left ear.

19. A method for facial feature sensing as defined in claim 1, wherein the node locations tracking step includes lip synching based on audio signals associated with movement of the node locations specific to a mouth generating the audio signals.

23. A method for facial feature sensing as defined in claim 1, wherein that the reinitializing step is performed using bunch graph matching.

24. A method for facial feature sensing as defined in claim 23, wherein that the bunch graph matching is performed using a partial bunch graph.

20. Apparatus for feature sensing on a sequence of image frames, comprising:

means for transforming each image frame using a wavelet transformation to generate a transformed image frame;

means for initializing nodes of a model graph, each node associated with a wavelet jet specific to a feature, to locations on the transformed image frame by moving the model graph across the transformed image frame and placing the model graph at a location in the transformed image frame of maximum jet similarity between the wavelet jets of the nodes and locations on the transformed image frame determined as the model graph is moved across the transformed image frame;

means for tracking the location of one or more nodes of the model graph between image frames; and

means for reinitializing a tracked node if the tracked node'"'"'s location deviates beyond a predetermined position constraint between image frames.

21. Apparatus for feature sensing as defined in claim 20, further comprising:

means for determining a facial characteristic; and

means for animating an avatar image having facial characteristics which are based upon the facial characteristics generated by the determining means.

22. Apparatus for feature sensing as defined in claim 21, wherein the model graph comprises 18 location nodes associated with distinguishing features on a human face.