Wavelet-based facial motion capture for avatar animation

• US 20010033675A1
• Filed: 05/31/2001
• Published: 10/25/2001
• Est. Priority Date: 04/13/1998
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Nevengineering Incorporated (Alphabet Inc.)
• Inventors: Elagin, Egor Valerievich, Steffens, Johannes Bernhard, Maurer, Thomas, Neven, Hartmut, Nocera, Luciano Pasquale Agostino
• Granted Patent: US 6,580,811 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/20064 Wavelet transform [DWT]
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/262 using transform domain methods, e.g. Fourier domain methods
• CPC Class Codes: G06V 10/426 Graphical representations
• CPC Class Codes: G06V 10/449 Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships

Abstract

The present invention is embodied in an apparatus, and related method, for sensing a person'"'"'s facial movements, features and characteristics and the like to generate and animate an avatar image based on facial sensing. The avatar apparatus uses an image processing technique based on model graphs and bunch graphs that efficiently represent image features as jets. The jets are composed of wavelet transforms processed at node or landmark locations on an image corresponding to readily identifiable features. The nodes are acquired and tracked to animate an avatar image in accordance with the person'"'"'s facial movements. Also, the facial sensing may use jet similarity to determine the person'"'"'s facial features and characteristic thus allows tracking of a person'"'"'s natural characteristics without any unnatural elements that may interfere or inhibit the person'"'"'s natural characteristics.

Claims

1. A method for feature sensing on a sequence of image frames, comprising:

a step for transforming each image frame using a wavelet transformation to generate a transformed image frame;

a step for initializing node locations associated with wavelets jets of a model graph to the transformed image frame by moving the model graph across the transformed image frame and placing the model graph at a location in the transformed image frame of maximum jet similarity between the wavelet jets at the node locations and the transformed image frame;

a step for tracking the location of one or more node locations of the model graph between image frames; and

a step for reinitializing a tracked node if the node'"'"'s position deviates beyond a predetermined position constraint between image frames.

2. A method for feature sensing as defined in

claim 1

, wherein the model graph used in the initializing step is based on a predetermined pose.

3. A method for feature sensing as defined in

claim 1

, wherein the tracking step tracks the node locations using elastic bunch graph matching.

4. A method for feature sensing as defined in

claim 1

, wherein the tracking step uses linear position prediction to predict node locations in a subsequent image frame and the reinitializing step reinitializes a node location based on a deviation from the predicted node location that is greater than a predetermined threshold deviation.

5. A method for feature sensing as defined in

claim 1

, wherein the predetermined position constraint is based on a geometrical position constraint associated with relative positions between the node locations.

6. A method for feature sensing as defined in

claim 1

, wherein the node locations are transmitted to a remote site for animating an avatar image.

7. A method for feature sensing as defined in

claim 1

, wherein the tracking step includes determining a facial characteristic.

8. A method for feature sensing as defined in

claim 7

, further comprising transmitting the node locations and facial characteristics to a remote site for animating an avatar image having facial characteristics which are based upon the facial characteristics determined by the tracking step.

9. A method for feature sensing as defined in

claim 7

, wherein the facial characteristic determined by the tracking step is whether mouth is open or closed.

10. A method for feature sensing as defined in

claim 7

, wherein the facial characteristic determined by the tracking step is whether eyes are open or closed.

11. A method for feature sensing as defined in

claim 7

, wherein the facial characteristic determined by the tracking step is whether tongue is visible in the mouth.

12. A method for feature sensing as defined in

claim 7

, wherein the facial characteristic determined by the tracking step is based on facial wrinkles detected in the image.

13. A method for feature-sensing as defined in

claim 7

, wherein the facial characteristic determined by the tracking step is based on hair type.

14. A method for feature sensing as defined in

claim 7

, wherein each facial characteristic is associated by training with an image tag that identifies an image segment of the image frame that is associated with the facial characteristic.

15. A method for feature sensing as defined in

claim 14

, wherein the image segments identified by the associated image tag are morphed into an avatar image.

16. A method for feature sensing as defined in

claim 14

, wherein the node locations and feature tags are used for volume morphing the corresponding image segments into a three-dimensional image.

17. A method for feature sensing as defined in

claim 7

, wherein the model graph comprises 18 location nodes associated with distinguishing features on a human face.

18. A method for feature sensing as defined in

claim 17

, wherein the 18 node locations of the face are associated with, respectively,

19. A method for facial feature sensing as defined in

claim 11

wherein the node locations tracking step includes lip synching based on audio signals associated with movement of the node locations of a mouth generating the audio signals.

20. A method for individualizing a head model based on facial feature finding, wherein facial feature finding is based on elastic bunch graph matching.

21. A method for individualizing a head model as defined in

claim 20

, wherein the matching is performed using a coarse-to-fine approach.

22. Apparatus for feature sensing on a sequence of image frames, comprising:

means for transforming each image frame using a wavelet transformation to generate a transformed image frame;

means for initializing node locations associated with wavelets jets of a model graph to the transformed image frame by moving the model graph across the transformed image frame and placing the model graph at a location in the transformed image frame of maximum jet similarity between the wavelet jets at the node locations and the transformed image frame;

means for tracking the location of one or more node locations of the model graph between image frames; and

means for reinitializing a tracked node if the node'"'"'s position deviates beyond a predetermined position constraint between image frames.

23. Apparatus for feature sensing as defined in

claim 22

, further comprising:

means for determining a facial characteristic; and

means for animating an avatar image having facial characteristics which are based upon the facial characteristics generated by the determining means.

24. Apparatus for feature sensing as defined in

claim 23

, wherein the model graph comprises 18 location nodes associated with distinguishing features on a human face.

25. A model graph for facial image analysis comprising 18 location nodes associated with distinguishing features on a human face.

26. A model graph for facial image analysis as defined in

claim 25

, wherein the 18 node locations of the face are associated with, respectively,