Method for automatically animating lip synchronization and facial expression of animated characters

• US 20020101422A1
• Filed: 09/21/2001
• Published: 08/01/2002
• Est. Priority Date: 10/02/1997
• Status: Active Grant

• Current Assignee: McRo, Inc.
• Original Assignee: Planet Blue
• Inventors: Rosenfeld, Maury
• Granted Patent: US 6,611,278 B2
• Time in Patent Office: Days
• US Class Current: 345/473
• CPC Class Codes: G06T 13/205 driven by audio data
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 2210/44 Morphing
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads
• CPC Class Codes: Y10S 345/951 Key frame processing
• CPC Class Codes: Y10S 345/955 Morphing

Abstract

A method for controlling and automatically animating lip synchronization and facial expressions of three dimensional animated characters using weighted morph targets and time aligned phonetic transcriptions of recorded text. The method utilizes a set of rules that determine the systems output comprising a stream of morph weight sets when a sequence of timed phonemes and/or other timed data is encountered. Other data, such as timed emotional state data or emotemes such as “surprise, “disgust, “embarrassment”, “timid smile”, or the like, may be inputted to affect the output stream of morph weight sets, or create additional streams.

Claims

1. A method for automatically animating lip synchronization and facial expression of three dimensional characters for films, videos, cartoons, and other animation products, comprising:

configuring a set of default correspondence rules between a plurality of timed input phonemes or other timed data and a plurality of morph weight sets; and

specifying a plurality of morph weight set transition rules for specifying durational data for the generation of transitionary curves between said plurality of morph weight sets, allowing for the production of a stream of specified morph weight sets to be processed by a computer animation system for integration with other animation, whereby animated lip synchronization and facial expression of animated characters may be automatically produced.

2. The method for automatically animating lip synchronization and facial expression of three dimensional characters of claim 1, further including the step of producing a morph weight stream having entries at a transition start and a transition end time which function as keyframes.

3. The method for automatically animating lip synchronization and facial expression of three dimensional characters of claim 2, further including the step of evaluating a selected morph weight set by interpolating between said keyframes to calculate an output stream at a desired time frame.

4. The method for automatically animating lip synchronization and facial expression of three dimensional characters of claim 3, further including the step of applying a plurality of post processing rules to modify an individual member or a set of members of said selected morph weight sets.

5. A method for automatically animating lip synchronization and facial expression of three dimensional characters for use with a computer animation system, comprising:

determining means for producing a stream of morph weight sets when a sequence of phonemes or other timed aligned data is encountered;

evaluating a plurality of time aligned phonetic transcriptions or other timed data against said determining means for producing a stream or morph weight sets;

applying said determining means for producing a stream of morph weight sets to generate an output morph weight set stream, allowing for an appropriate morph weight set correspondence with each of a plurality of time aligned phonetic transcription sub-sequences and correct time parameters applied to a plurality of morph weight set transitions between a representation of a prior time aligned phonetic transcription sub-sequence and a current one, whereby lip synchronization and facial expressions of animated characters is automatically controlled and produced.

6. The method for automatically animating lip synchronization and facial expression of three dimensional characters for use with a computer animation system or claim 5, wherein said determining means for producing a stream of morph weight sets when a sequence of phonemes and/or other timed aligned data is encountered comprises a set of predetermined rules determining what the output morph eight set stream will be when any sequence of phonemes is encountered.

7. The method for automatically animating lip synchronization and facial expression of three dimensional characters for use with a computer animation system of claim 6, wherein said set of predetermined rules includes a set of default rules.

8. The method for automatically animating lip synchronization and facial expression of three dimensional characters for use with a computer animation system of claim 6, wherein said set of predetermined rules includes a set of secondary rules for substituting an alternative morph weight set correspondence and/or a transition rule if preselected criteria are met.

9. The method for automatically animating lip synchronization and facial expression of three dimensional characters for use with a computer animation system of claim 6, wherein said set of predetermined rules includes a set of post processing rules allowing for the further manipulation of said morph weight set stream after said default rule and/or secondary rule is applied.

10. The method for automatically animating lip synchronization and facial expression of three dimensional characters for use with a computer animation system of claim 9, wherein said set of post processing rules are utilized to modify a group of selected members of said morph weight sets as determined by said default rules and said secondary rules.