Mouth shape synthesizing

• US 6,332,123 B1
• Filed: 01/19/1994
• Issued: 12/18/2001
• Est. Priority Date: 03/08/1989
• Status: Expired due to Fees

• Current Assignee: KDDI Corporation
• Original Assignee: Kokusai Densin Denwa Kabushiki Kaisha
• Inventors: Yamamoto, Seiichi, Higuchi, Norio, Hatori, Yoshinori, Koike, Atsushi, Kaneko, Masahide
• Primary Examiner(s): Knepper, David D.
• Application Number:
  US08/183,671
• Time in Patent Office: 2,890 Days
• Field of Search: 395/2, 395/2.44, 395/2.69, 395/2.67, 395/2.79, 395/2.85, 395/2.87, 381/44, 381/48, 345/122, 358/426, 434/185
• US Class Current: 704/276
• CPC Class Codes: G06T 13/205 driven by audio data
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads

Abstract

A picture synthesizing apparatus, and method for synthesizing a moving picture of a person’s face having mouth-shape variations from a train of input characters, wherein the method steps comprise developing from the train of input character a train of phonemes, utilizing a speech synthesis technique outputting, for each phoneme, a corresponding vocal sound feature including articulation mode and its duration of each corresponding phoneme of the train of phonemes. Determining for each phoneme a mouth-shape feature corresponding to each phoneme on the basis of the corresponding vocal sound feature, the mouth-shape feature including the degree of opening of the mouth, the degree of roundness of the lips, the height of the lower jaw in a raised and a lowered position, and the degree to which the tongue is seen. Determining values of mouth-shape parameters, for each phoneme, for representing a concrete mouth-shape on the basis of the mouth-shape feature; and controlling the values of the mouth-shape parameters for each phoneme, for each frame of the moving picture in accordance with the duration of each phoneme, thereby synthesizing the moving picture having mouth-shape variations matched with a speech output audible in case of reading the train of input characters.

Claims

1. A picture synthesizing method for synthesizing a moving picture of a person'"'"'s face having mouth-shape variations from a train of input characters,

comprising the steps of:

developing from the train of input characters a train of phonemes, utilizing a speech synthesis technique outputting, for each phoneme, a corresponding vocal sound feature including articulation mode and its duration of each corresponding phoneme of the train of phonemes;

determining for each phoneme a mouth-shape feature corresponding to each phoneme on the basis of the corresponding vocal sound feature, said mouth-shape feature including the degree of opening of the mouth, the degree of roundness of the lips, the height of the lower jaw in a raised and a lowered position, and the degree to which the tongue is seen,

determining values of mouth-shape parameters, for each phoneme, for representing a concrete mouth-shape on the basis of the mouth-shape feature; and

controlling the values of the mouth-shape parameters, for each phoneme, for each frame of the moving picture in accordance with the duration of each phoneme, thereby synthesizing the moving picture having mouth-shape variations matched with a speech output audible in case of reading the train of input characters.

2. A picture synthesizing apparatus comprising:

an input terminal for receiving a train of input characters;

a speech synthesizer for developing from the train of input characters a train of phonemes, utilizing a speech synthesis technique and outputting, for each phoneme, a corresponding vocal sound feature including articulation mode and its duration of each corresponding phoneme of the train of phonemes;

a converter for converting the corresponding vocal sound feature for each corresponding phoneme into a mouth-shape feature including the degree of opening the mouth, the degree of roundness of the lips, the height of the lower jaw in a raised and lowered position, and the degree to which the tongue is seen;

means for defining a conversion table having established correspondence between various mouth-features and mouth-shape parameters for representing concrete mouth-shape;

means for obtaining from the conversion table mouth-shape parameters each corresponding to an individual mouth-shape feature for each phoneme provided by the converter;

a time adjuster having an output whereby values of the mouth-shape parameters from said means for obtaining are controlled in accordance with the duration of each corresponding phoneme from the speech synthesizer for producing a moving picture as a train of pictures spaced apart for a fixed period of time; and

a picture generator for generating the moving picture having mouth-shape variations matched with a speech output audible in case of reading the train of input characters in accordance with the values of the mouth-shape parameters from said means for obtaining mouth-shape parameters under control of the time adjuster.

3. A picture synthesizing apparatus according to claim 2, further comprising:

a transition detector for detecting a transition from a certain phoneme to the next in accordance with the output of the time adjuster, a memory capable of storing for at least one frame period the values of the mouth-shape parameters used in the picture generator, and a mouth-shape parameter modifier for obtaining an intermediate value between the value of the mouth-shape parameter stored in the memory and the value of the mouth-shape parameter provided from said means for obtaining the mouth-shape parameters, whereby during the transition from the certain phoneme to the next an intermediate mouth shape corresponding to said intermediate value is generated, producing the moving picture of a person'"'"'s face with smooth mouth-shape variations.