METHOD AND APPARATUS FOR CREATING FACE CHARACTER BASED ON VOICE

• US 20100094634A1
• Filed: 08/26/2009
• Published: 04/15/2010
• Est. Priority Date: 10/14/2008
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: PARK, Bong-cheol
• Granted Patent: US 8,306,824 B2
• Time in Patent Office: Days
• US Class Current: 704/270
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence

Abstract

An apparatus and method of creating a face character which corresponds to a voice of a user is provided. To create various facial expressions with fewer key models, a face character is divided in a plurality of areas and a voice sample is parameterized corresponding to pronunciation and emotion. If the user'"'"'s voice is input, a face character image corresponding to divided face areas is synthesized using key models and data about parameters corresponding to the voice sample to synthesize an overall face character image using the synthesized face character image corresponding to the divided face areas.

Claims

1. An apparatus to create a face character based on a voice of a user, comprising:

a preprocessor configured to divide a face character image in a plurality of areas using multiple key models corresponding to the face character image, and to extract data about at least one parameter to recognize pronunciation and emotion from an analyzed voice sample; and

a face character creator configured to extract data about at least one parameter from input voice in frame units, and to synthesize in frame units the face character image corresponding to each divided face character image area based on the data about at least one parameter.

2. The apparatus of claim 1, wherein the face character creator calculates a mixed weight to determine a mixed ratio of the multiple key models using the data about at least one parameter.

3. The apparatus of claim 1, wherein the multiple key models comprise key models corresponding to pronunciations of vowels and consonants and key models corresponding to emotions.

4. The apparatus of claim 1, wherein the preprocessor divides the face character image using data modeled in a spring-mass network having masses corresponding to vertices of the face character image and springs corresponding to edges of the face character image.

5. The apparatus of claim 4, wherein the preprocessor selects feature points having a spring variation more than a predetermined threshold in springs between a mass and neighboring masses with respect to a reference model corresponding to each of the key models, measures coherency in organic motion of the feature points to form groups of the feature points, and divides the vertices by grouping the remaining masses not selected as the feature points into the feature point groups.

6. The apparatus of claim 1, wherein in response to creating the parameters corresponding to the user'"'"'s voice, the preprocessor represents parameters for each vowel on a three formant parameter space from the voice sample, creates consonant templates to identify each consonant from the voice sample, and sets space areas corresponding to each emotion on an emotion parameter space to represent parameters corresponding to the analyzed pitch, intensity and tempo of the voice sample.

7. The apparatus of claim 6, wherein the face character creator:

calculates weight of each vowel key model based on a distance between a position of a vowel parameter extracted from the input voice frame and a position of each vowel parameter extracted from the voice sample on the formant parameter space;

determines a consonant key model through pattern matching between the consonant template extracted from the input voice frame and the consonant templates of the voice sample; and

calculates weight of each emotion key model based on a distance between a position of an emotion parameter extracted from the input voice frame and the emotion area on the emotion parameter space.

8. The apparatus of claim 7, wherein the face character creator:

synthesizes a lower face area by applying the weight of each vowel key model to displacement of vertices of each vowel key model with respect to a reference key model or using the selected consonant key models; and

synthesizes an upper face area by applying the weight of each emotion key model to displacement of vertices of each emotion key model with respect to a reference key model.

9. The apparatus of claim 8, wherein the face character creator creates a face character image corresponding to input voice in frame units by synthesizing an upper face area and a lower face area.

10. A method of creating a face character based on voice, the method comprising:

dividing a face character image in a plurality of areas using multiple key models corresponding to the face character image;

extracting data about at least one parameter to recognize pronunciation and emotion from an analyzed voice sample;

in response to a voice being input, extracting data about at least one parameter from voice in frame units; and

synthesizing in frame units the face character image corresponding to each divided face character image area based on the data about at least one parameter.

11. The method of claim 10, wherein the synthesizing comprises calculating a mixed weight to determine a mixed ratio of the multiple key models using the data about at least one parameter.

12. The method of claim 10, wherein the multiple key models comprise key models corresponding to pronunciations of vowels and consonants and key models corresponding to emotions.

13. The method of claim 12, wherein the dividing comprises using data modeled in a spring-mass network having masses corresponding to vertices of the face character image and springs corresponding to edges of the face character image.

14. The method of claim 13, wherein the dividing comprises:

selecting feature points having a spring variation more than a predetermined threshold in springs between a mass and neighboring masses with respect to a reference model corresponding to each of the key models;

measuring coherency in organic motion of the feature points to form groups of the feature points; and

dividing the vertices by grouping the remaining masses not selected as the feature points into the feature point groups.

15. The method of claim 10, wherein the extracting of the data about the at least one parameter to recognize pronunciation and emotion from the analyzed voice sample comprises:

representing parameters corresponding to each vowel on a three formant parameter space from the voice sample;

creating consonant templates to identify each consonant from the voice sample; and

setting space areas corresponding to each emotion on an emotion parameter space to represent parameters corresponding to analyzed pitch, intensity and tempo of the voice sample.

16. The method of claim 15, wherein the synthesizing comprises:

calculating weight of each vowel key model based on a distance between a position of a vowel parameter extracted from the input voice frame and a position of each vowel parameter extracted from the voice sample on the formant parameter space;

determining a consonant key model through pattern matching between the consonant template extracted from the input voice frame and the consonant templates of the voice sample; and

calculating weight of each emotion key model based on a distance between a position of an emotion parameter extracted from the input voice frame and the emotion area on the emotion parameter space.

17. The method of claim 16, wherein the synthesizing comprises:

synthesizing a lower face area by applying the weight of each vowel key model to displacement of vertices of each vowel key model with respect to a reference key model or using the selected consonant key models; and

synthesizing an upper face area by applying the weight of each emotion key model to displacement of vertices of each emotion key model with respect to a reference key model.

18. The method of claim 17, further comprising creating a face character image corresponding to input voice in frame units by synthesizing an upper face area and a lower face area.