FACIAL EXPRESSION REPRESENTATION APPARATUS

• US 20100211397A1
• Filed: 01/28/2010
• Published: 08/19/2010
• Est. Priority Date: 02/18/2009
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: KIM, Jung-bae, HWANG, Young-Kyoo, PARK, Chi-youn
• Granted Patent: US 8,396,708 B2
• Time in Patent Office: Days
• US Class Current: 704/276
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads

Abstract

An avatar facial expression representation technology is provided. The avatar facial expression representation technology estimates changes in emotion and emphasis in a user'"'"'s voice from vocal information, and changes in mouth shape of the user from pronunciation information of the voice. The avatar facial expression technology tracks a user'"'"'s facial movements and changes in facial expression from image information and may represent avatar facial expressions based on the result of the these operations. Accordingly, the avatar facial expressions can be obtained which are similar to actual facial expressions of the user.

Claims

1. An avatar facial expression representation apparatus, comprising:

a vocal information processing unit to output vocal information including at least one of an emotional change of a user and a point of emphasis of the user from vocal information generated by a user'"'"'s voice;

a pronunciation information processing unit to output pronunciation information including a change in mouth shape of the user from pronunciation information generated by the user'"'"'s voice;

an image information processing unit to output facial information by tracking facial movements and changes in facial expression of the user from image information; and

a facial expression processing unit to represent a facial expression of an avatar using at least one of the vocal information output from the vocal information processing unit, the pronunciation information output from the pronunciation information processing unit, and the facial information output from the image information processing unit.

2. The avatar facial expression representation apparatus of claim 1, wherein the vocal information processing unit comprises:

a parameter extracting portion to extract a parameter related to a change in emotion from the vocal information of the voice;

an emotion change estimating portion to estimate a change in emotion by monitoring a long-term change of the parameter extracted by the parameter extracting portion;

a point of emphasis estimating portion to estimate a point of emphasis by monitoring a short-term change of the parameter extracted by the parameter extracting portion; and

a vocal information output portion to create the vocal information based on the change in emotion estimated by the emotion change estimating portion and the point of emphasis estimated by the point of emphasis estimating portion, and to output the created vocal information.

3. The avatar facial expression representation apparatus of claim 2, wherein the parameter related to a change in emotion includes an intensity of a voice signal, a pitch of a voice sound, and voice quality information.

4. The avatar facial expression representation apparatus of claim 2, wherein the long-term change of the parameter is identified by detecting a change of the parameter or a changing speed of the parameter during a predetermined first reference time.

5. The avatar facial expression representation apparatus of claim 4, wherein the short-term change of the parameter is identified by detecting a change of the parameter or a changing speed of the parameter during a predetermined second reference time which is set shorter than the predetermined first reference time.

6. The avatar facial expression representation apparatus of claim 5, wherein the point of emphasis estimating portion estimates a point of emphasis as a vocal point where a sound is changed.

7. The avatar facial expression representation apparatus of claim 1, wherein the pronunciation information processing unit comprises:

a parameter extracting portion to extract a parameter related to a change in mouth shape from the pronunciation information generated by the user'"'"'s voice;

a mouth shape estimating unit to estimate a change in mouth shape of the user based on the parameter extracted by the parameter extracting unit; and

a pronunciation information output portion to create the pronunciation information based on the change in mouth shape estimated by the mouth shape estimating portion, and to output the created pronunciation information.

8. The avatar facial expression representation apparatus of claim 7, wherein the parameters related to the change in mouth shape includes information about at least one of how far the upper and lower lips are apart, how wide the lips are open, and how far the lips are pushed forward.

9. The avatar facial expression representation apparatus of claim 7, wherein the mouth shape estimating portion searches a database for an articulation group to which a sound of the user'"'"'s voice belongs to, extracts a parameter corresponding to a found articulation group, and estimates the change in mouth shape of the user based on the extracted parameter, wherein the database stores articulation groups into groups that include phonemes similar in lip shape.

10. The avatar facial expression representation apparatus of claim 1, wherein the image information processing unit comprises:

an image data analyzing portion to extract a location of a feature point that represents a facial expression of the user from the image information;

a facial expression change portion to track facial movements and the changes in facial expression of the user based on the location of a feature point extracted by the image information analyzing portion; and

a facial information output portion to create the facial information based on the facial movements and changes in facial expressions of the user tracked by the facial expression change tracking portion and to output the created facial information.

11. The avatar facial expression representation apparatus of claim 1, wherein the facial expression processing unit comprises:

a facial expression representing portion to represent a general facial expression of an avatar according to changes in emotion included in the vocal information output from the vocal information processing unit;

a first correcting portion to correct the facial expression of the avatar according to the change in mouth shape of the user included in the pronunciation information output from the pronunciation information processing unit;

a second correcting portion to correct the facial expression of the avatar according to the point of emphasis included in the vocal information output from the vocal information processing unit; and

a third correcting portion to correct the facial expression of the avatar according to the facial movements and changes in facial expression of the user included in the facial information output from the image information processing unit.

12. The avatar facial expression representation apparatus of claim 1, further comprising:

a reliability evaluating unit to evaluate a reliability of at least one of the vocal information output from the vocal information processing unit, a reliability of the pronunciation information output from the pronunciation information processing unit, and a reliability of the facial information output from the image information processing unit.

13. The avatar facial expression representation apparatus of claim 12, wherein the facial expression processing unit discards, when representing the facial expression of the avatar, information which is determined by the reliability evaluating unit to have a lower reliability.

14. The avatar facial expression representation apparatus of claim 12, wherein the reliability evaluating unit evaluates the reliability of the vocal information based on correlation between a facial expression due to the change in emotion included in the vocal information and a facial expression according to an emotion model.

15. The avatar facial expression representation apparatus of claim 12, wherein the reliability evaluating unit evaluates the reliability of the pronunciation information based on a length of silence or the quantity of noise included in a voice sound.

16. The avatar facial expression representation apparatus of claim 12, wherein the reliability evaluating unit evaluates the reliability of the facial information based on locations of feature points that represent the facial expression of the user and changes in facial expression.