Facial expression representation apparatus

• US 8,396,708 B2
• Filed: 01/28/2010
• Issued: 03/12/2013
• Est. Priority Date: 02/18/2009
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Hwang, Young-kyoo, Kim, Jung-bae, Park, Chi-youn
• Primary Examiner(s): PULLIAS, JESSE SCOTT
• Application Number:
  US12/695,185
• Publication Number:
  US 20100211397A1
• Time in Patent Office: 1,139 Days
• Field of Search: 704/231, 704/258, 704/270
• US Class Current: 704/231
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads

Abstract

An avatar facial expression representation technology is provided. The avatar facial expression representation technology estimates changes in emotion and emphasis in a user'"'"'s voice from vocal information, and changes in mouth shape of the user from pronunciation information of the voice. The avatar facial expression technology tracks a user'"'"'s facial movements and changes in facial expression from image information and may represent avatar facial expressions based on the result of the these operations. Accordingly, the avatar facial expressions can be obtained which are similar to actual facial expressions of the user.

Claims

1. An avatar facial expression representation apparatus, comprising:

a vocal information processor to output vocal information including at least one of an emotional change of a user and a point of emphasis of the user from vocal information generated by the voice of the user;

a pronunciation information processor to output pronunciation information including a change in mouth shape of the user from pronunciation information generated by the voice of the user;

an image information processor to output facial information by tracking facial movements and changes in facial expression of the user from image information; and

a facial expression processor to represent a facial expression of an avatar using at least one of the vocal information output from the vocal information processor, the pronunciation information output from the pronunciation information processor, and the facial information output from the image information processor.

2. The avatar facial expression representation apparatus of claim 1, wherein the vocal information processor comprises:

a parameter extracting portion to extract a parameter related to a change in emotion from the vocal information of the voice of the user;

an emotion change estimating portion to estimate a change in emotion by monitoring a long-term change of the parameter extracted by the parameter extracting portion;

a point of emphasis estimating portion to estimate a point of emphasis by monitoring a short-term change of the parameter extracted by the parameter extracting portion; and

a vocal information output portion to create the vocal information based on the change in emotion estimated by the emotion change estimating portion and the point of emphasis estimated by the point of emphasis estimating portion, and to output the created vocal information.

3. The avatar facial expression representation apparatus of claim 2, wherein the parameter related to a change in emotion includes an intensity of a voice signal, a pitch of a voice sound, and voice quality information.

4. The avatar facial expression representation apparatus of claim 2, wherein the long-term change of the parameter is identified by detecting a change of the parameter or a changing speed of the parameter during a predetermined first reference time.

5. The avatar facial expression representation apparatus of claim 4, wherein the short-term change of the parameter is identified by detecting a change of the parameter or a changing speed of the parameter during a predetermined second reference time which is set shorter than the predetermined first reference time.

6. The avatar facial expression representation apparatus of claim 5, wherein the point of emphasis estimating portion estimates a point of emphasis as a vocal point where a sound is changed.

7. The avatar facial expression representation apparatus of claim 1, wherein the pronunciation information processor comprises:

a parameter extracting portion to extract a parameter related to a change in mouth shape from the pronunciation information generated by the voice of the user;

a mouth shape estimator to estimate a change in mouth shape of the user based on the parameter extracted by a parameter extractor; and

a pronunciation information output portion to create the pronunciation information based on the change in mouth shape estimated by the mouth shape estimating portion, and to output the created pronunciation information.

8. The avatar facial expression representation apparatus of claim 7, wherein the parameters related to the change in mouth shape include information about at least one of how far the upper and lower lips are apart, how wide the lips are open, and how far the lips are pushed forward.

9. The avatar facial expression representation apparatus of claim 7, wherein the mouth shape estimating portion searches a database for an articulation group to which a sound of the voice of the user belongs to, extracts a parameter corresponding to a found articulation group, and estimates the change in mouth shape of the user based on the extracted parameter, wherein the database stores articulation groups into groups that include phonemes similar in lip shape.

10. The avatar facial expression representation apparatus of claim 1, wherein the image information processor comprises:

an image data analyzing portion to extract a location of a feature point that represents a facial expression of the user from the image information;

a facial expression change portion to track facial movements and the changes in facial expression of the user based on the location of a feature point extracted by the image information analyzing portion; and

a facial information output portion to create the facial information based on the facial movements and changes in facial expressions of the user tracked by the facial expression change tracking portion and to output the created facial information.

11. The avatar facial expression representation apparatus of claim 1, wherein the facial expression processor comprises:

a facial expression representing portion to represent a general facial expression of an avatar according to changes in emotion included in the vocal information output from the vocal information processor;

a first correcting portion to correct the facial expression of the avatar according to the change in mouth shape of the user included in the pronunciation information output from the pronunciation information processor;

a second correcting portion to correct the facial expression of the avatar according to the point of emphasis included in the vocal information output from the vocal information processor; and

a third correcting portion to correct the facial expression of the avatar according to the facial movements and changes in facial expression of the user included in the facial information output from the image information processor.

12. The avatar facial expression representation apparatus of claim 1, further comprising:

a reliability evaluator to evaluate a reliability of at least one of the vocal information output from the vocal information processor, a reliability of the pronunciation information output from the pronunciation information processor, and a reliability of the facial information output from the image information processor.

13. The avatar facial expression representation apparatus of claim 12, wherein the facial expression processor discards, when representing the facial expression of the avatar, information which is determined by the reliability evaluator to have a lower reliability.

14. The avatar facial expression representation apparatus of claim 12, wherein the reliability evaluator evaluates the reliability of the vocal information based on correlation between a facial expression due to the change in emotion included in the vocal information and a facial expression according to an emotion model.

15. The avatar facial expression representation apparatus of claim 12, wherein the reliability evaluator evaluates the reliability of the pronunciation information based on a length of silence or the quantity of noise included in a voice sound.

16. The avatar facial expression representation apparatus of claim 12, wherein the reliability evaluator evaluates the reliability of the facial information based on locations of feature points that represent the facial expression of the user and changes in facial expression.

17. The avatar facial expression representation apparatus of claim 1, wherein the point of emphasis is a shout.

18. The avatar facial expression representation apparatus of claim 1, wherein the facial expression processor uses at least one of the vocal information related to emotional changes of the user and the point of emphasis made by the user, the pronunciation information related to changes in mouth shape of the user, and the facial information related to the facial movements of the user and changes in facial expression to represent the facial expression of an avatar in synchronization with the actual facial expressions of the user.

19. The avatar facial expression representation apparatus of claim 1, wherein a probability of a sound belonging to an each articulation group is calculated and a mouth shape of the each articulation group is determined by averaging with the calculated probability as a weight.