EMOTION RECOGNITION APPARATUS USING FACIAL EXPRESSION AND EMOTION RECOGNITION METHOD USING THE SAME

• US 20140022370A1
• Filed: 12/12/2012
• Published: 01/23/2014
• Est. Priority Date: 07/23/2012
• Status: Abandoned Application

• Current Assignee: The Industry & Academic Cooperation In Chungnam National University
• Original Assignee: Industry-Academic Cooperation Foundation Yonsei University
• Inventors: SOHN, Jin Hun, KIM, Eung-Hee, CHOI, Sangsup, LEE, Kyungmin
• Application Number:
  US13/711,781
• Publication Number:
  US 20140022370A1
• Time in Patent Office: Days
• US Class Current: 348/78
• CPC Class Codes: G06V 40/176 Dynamic expression

Abstract

The present invention relates to an emotion recognition apparatus using facial expressions including: a camera adapted to acquire a first video of an object corresponding to each of a plurality of emotions classified by a previously set reference; a user input unit adapted to receive a plurality of first frames in the first video designated by a user; a control unit adapted to recognize the face of the object contained in the plurality of first frames, extracting the facial elements of the object by using the recognized face, and extracting the variation patterns of the plurality of emotions by using the facial elements; and a memory adapted to store the extracted variation patterns of the plurality of emotions.

Claims

1. An emotion recognition apparatus using facial expressions comprising:

a camera adapted to acquire a first video of an object corresponding to each of a plurality of emotions classified by a previously set reference;

a user input unit adapted to receive a plurality of first frames in the first video designated by a user;

a control unit adapted to recognize the face of the object contained in the plurality of first frames, extracting the facial elements of the object by using the recognized face, and extracting the variation patterns of the plurality of emotions by using the facial elements; and

a memory adapted to store the extracted variation patterns of the plurality of emotions, wherein if a second video of the object is acquired through the camera, a first variation pattern of the facial elements of the object contained in the second video is extracted, and the emotion corresponding to the variation pattern that is the same as the first variation pattern from the plurality of variations patterns stored in the memory is determined as the emotion of the object by means of the control unit.

3. The emotion recognition apparatus using facial expressions according to claim 1, wherein the plurality of emotions classified by the previously set reference is joy, surprise, sadness, anger, fear and disgust.

4. The emotion recognition apparatus using facial expressions according to claim 1, wherein the first video and the second video are moving video data or still video data acquired by photographing the face of the object.

5. The emotion recognition apparatus using facial expressions according to claim 1, wherein the facial elements of the object comprise at least one of eyes, eyebrows, the middle of the forehead, and mouth.

6. The emotion recognition apparatus using facial expressions according to claim 1, wherein the control unit extracts the facial elements of the object by using an ASM (Active Shape Model) algorithm.

7. The emotion recognition apparatus using facial expressions according to claim 6, wherein the control unit extracts the features of a plurality of coordinates x and y by using the face of the object contained in the first video and the second video and thus extracts the facial elements of the object by using the extracted features of the plurality of coordinates x and y.

8. The emotion recognition apparatus using facial expressions according to claim 6, wherein the first video is divided into 9 sections through the user input unit, and the number of the plurality of first frames is 10 inclusive of start and end frames of each section.

9. The emotion recognition apparatus using facial expressions according to claim 1, wherein the control unit extracts the variation pattern of each of the plurality of emotions by using an HMM (Hidden Markov Model) algorithm.

2. An emotion recognition apparatus using facial expressions comprising:

a camera adapted to acquire a first video of an object corresponding to each of a plurality of emotions classified by a previously set reference;

a user input unit adapted to receive a plurality of first frames in the first video designated by a user;

a control unit adapted to recognize the face of the object contained in the plurality of first frames, extracting the facial elements of the object by using the recognized face, and extracting the variation patterns of the plurality of emotions by using the facial elements; and

a memory adapted to store the extracted variation patterns of the plurality of emotions, wherein if a second video of the object is acquired through the camera, a first variation pattern of the facial elements of the object contained in the second video is extracted, and the emotion corresponding to the variation pattern that is most similar to the first variation pattern from the plurality of variations patterns stored in the memory is determined as the emotion of the object by means of the control unit.

10. An emotion recognition method using facial expressions comprising the steps of:

acquiring a first video of an object corresponding to each of a plurality of emotions classified by a previously set reference;

receiving a plurality of first frames designated in the first video;

recognizing the face of the object contained in the plurality of first frames;

extracting the facial elements of the object by using the recognized face;

extracting the variation patterns of the plurality of emotions by using the facial elements;

storing the extracted variation patterns of the plurality of emotions;

acquiring a second video of the object;

extracting a first variation pattern of the facial elements of the object contained in the second video; and

determining the emotion corresponding to the variation pattern that is the same as the first variation pattern from the plurality of stored variations patterns as the emotion of the object.

12. The emotion recognition method using facial expressions according to claim 10, wherein the plurality of emotions classified by the previously set reference is joy, surprise, sadness, anger, fear and disgust.

13. The emotion recognition method using facial expressions according to claim 10, wherein the first video and the second video are moving video data or still video data acquired by photographing the face of the object.

14. The emotion recognition method using facial expressions according to claim 10, wherein the facial elements of the object comprise at least one of eyes, eyebrows, the middle of the forehead, and mouth.

15. The emotion recognition method using facial expressions according to claim 10, wherein the control unit extracts the facial elements of the object by using an ASM (Active Shape Model) algorithm.

16. The emotion recognition method using facial expressions according to claim 15, wherein the step of extracting the facial elements of the object comprises the steps of:

extracting the features of a plurality of coordinates x and y by using the face of the object contained in the first video and the second video; and

extracting the facial elements of the object by using the extracted features of the plurality of coordinates x and y.

17. The emotion recognition method using facial expressions according to claim 15, wherein the step of receiving the plurality of first frames designated in the first video comprises the steps of:

dividing the first video into 9 sections; and

designating 10 frames inclusive of start and end frames of each of the 9 sections as the plurality of first frames.

18. The emotion recognition method using facial expressions according to claim 10, wherein the control unit extracts the variation pattern of each of the plurality of emotions by using an HMM (Hidden Markov Model) algorithm.

11. An emotion recognition method using facial expressions comprising the steps of:

acquiring a first video of an object corresponding to each of a plurality of emotions classified by a previously set reference;

receiving a plurality of first frames designated in the first video;

recognizing the face of the object contained in the plurality of first frames;

extracting the facial elements of the object by using the recognized face;

extracting the variation patterns of the plurality of emotions by using the facial elements;

storing the extracted variation patterns of the plurality of emotions;

acquiring a second video of the object;

extracting a first variation pattern of the facial elements of the object contained in the second video; and

determining the emotion corresponding to the variation pattern that is most similar to the first variation pattern from the plurality of stored variations patterns as the emotion of the object.

19. An emotion recognition method using facial expressions in a recording medium where programs of commands executed by a digital processing device are typologically set in such a manner as to be readable by means of the digital processing device, the method comprising the steps of:

acquiring a first video of an object corresponding to each of a plurality of emotions classified by a previously set reference;

receiving a plurality of first frames designated in the first video;

recognizing the face of the object contained in the plurality of first frames;

extracting the facial elements of the object by using the recognized face;

extracting the variation patterns of the plurality of emotions by using the facial elements;

storing the extracted variation patterns of the plurality of emotions;

acquiring a second video of the object;

extracting a first variation pattern of the facial elements of the object contained in the second video; and

determining the emotion corresponding to the variation pattern that is the same as the first variation pattern from the plurality of stored variations patterns as the emotion of the object.

21. The emotion recognition method using facial expressions according to claim 19, wherein the plurality of emotions classified by the previously set reference is joy, surprise, sadness, anger, fear and disgust.

22. The emotion recognition method using facial expressions according to claim 19, wherein the first video and the second video are moving video data or still video data acquired by photographing the face of the object.

23. The emotion recognition method using facial expressions according to claim 19, wherein the facial elements of the object comprise at least one of eyes, eyebrows, the middle of the forehead, and mouth.

24. The emotion recognition method using facial expressions according to claim 19, wherein the control unit extracts the facial elements of the object by using an ASM (Active Shape Model) algorithm.

25. The emotion recognition method using facial expressions according to claim 24, wherein the step of extracting the facial elements of the object comprises the steps of:

extracting the features of a plurality of coordinates x and y by using the face of the object contained in the first video and the second video; and

extracting the facial elements of the object by using the extracted features of the plurality of coordinates x and y.

26. The emotion recognition method using facial expressions according to claim 24, wherein the step of receiving the plurality of first frames designated in the first video comprises the steps of:

dividing the first video into 9 sections; and

designating 10 frames inclusive of start and end frames of each of the 9 sections as the plurality of first frames.

27. The emotion recognition method using facial expressions according to claim 19, wherein the control unit extracts the variation pattern of each of the plurality of emotions by using an HMM (Hidden Markov Model) algorithm.

20. An emotion recognition method using facial expressions in a recording medium where programs of commands executed by a digital processing device are typologically set in such a manner as to be readable by means of the digital processing device, the method comprising the steps of:

acquiring a first video of an object corresponding to each of a plurality of emotions classified by a previously set reference;

receiving a plurality of first frames designated in the first video;

recognizing the face of the object contained in the plurality of first frames;

extracting the facial elements of the object by using the recognized face;

extracting the variation patterns of the plurality of emotions by using the facial elements;

storing the extracted variation patterns of the plurality of emotions;

acquiring a second video of the object;

extracting a first variation pattern of the facial elements of the object contained in the second video; and

determining the emotion corresponding to the variation pattern that is most similar to the first variation pattern from the plurality of stored variations patterns as the emotion of the object.