Facial expression recognition apparatus, image sensing apparatus, facial expression recognition method, and computer-readable storage medium

• US 9,147,107 B2
• Filed: 05/27/2014
• Issued: 09/29/2015
• Est. Priority Date: 04/09/2008
• Status: Expired due to Fees

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Kaneda, Yuji
• Primary Examiner(s): HERNANDEZ, NELSON D
• Application Number:
  US14/287,441
• Publication Number:
  US 20140270376A1
• Time in Patent Office: 490 Days
• Field of Search: 348345-357, 348/222.1
• US Class Current: 1/1
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/175 Static expression
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body

Abstract

A facial expression recognition apparatus (10) detects a face image of a person from an input image, calculates a facial expression evaluation value corresponding to each facial expression from the detected face image, updates, based on the face image, the relationship between the calculated facial expression evaluation value and a threshold for determining a facial expression set for the facial expression evaluation value, and determines the facial expression of the face image based on the updated relationship between the facial expression evaluation value and the threshold for determining a facial expression.

Claims

1. An apparatus comprising:

an image input unit configured to input an image;

a face detection unit configured to detect a face image of a person from the image input by said image input unit;

a first calculation unit configured to calculate a first expression evaluation value corresponding to a first facial expression for the detected face based on the face image detected by said face detection unit;

a second calculation unit configured to calculate a second expression evaluation value corresponding to a second facial expression for the detected face based on the face image detected by said face detection unit, wherein the second facial expression is different from the first facial expression;

an updating unit configured to update, based on the first expression value, a relationship between the second expression value and a threshold value for determining a facial expression of a person; and

a judgment unit configured to judge whether the detected face shows the second facial expression or not based on the relationship between the second expression value and the threshold value which is updated by said updating unit.

2. The apparatus according to claim 1, wherein said updating unit updates the relationship between the second expression value and a threshold value, in a case that the first expression value is equal to or greater than a predetermined value.

3. The apparatus according to claim 1, wherein said updating unit updates the second expression evaluation value by weighting the second expression evaluation value.

4. The apparatus according to claim 1, wherein said updating unit updates the threshold by weighting the threshold for determining the facial expression.

5. The apparatus according to claim 1, wherein

each facial expression is classified into a category, and

said updating unit updates, for each category into which each facial expression is classified, a relationship between an expression evaluation value corresponding to each facial expression belonging to the category and each threshold corresponding to each expression evaluation value.

6. The apparatus according to claim 1, further comprising:

an image sensing unit configured to perform image capturing, based on the facial expression of the face image determined by said face detection unit.

7. The apparatus according to claim 6, wherein said image sensing unit performs image capturing in a case where the judgment unit does not judge that the detected face shows the second facial expression.

8. The apparatus according to claim 1, wherein the second calculation unit further calculates eye open/closed degree evaluation values.

9. The apparatus according to claim 1, wherein the first facial expression is similar to the second facial expression.

10. The apparatus according to claim 1, wherein the first facial expression includes a smiling facial expression, and the second facial expression includes an eye-closed facial expression.

17. A non-transitory computer-readable storage medium storing a program for causing a computer to function as respective units of an apparatus according to claim 1.

11. An apparatus comprising:

an image input unit configured to input an image;

a face detection unit configured to detect a face image of a person from the image input by said image input unit;

a first calculation unit configured to calculate a first expression evaluation value corresponding to a first facial expression for the detected face based on the face image detected by said face detection unit;

a second calculation unit configured to calculate a second expression evaluation value corresponding to a second facial expression for the detected face based on the face image detected by said face detection unit, wherein the second facial expression is different from the first facial expression;

a facial expression history obtaining unit configured to obtain a facial expression history;

an updating unit configured to update a relationship between the second expression value and a threshold value for determining a facial expression of a person, wherein said updating unit updates the relationship between the second expression value and the threshold value based on the obtained facial expression history; and

a judgment unit configured to judge whether the detected face shows the second facial expression or not based on the relationship between the second expression value and the threshold value which is updated by said updating unit.

12. An apparatus comprising:

an image input unit configured to input an image;

a face detection unit configured to detect a face image of a person from the image input by said image input unit;

a first calculation unit configured to calculate a first expression evaluation value corresponding to a first facial expression for the detected face based on the face image detected by said face detection unit;

a second calculation unit configured to calculate a second expression evaluation value corresponding to a second facial expression for the detected face based on the face image detected by said face detection unit, wherein the second facial expression is different from the first facial expression;

an age estimation unit configured to estimate an age of a person whose face is detected by said face detection unit;

an updating unit configured to update a relationship between the second expression value and a threshold value for determining a facial expression of a person, wherein said updating unit updates the relationship between the second expression value and the threshold value based on the age estimated by said age estimation unit; and

a judgment unit configured to judge whether the detected face shows the second facial expression or not based on the relationship between the second expression value and the threshold value which is updated by said updating unit.

13. An apparatus comprising:

an image input unit configured to input an image;

a face detection unit configured to detect a face image of a person from the image input by said image input unit;

a first calculation unit configured to calculate a first expression evaluation value corresponding to a first facial expression for the detected face based on the face image detected by said face detection unit;

a second calculation unit configured to calculate a second expression evaluation value corresponding to a second facial expression for the detected face based on the face image detected by said face detection unit, wherein the second facial expression is different from the first facial expression;

a sex estimation unit configured to estimate a sex of a person whose face is detected by said face detection unit;

an updating unit configured to update a relationship between the second expression value and a threshold value for determining a facial expression of a person, wherein said updating unit updates the relationship between the second expression value and the threshold value based on the sex estimated by said sex estimation unit; and

a judgment unit configured to judge whether the detected face shows the second facial expression or not based on the relationship between the second expression value and the threshold value which is updated by said updating unit.

14. A method comprising:

an image input step of inputting an image;

a face detection step of detecting a face image of a person from the image input in the image input step;

a first calculation step of calculating a first expression evaluation value corresponding to a first facial expression for the detected face based on the face image detected in the face detection step;

a second calculation step of calculating a second expression value corresponding to a second facial expression for the detected face based on the face image detected in the face detection step, wherein the second facial expression is different from the first facial expression;

an updating step of updating, based on the first expression value, a relationship between the second expression value and a threshold value for determining a facial expression of a person; and

a judgment step of judging whether the detected face shows the second facial expression or not based on the relationship between the second expression value and the threshold value which is updated in the updating step.

15. The method according to claim 14, wherein the first facial expression is similar to the second facial expression.

16. The method according to claim 14, wherein the first facial expression includes a smiling facial expression, and the second facial expression includes an eye-closed facial expression.