Facial expression recognition apparatus, image sensing apparatus, facial expression recognition method, and computer-readable storage medium

• US 9,258,482 B2
• Filed: 08/20/2015
• Issued: 02/09/2016
• Est. Priority Date: 04/09/2008
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Kaneda, Yuji
• Primary Examiner(s): HERNANDEZ, NELSON D
• Application Number:
  US14/831,131
• Publication Number:
  US 20150356348A1
• Time in Patent Office: 173 Days
• Field of Search: 348/222.1
• US Class Current: 1/1
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/175 Static expression
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body

Abstract

A facial expression recognition apparatus (10) detects a face image of a person from an input image, calculates a facial expression evaluation value corresponding to each facial expression from the detected face image, updates, based on the face image, the relationship between the calculated facial expression evaluation value and a threshold for determining a facial expression set for the facial expression evaluation value, and determines the facial expression of the face image based on the updated relationship between the facial expression evaluation value and the threshold for determining a facial expression.

Claims

1. An apparatus comprising:

an image input unit configured to input an image;

a detection unit configured to detect a face of a person from the image input by said image input unit;

a first judgment unit configured to judge whether the face detected by said detection unit shows a smiling face; and

a second judgment unit configured to judge whether the face detected by said detection unit shows an eye closed face;

wherein even if faces detected by said detection unit each have the same eye closing degree,

said second judgment unit judges that the detected face does not show the eye closed face in a case where said first judgment unit judges that the detected face shows a first smiling face, and

said second judgment unit judges that the detected face shows the eye closed face in a case where said first judgment unit judges that the detected face shows a second smiling face.

2. The apparatus according to claim 1, wherein said first judgment unit judges that a degree of the first smiling face is higher than a degree of the second smiling face.

3. The apparatus according to claim 1, further comprising

a first calculation unit configured to calculate a score of the smiling face based on feature amount of an expressionless facial image prepared in advance and feature amount of the face detected by said detection unit,

wherein said first judgment unit judges whether the face detected by said detection unit shows the smiling face based on the score calculated by said first calculation unit.

4. The apparatus according to claim 3, wherein a score of the first smiling face is higher than a score of the second smiling face.

5. The apparatus according to claim 3, wherein said first judgment unit judges that the detected face shows the first smiling face in a case where the score of the first smiling face is more than or equal to a predetermined first threshold value, and said first judgment unit judges that the detected face shows the second smiling face in a case where the score of the second smiling face is more than or equal to a predetermined second threshold value lower than the predetermined first threshold value.

6. The apparatus according to claim 1, further comprising

a second calculation unit configured to calculate a score of the smiling face based on an eye open/closed degree,

wherein said second judgment unit judges whether the face detected by said detection unit shows the eye closed face based on the score calculated by said second calculation unit.

7. The apparatus according to claim 6, further comprising

an image sensing unit configured to perform image capturing in accordance with a case where said second judgment unit judges that the face detected by said detection unit shows the eye closed face.

8. An apparatus comprising:

an image input unit configured to input an image;

a detection unit configured to detect a face of a person from the image input by said image input unit;

a judgment unit configured to judge whether a facial expression of the face detected by said detection unit is classified into a category of a plurality of facial expressions including at least a smiling face and an eye closed face,

wherein said judgment unit performs judgment so that a degree of the eye closed face is decreased in accordance with an increase in a degree of the smiling face.

9. A method comprising:

inputting an image;

detecting a face of a person from the image input in the inputting step;

judging whether the face detected in the detecting step shows a smiling face; and

judging whether the face detected in the detecting step shows an eye closed face;

wherein even if faces detected in the detecting step each have the same eye closing degree,

judging that the detected face does not show the eye closed face in a case where it is judged that the detected face shows a first smiling face, and

judging that the detected face shows the eye closed face in a case where it is judged that the detected face shows a second smiling face.

10. A non-transitory computer readable storage medium storing a program for controlling a computer to execute respective steps of a method, the method comprising:

inputting an image;

detecting a face of a person from the image input in the inputting step;

judging whether the face detected in the detecting step shows a smiling face; and

judging whether the face detected in the detecting step shows an eye closed face;

wherein even if faces detected in the detecting step each have the same eye closing degree,

judging that the detected face does not show the eye closed face in a case where it is judged that the detected face shows a first smiling face, and

judging that the detected face shows the eye closed face in a case where it is judged that the detected face shows a second smiling face.