FACIAL EXPRESSION RECOGNITION APPARATUS, IMAGE SENSING APPARATUS, FACIAL EXPRESSION RECOGNITION METHOD, AND COMPUTER-READABLE STORAGE MEDIUM

• US 20110032378A1
• Filed: 03/31/2009
• Published: 02/10/2011
• Est. Priority Date: 04/09/2008
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Kaneda, Yuji
• Granted Patent: US 8,780,221 B2
• Time in Patent Office: Days
• US Class Current: 348/222.1
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/175 Static expression
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body

Abstract

A facial expression recognition apparatus (10) detects a face image of a person from an input image, calculates a facial expression evaluation value corresponding to each facial expression from the detected face image, updates, based on the face image, the relationship between the calculated facial expression evaluation value and a threshold for determining a facial expression set for the facial expression evaluation value, and determines the facial expression of the face image based on the updated relationship between the facial expression evaluation value and the threshold for determining a facial expression.

Claims

1. A facial expression recognition apparatus by comprising:

an image input unit configured to input an image;

a face detection unit configured to detect a face image of a person from an image input by said image input unit;

a calculation unit configured to calculate a facial expression evaluation value corresponding to each facial expression from a face image detected by said face detection unit;

an updating unit configured to update, based on the face image, a relationship between a facial expression evaluation value calculated by said calculation unit and a threshold for determining a facial expression set for the facial expression evaluation value; and

a determination unit configured to determine a facial expression of the face image based on the relationship between the facial expression evaluation value and the threshold for determining a facial expression which is updated by said updating unit.

2. The apparatus according to claim 1, wherein said updating unit updates the relationship based on the facial expression evaluation value calculated by said calculation unit.

3. The apparatus according to claim 2, wherein said updating unit performs, based on the facial expression evaluation value calculated by said calculation unit, the updating operation by weighting the facial expression evaluation value.

4. The apparatus according to claim 2, wherein said updating unit performs, based on the facial expression evaluation value calculated by said calculation unit, the updating operation by weighting the threshold for determining the facial expression.

5. The apparatus according to claim 1, wherein said updating unit updates the relationship based on a facial expression detecting history.

6. The apparatus according to claim 1, wherein said updating unit updates the relationship based on an age estimated from a face image detected by said face detection unit.

7. The apparatus according to claim 1, wherein said determination unit updates the relationship based on a sex estimated from the face image detected by said face detection unit.

8. The apparatus according to claim 1, wherein

each facial expression is classified into a category, and

said updating unit updates, for each category into which each facial expression is classified, a relationship between the facial expression evaluation value of each facial expression belonging to the category and the threshold for determining a facial expression.

9. An image sensing apparatus by comprising:

an image input unit configured to input an image;

a face detection unit configured to detect a face image of a person from an image input by said image input unit;

a calculation unit configured to calculate a facial expression evaluation value corresponding to each facial expression from a face image detected by said face detection unit;

an updating unit configured to update, based on the face image, a relationship between a facial expression evaluation value calculated by said calculation unit and a threshold for determining a facial expression set for the facial expression evaluation value;

a determination unit configured to determine a facial expression of the face image based on the relationship between the facial expression evaluation value and the threshold for determining a facial expression which is updated by said updating unit; and

an image sensing unit configured to perform image capturing based on the facial expression of the face image determined by said determination unit.

10. An image sensing apparatus by comprising:

an image input unit configured to input an image;

a face detection unit configured to detect a face image of a person from an image input by said image input unit;

a calculation unit configured to calculate a facial expression evaluation value corresponding to each facial expression from a face image detected by said face detection unit; and

an image sensing unit configured to perform image capturing based on a relationship between facial expression evaluation values respectively corresponding to a first facial expression and a second facial expression, which are calculated by said calculation unit, and a threshold for determining a facial expression set for said each facial expression evaluation value.

11. The apparatus according to claim 10, wherein said image sensing unit performs image capturing if the facial expression evaluation value of the first facial expression is not less than the threshold for determining a facial expression set for the facial expression evaluation value and the facial expression evaluation value of the second facial expression is less than the threshold for determining a facial expression set for the facial expression evaluation value.

12. A facial expression recognition method in a facial expression recognition apparatus, by comprising:

an image input step of inputting an image;

a face detection step of detecting a face image of a person from an image input in the image input step;

a calculation step of calculating a facial expression evaluation value corresponding to each facial expression from a face image detected in the face detection step;

an updating step of updating, based on the face image, a relationship between a facial expression evaluation value calculated in the calculation step and a threshold for determining a facial expression set for the facial expression evaluation value; and

a determination step of determining a facial expression of the face image based on the relationship between the facial expression evaluation value and the threshold for determining a facial expression which is updated in the updating step.

13. A computer-readable storage medium storing a facial expression recognition program for causing a computer to function as

a face detection unit configured to detect a face image of a person from an image,

a calculation unit configured to calculate a facial expression evaluation value corresponding to each facial expression from a face image detected by the face detection unit,

an updating unit configured to update, based on the face image, a relationship between a facial expression evaluation value calculated by the calculation unit and a threshold for determining a facial expression set for the facial expression evaluation value, and

a determination unit configured to determine a facial expression of the face image based on the relationship between the facial expression evaluation value and the threshold for determining a facial expression which is updated by the updating unit.