FACIAL EXPRESSION RECOGNITION APPARATUS AND METHOD, AND IMAGE CAPTURING APPARATUS

• US 20100189358A1
• Filed: 06/17/2008
• Published: 07/29/2010
• Est. Priority Date: 06/18/2007
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Matsugu, Masakazu, Kaneda, Yuji
• Granted Patent: US 8,254,691 B2
• Time in Patent Office: Days
• US Class Current: 382/195
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body

Abstract

A facial expression recognition apparatus includes an image input unit configured to sequentially input images, a face detection unit configured to detect faces in images obtained by the image input unit, and a start determination unit configured to determine whether to start facial expression determination based on facial image information detected by the face detection unit. When the start determination unit determines that facial expression determination should be started, an acquisition unit acquires reference feature information based on the facial image information detected by the face detection unit and a facial expression determination unit extracts feature information from the facial image information detected by the face detection unit and determines facial expressions of the detected faces based on the extracted feature information and the reference feature information.

Claims

1. A facial expression recognition apparatus, comprising:

an image input unit configured to sequentially input images;

a face detection unit configured to detect faces in images obtained by the image input unit;

a start determination unit configured to determine whether to start facial expression determination based on facial image information detected by the face detection unit;

an acquisition unit configured to acquire reference feature information based on the facial image information detected by the face detection unit, when the start determination unit determines that facial expression determination should be started; and

a facial expression determination unit configured to extract feature information from the facial image information detected by the face detection unit, when the start determination unit determines that the facial expression determination should be started, and to determine facial expressions of the detected faces based on the extracted feature information and the reference feature information.

2. The apparatus according to claim 1, wherein the facial expression determination unit determines facial expressions based on a difference or ratio between the extracted feature information and the reference feature information.

3. The apparatus according to claims 1, further comprising a validity determination unit configured to detect at least one of the face size, face orientation and detection reliability for the faces detected by the face detection unit and to determine the validity of the detected faces based on the detection results,

wherein the start determination unit and the facial expression determination unit carry out determination with respect to image information of faces that have been determined to be valid by the validity determination unit.

4. The apparatus according to claim 1, wherein the start determination unit determines whether to start facial expression determination based on the coordinate values of the feature points constituting facial organs in relation to a specific position on the faces.

5. The apparatus according to claim 1, wherein the start determination unit determines whether to start facial expression determination based on the distance between a specific position on the detected faces and feature points constituting facial organs.

6. The apparatus according to claim 1, wherein the start determination unit determines whether to start facial expression determination based on output values obtained by carrying out spatial filtering on the detected faces.

7. The apparatus according to claim 1, wherein the start determination unit determines that facial expression determination should be started when the variation of the features obtained from the facial image information detected in a series of images over a predetermined number of frames or a predetermined time stays within a prescribed range.

8. The apparatus according to claim 1, wherein the start determination unit holds criteria set by statistically pre-processing features obtained from a plurality of facial images and determines that facial expression determination should be started when the features obtained from the detected facial image information are within a prescribed range from the criteria.

9. The apparatus according to claim 1, wherein the start determination unit determines that facial expression determination should be started when the pixel count of a specific region in the detected facial images satisfies a prescribed condition.

10. The apparatus according to claim 9, wherein the prescribed condition is satisfied by the pixel count of the specific region when the pixel count of the pupillary region is not less than a prescribed value.

11. The apparatus according to claim 9, wherein the prescribed condition is satisfied by the pixel count of the specific region when the pixel count of the tooth-colored region does not exceed a prescribed value.

12. The apparatus according to claim 1, wherein the reference feature information is updated using reference feature information acquired by the acquisition unit after a start determination is made again when the start determination unit determines again that facial expression determination should be started, until the facial expression determination unit determines the facial expression on the face.

13. The apparatus according to claim 1, wherein the facial expression determination unit weights the variation of the features contained in the reference feature information and the features contained in the extracted feature information in order to determine the facial expressions of the detected faces based on the weighted variations.

14. The apparatus according to claim 1, further comprising a personal identity authentication unit configured to carry out personal identity authentication based on the facial images detected by the face detection unit.

15. An image capturing apparatus comprising:

the facial expression recognition apparatus according to claim 1;

a supplying unit configured to acquire images in chronological order using an image capturing unit and to supply the acquired images to the image input unit; and

a recording unit configured to record the images as captured images when the facial expression determination unit determines that the facial expression is a prescribed facial expression.

16. A facial expression recognition apparatus comprising:

an image input unit configured to sequentially input images;

a face detection unit configured to detect faces in the images obtained by the image input unit;

a feature extraction unit configured to extract features from faces detected by the face detection unit;

a storage unit configured to store the features in a memory, when the reliability of the features extracted by the feature extraction unit is not less than a threshold value; and

a facial expression determination unit configured to determine the facial expressions of the faces based on the variation of the features stored in the memory by the storage unit.

17. A facial expression recognition apparatus comprising:

a detection unit configured to detect a face from sequentially input images;

a state determination unit configured to determine a state of a feature of the face detected by the detection unit;

an acquisition unit configured to acquire reference feature information based on image information of the face when the state determination unit determines that the feature of the face is in a predetermined state; and

a facial expression determination unit configured to determine a facial expression of a face newly detected by the detection unit, based on the newly detected face and the reference feature information.

18. A facial expression recognition method, comprising the steps of:

image input, which involves sequentially inputting images;

face detection, which involves detecting faces in images obtained in the image input step;

start determination, which involves determining whether to start facial expression determination based on facial image information detected in the face detection step;

acquisition, which involves acquiring reference feature information based on the facial image information detected in the face detection step when the start determination step determines that facial expression determination should be started; and

facial expression determination, which involves extracting feature information from the facial image information detected in the face detection step, when it is determined to start the facial expression determination in the start determination step, and determining facial expressions of the detected faces based on the extracted feature information and the reference feature information.

22. A computer-readable storage medium storing a computer program for causing a computer to execute the facial expression recognition method according to claim 18.

19. A facial expression recognition method based on an information processor comprising the steps of:

image input, which involves sequentially inputting images;

face detection, which involves detecting faces in the images obtained in the image input step;

feature extraction, which involves extracting features from faces detected in the face detection step;

storage, which involves storing the features in a memory when the reliability of the features extracted in the feature extraction step is not less than a threshold value; and

facial expression determination, which involves determining the facial expressions of the faces based on the variation of the features stored in the memory in the storage step.

23. A computer-readable storage medium storing a computer program for causing a computer to execute the facial expression recognition method according to claim 19.

20. A facial expression recognition method comprising the steps of:

detecting a face from sequentially input images;

determining a state of a feature of the face detected in the detection step;

acquiring reference feature information based on image information of the face, when it is determined, in the determining step, that the feature of the face is in a predetermined state; and

facial expression determination, which involves determining a facial expression of a face newly detected in the detecting step, based on the newly detected face and the reference feature information.

24. A computer-readable storage medium storing a computer program for causing a computer to execute the facial expression recognition method according to claim 20.

21. (canceled)