Expression determination device, expression determination method, and recording medium

• US 10,713,477 B2
• Filed: 05/25/2018
• Issued: 07/14/2020
• Est. Priority Date: 06/19/2017
• Status: Active Grant

• Current Assignee: Casio Computer Company Limited
• Original Assignee: Casio Computer Company Limited
• Inventors: Nakagome, Kouichi, Shimada, Keisuke
• Primary Examiner(s): Perlman, David
• Application Number:
  US15/989,688
• Publication Number:
  US 20180365483A1
• Time in Patent Office: 781 Days
• Field of Search: None
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/174 Facial expression recognition

Abstract

A determination result is easily obtained even in expression determination on a face image that is not a front view. A Robot includes a camera, a face detector, a face angle estimator, and an expression determiner. The camera acquires image data. The face detector detects a face of a person from the image data acquired by the camera. The face angle estimator estimates an angle of the face detected by the face detector. The expression determiner determines an expression of the face based on the angle estimated by the face angle estimator.

Claims

1. An expression determination device comprising a processor:

wherein the processor is configured to;

acquire image data;

detect a face of a person from the acquired image data;

estimate an angle of the detected face;

acquire a single expression score indicating a degree of expression of an expression worn by the face; and

determine the expression worn by the face by calculating a single corrected score that is obtained by correcting the acquired single expression score based on the estimated angle of the face.

2. The expression determination device according to claim 1, further comprising:

an imager configured to capture an image,

wherein the processor is configured to acquire image data of the image captured by the imager, and

wherein the angle of the face estimated by the processor is an angle with respect to an optical axis direction of the imager.

3. The expression determination device according to claim 1, wherein the angle of the face that is estimated by the processor is at least one of a pitch angle, a roll angle, or a yaw angle.

4. The expression determination device according to claim 1, wherein the processor is configured to perform the determination of the expression worn by the face when an absolute value of the estimated angle of the face is equal to or less than an expression identifiable angle.

5. The expression determination device according to claim 1, wherein the processor is configured to acquire, after the processor detects the face, the image data during an image-acquisition reference time that is a time necessary for determining the expression.

6. The expression determination device according to claim 1, wherein the processor is configured to acquire, after the processor detects the face, the image data of an image-acquisition reference number of frames that is a number of frames necessary for determining the expression.

7. The expression determination device according to claim 1, wherein the processor is configured to detect a face part from the image data of the face detected by the processor and estimate the angle of the detected face by applying the detected face part to a three-dimensional shape model of a standard front view face.

8. The expression determination device according to claim 1, wherein the processor is configured to:

generate image data that is normalized so as to include eyes and a mouth in the detected face, and

determine the expression of the face that is included in the normalized image data generated.

9. The expression determination device according to claim 1, wherein the processor is configured to:

selectively perform, based on the single expression score, a first determination to determine the expression of the face without correcting the single expression score based on the estimated angle of the face and a second determination to determine the expression of the face by calculating the single corrected score obtained by correcting the single expression score based on the angle of the face.

10. The expression determination device according to claim 1, wherein the processor is configured to:

estimate a pitch angle that is an angle of the face in a vertical direction;

acquire, as the single expression score, a single smile score that is a numeric value that indicates a degree of smile of the expression worn by the face; and

determine the expression of the face by calculating, as the corrected score, a corrected smile score that is obtained by increasing the acquired single smile score in accordance with the pitch angle when the estimated pitch angle is an upward angle.

11. The expression determination device according to claim 1, wherein the processor is configured to:

acquire, as the expression score, a single smile score that is a numeric value that indicates a degree of smile of the expression worn by the face, and further acquire a single negative score that is a numeric value that indicates a degree of negativity of the expression of worn by the face;

estimate a pitch angle that is an angle of the face in a vertical direction;

when the estimated pitch angle is an upward angle, calculate, as the corrected score, a corrected smile score that is obtained by increasing the acquired smile score in accordance with the pitch angle; and

calculate a corrected negative score that is obtained by decreasing the acquired negative score in accordance with the pitch angle, and determine the expression of the face based on the calculated corrected smile score and the corrected negative score.

12. The expression determination device according to claim 1, wherein the processor is configured to:

acquire a single joy score as the expression score of the expression of the detected face, and further acquire a single surprise score, a single fear score, a single hate score, a single anger score, a single sadness score, and a single ordinary score on the expression of the detected face; and

determine the expression of the face in joy, surprise, fear, hate, anger, sadness, and ordinary based on a corrected joy score, a corrected surprise score, a corrected fear score, a corrected hate score, a corrected anger score, a corrected sadness score, and a corrected ordinary score that are obtained by correcting the acquired joy score, surprise score, fear score, hate score, anger score, sadness score, and ordinary score, respectively, based on the angle of the face.

13. The expression determination device according to claim 1, further comprising a memory;

wherein the processor is configured to;

write in the memory a result of the determination along with a determination time when an absolute value of the estimated angle of the face is equal to or less than an expression identifiable angle, and

designate, as a determination result of the expression of the face, a determination result having the determination time within effective time among determination results stored in the memory when the absolute value of the estimated angle of the face exceeds the expression identifiable angle.

14. The expression determination device according to claim 13, wherein the processor is configured to designate, as a determination result of the expression of the face, a latest determination result among determination results stored in the memory when the absolute value of the estimated angle of the face exceeds the expression identifiable angle.

15. An expression determination method, comprising:

acquiring image data;

detecting a face of a person from the acquired image data;

estimating an angle of the detected face; and

acquiring a single expression score indicating a degree of expression of an expression worn by the face, and determining the expression worn by the face by calculating a single corrected score that is obtained by correcting the acquired single expression score based on the estimated angle of the face.

16. A non-transitory computer-readable recording medium storing a program, the program causing a computer to:

acquire image data;

detect a face of a person from the acquired image data;

estimate an angle of the detected face; and

acquire a single expression score indicating a degree of expression of an expression worn by the face, and determine the expression worn by the face by calculating a single corrected score obtained by correcting the acquired single expression score based on the estimated angle of the face.