Face pose estimation device, face pose estimation method and face pose estimation program

• US 8,401,251 B2
• Filed: 01/16/2009
• Issued: 03/19/2013
• Est. Priority Date: 01/16/2008
• Status: Active Grant

• Current Assignee: Asahi Kasei Kabushiki Kaisha
• Original Assignee: Asahi Kasei Kabushiki Kaisha
• Inventors: Sasahara, Hideaki, Noguchi, Yoshihiro, Shimada, Keiji
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US12/863,132
• Publication Number:
  US 20110052013A1
• Time in Patent Office: 1,523 Days
• Field of Search: 382/103, 382/118, 382/159, 382/170, 382/181, 382/190, 382/216, 345/419, 345/423, 345/426, 345/427
• US Class Current: 382/118
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30268 Vehicle interior
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/149 involving deformable models, e.g. active contour models
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships

Abstract

An object is to provide a face pose estimating device, a face pose estimating method, and a face pose estimating program that allow high accuracy estimation of a face pose even if a face organ cannot be detected from an image. A face organ detector 11 of the face pose estimating device 10 detects a face organ from an image frame. A face pose candidate set generator 16 generates a face pose candidate set. A first similarity estimator 12 computes 3D-model-based similarities. A second similarity estimator 13 computes appearance-based similarities. A first likelihood estimator 141 computes a first likelihood corresponding to the 3D-model-based similarity. A second likelihood estimator 142 computes a second likelihood corresponding to the appearance-based similarities. An integrated likelihood estimator 143 computes an integrated likelihood by using the first and second likelihoods. A face pose estimator 15 estimates the face pose by using the integrated likelihood.

Claims

1. A face pose estimation device estimating a face pose representing at least an orientation of a face from a face image in which the face is captured in a time series manner, the device comprising:

a face organ detector that detects a face organ from the face image;

a face pose candidate set generator that generates a face pose candidate set, which is a set of face pose candidates to be estimated;

a first similarity estimator that computes a first similarity according to a first parameter corresponding to respective positions of each face organ of each element of the face pose candidate set generated by the face pose candidate set generator and an actual face organ detected by the face organ detector;

a second similarity estimator that computes a second similarity according to a second parameter corresponding to a pixel value according to displacements of each face image of the face pose candidate set generated by the face pose candidate set generator and an actual face image detected as a detection target by the face organ detector with respect to each predetermined reference pose;

a first likelihood estimator that computes a first likelihood corresponding to the first similarity computed by the first similarity estimator;

a second likelihood estimator that computes a second likelihood corresponding to the second similarity computed by the second similarity estimator;

an integrated likelihood estimator that computes an integrated likelihood representing a degree of appropriateness of each element of the face pose candidate set by using the first and second likelihoods; and

a face pose estimator that estimates the face pose on the basis of the face pose candidate having the highest integrated likelihood computed by the integrated likelihood estimator,

the integrated likelihood being considered by the face pose candidate set generator for generating a face pose candidate set in the next time step.

2. The face pose estimation device according to claim 1, wherein the integrated likelihood estimator computes the integrated likelihood by using only either the first likelihood or the second likelihood in the case that the other likelihood is not available.

3. The face pose estimation device according to claim 1, wherein the first and the second similarity estimators have a complementary relationship complementing respective performance characteristics of each other.

4. The face pose estimation device according to claim 1, wherein the integrated likelihood estimator multiplies the first and second likelihoods by each other.

5. The face pose estimation device according to claim 1, wherein the first similarity estimator adopts a three-dimensional face shape model based similarity, the first parameter is an error in position, the second similarity estimator adopts a projective transformation appearance based similarity, and the second parameter is an error in pixel value.

6. The face pose estimation device according to claim 1, wherein the face pose estimator estimates the face pose on the basis of any one of an average value, a median value and a barycenter, which is an average value weighted by the integrated likelihood, of numerical values based on at least a part of the face pose candidate set generated by the face pose candidate set generator, instead of estimating the face pose on the basis of the face pose candidate having the highest integrated likelihood estimated by the integrated likelihood estimator.

7. The face pose estimating device according to claim 1, wherein the face pose estimator estimates the face pose on the basis of the face pose candidate corresponding to the highest integrated likelihood in the case that the highest integrated likelihood computed by the integrated likelihood estimator exceeds a predetermined threshold, and,

the face pose estimator estimates the face pose on the basis of any one of an average value, a median value and a barycenter, which is an average value weighted by the integrated likelihood, of numerical values based on at least a part of the face pose candidate set generated by the face pose candidate set generator in the case that the highest integrated likelihood is equal to or less than the predetermined threshold.

8. The face pose estimation device according to claim 1, wherein the face pose candidate set generator generates the face pose candidate set used for computing the first and second similarities in the next time step on the basis of the face pose candidate set corresponding to a relatively higher integrated likelihood computed by the integrated likelihood estimator.

9. The face pose estimation device according to claim 1, wherein the face image is an image in which a driver'"'"'s face of a vehicle has been captured.

10. A vehicle comprising the face pose estimation device according to claim 1.

11. A face pose estimation method estimating a face pose representing at least an orientation of a face from a face image in which the face is captured in a time series manner, the method comprising:

a face organ detecting step for detecting a face organ from the face image;

a face pose candidate set generating step of generating a face pose candidate set, which is a set of face pose candidates to be estimated;

a first similarity computing step of computing a first similarity according to a first parameter corresponding to respective positions of each face organ of each element of the face pose candidate set generated in the face pose candidate set generating step and an actual face organ detected by the face organ detecting step;

a second similarity computing step of computing a second similarity according to a second parameter corresponding to a pixel value according to displacements of each face image of the face pose candidate set generated in the face pose candidate set generating step and an actual face image detected as a detection target in the face organ detecting step with respect to each predetermined reference pose;

a first likelihood computing step of computing a first likelihood corresponding to the first similarity computed in the first similarity computing step;

a second likelihood computing step of computing a second likelihood corresponding to the second similarity computed in the second similarity computing step;

an integrated likelihood computing step of computing an integrated likelihood representing a degree of appropriateness of each element of the face pose candidate set on the basis of the face pose candidate having the highest first and second likelihoods; and

a face pose estimating step of estimating the face pose by using the integrated likelihood computed in the integrated likelihood computing step.

12. A non-transitory computer readable storage medium that stores a face pose estimation program, the program when executed causing a computer to perform:

a face organ detecting step for detecting face organ from a face image in which the face is captured in a time series manner;

a face pose candidate set generating step of generating a face pose candidate set, which is a set of face pose candidates to be estimated;

a first similarity computing step of computing a first similarity according to a first parameter corresponding to respective positions of each face organ of each element of the face pose candidate set generated in the face pose candidate set generating step and an actual face organ detected by the face organ detecting step;

a second similarity computing step of computing a second similarity according to a second parameter corresponding to a pixel value according to displacements of each face image of the face pose candidate set generated in the face pose candidate set generating step and an actual face image detected as a detection target in the face organ detecting step with respect to each predetermined reference pose;

a first likelihood computing step of computing a first likelihood corresponding to the first similarity computed in the first similarity computing step;

a second likelihood computing step of computing a second likelihood corresponding to the second similarity computed in the second similarity computing step;

an integrated likelihood computing step of computing an integrated likelihood representing a degree of appropriateness of each element of the face pose candidate set by using the first and second likelihoods; and

a face pose estimating step of estimating the face pose on the basis of the face pose candidate having the highest integrated likelihood computed in the integrated likelihood computing step.