Weak hypothesis generation apparatus and method, learning apparatus and method, detection apparatus and method, facial expression learning apparatus and method, facial expression recognition apparatus and method, and robot apparatus

• US 7,587,069 B2
• Filed: 03/07/2008
• Issued: 09/08/2009
• Est. Priority Date: 07/24/2003
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.), University of California San Diego (University of California)
• Original Assignee: Sony Corporation (Sony Group Corp.), University of California San Diego (University of California)
• Inventors: Carlson, Eric C., Hershey, John, Movellan, Javier R., Fasel, Ian R., Bartlett, Marian S., Littlewort, Gwendolen C., Susskind, Josh, Sabe, Kohtaro, Kawamoto, Kenta, Hidai, Kenichi
• Primary Examiner(s): Chawan; Sheela C
• Application Number:
  US12/075,080
• Publication Number:
  US 20080247598A1
• Time in Patent Office: 550 Days
• Field of Search: 382/100, 382/103, 382/118, 382/155, 382/159, 382/181, 382/224, 706/12, 706/17, 706/25, 706/59, 375/150, 375/343, 702/181, 707/1
• US Class Current: 382/118
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06N 3/004 Artificial life, i.e. computing arrangements simulating life
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/175 Static expression

Abstract

A facial expression recognition system that uses a face detection apparatus realizing efficient learning and high-speed detection processing based on ensemble learning when detecting an area representing a detection target and that is robust against shifts of face position included in images and capable of highly accurate expression recognition, and a learning method for the system, are provided. When learning data to be used by the face detection apparatus by Adaboost, processing to select high-performance weak hypotheses from all weak hypotheses, then generate new weak hypotheses from these high-performance weak hypotheses on the basis of statistical characteristics, and select one weak hypothesis having the highest discrimination performance from these weak hypotheses, is repeated to sequentially generate a weak hypothesis, and a final hypothesis is thus acquired. In detection, using an abort threshold value that has been learned in advance, whether provided data can be obviously judged as a non-face is determined every time one weak hypothesis outputs the result of discrimination. If it can be judged so, processing is aborted. A predetermined Gabor filter is selected from the detected face image by an Adaboost technique, and a support vector for only a feature quantity extracted by the selected filter is learned, thus performing expression recognition.

Claims

1. A facial expression learning apparatus for learning data to be used by a facial expression recognition apparatus, the facial expression recognition apparatus being adapted for recognizing an expression of a provided face image by using an expression learning data set including plural face images representing specific expressions as recognition targets and plural face images representing expressions different from the specific expressions,

the facial expression learning apparatus comprising an expression learning unit for learning data to be used by the facial expression recognition apparatus, the facial expression recognition apparatus identifying the face images representing the specific expressions from provided face images on the basis of a face feature extracted from the expression learning data set by using a Gabor filter.

2. The facial expression learning apparatus as claimed in claim 1, wherein the expression learning unit learns a support vector for identifying a face image representing the specific expression on the basis of the face feature extracted from the expression learning data set by using the Gabor filter.

3. The facial expression learning apparatus as claimed in claim 1, wherein the expression learning unit has:

a weak hypothesis generation unit for repeating processing to generate a weak hypothesis for estimating whether a provided face image is of the specific expression or not on the basis of the result of filtering by one Gabor filter selected from plural Gabor filters;

a reliability calculation unit for calculating reliability of the weak hypothesis generated by the weak hypothesis generation unit on the basis of estimation performance of the weak hypothesis with respect to the expression learning data set; and

a data weighting update unit for updating data weighting set for the expression learning data set on the basis of the reliability, and

wherein the weak hypothesis generation unit repeats processing to generate the weak hypothesis while selecting one Gabor filter having the highest estimation performance with respect to the expression learning data set every time the data weighting is updated.

4. The facial expression learning apparatus as claimed in claim 1, wherein the expression learning unit has:

a weak hypothesis generation unit for repeating processing to generate a weak hypothesis for estimating whether a provided face image is of the specific expression or not on the basis of the result of filtering by one Gabor filter selected from plural Gabor filters;

a reliability calculation unit for calculating reliability of the weak hypothesis generated by the weak hypothesis generation unit on the basis of estimation performance of the weak hypothesis with respect to the expression learning data set;

a data weighting update unit for updating data weighting set for the expression learning data set on the basis of the reliability; and

a support vector learning unit for learning a support vector for identifying a face image representing the specific expression on the basis of the face feature extracted form the expression learning data set by a predetermined Gabor filter, and

wherein the weak hypothesis generation unit repeats processing to generate the weak hypothesis while selecting one Gabor filter having the highest estimation performance with respect to the expression learning data set every time the data weighting is updated, and

the support vector learning unit extracts the face feature by using the Gabor filter selected by the weak hypothesis generated by the weak hypothesis generation unit, and thus learns the support vector.

5. A facial expression learning method for learning data to be used by a facial expression recognition apparatus, the facial expression recognition apparatus being adapted for recognizing an expression of a provided face image by using an expression learning data set including plural face images representing specific expressions as recognition targets and plural face images representing expressions different from the specific expressions,

the facial expression learning method, performed by a processor, comprising an expression learning step of learning data to be used by the facial expression recognition apparatus, the facial expression recognition apparatus identifying the face images representing the specific expressions from provided face images on the basis of a face feature extracted from the expression learning data set by using a Gabor filter.

6. The facial expression learning method, performed by a processor, as claimed in claim 5, wherein the expression learning step includes learning a support vector for identifying a face image representing the specific expression on the basis of the face feature extracted from the expression learning data set by using the Gabor filter.

7. The facial expression learning method, performed by a processor, as claimed in claim 5, wherein the expression learning step includes:

a weak hypothesis generation step of repeating processing to generate a weak hypothesis for estimating whether a provided face image is of the specific expression or not on the basis of the result of filtering by one Gabor filter selected from plural Gabor filters;

a reliability calculation step of calculating reliability of the weak hypothesis generated at the weak hypothesis generation step on the basis of estimation performance of the weak hypothesis with respect to the expression learning data set; and

a data weighting update step of updating data weighting set for the expression learning data set on the basis of the reliability, and

wherein the weak hypothesis generation step includes generating the weak hypothesis while selecting one Gabor filter having the highest estimation performance with respect to the expression learning data set every time the data weighting is updated.

8. The facial expression learning method, performed by a processor, as claimed in claim 5 wherein the expression learning step includes:

a weak hypothesis generation step of repeating processing to generate a weak hypothesis for estimating whether a provided face image is of the specific expression or not on the basis of the result of filtering by one Gabor filter selected from plural Gabor filters;

a reliability calculation step of calculating reliability of the weak hypothesis generated at the weak hypothesis generation step on the basis of estimation performance of the weak hypothesis with respect to the expression learning data set;

a data weighting update step of updating data weighting set for the expression learning data set on the basis of the reliability; and

a support vector learning step of learning a support vector for identifying a face image representing the specific expression on the basis of the face feature extracted form the expression learning data set by a predetermined Gabor filter, and

wherein the weak hypothesis generation step includes generating the weak hypothesis while selecting one Gabor filter having the highest estimation performance with respect to the expression learning data set every time the data weighting is updated, and

the support vector learning step includes extracting the face feature by using the Gabor filter selected by the weak hypothesis generated by the weak hypothesis generation unit, and thus learning the support vector.