Weak hypothesis generation apparatus and method, learning apparatus and method, detection apparatus and method, facial expression learning apparatus and method, facial expression recognition apparatus and method, and robot apparatus

• US 20080247598A1
• Filed: 03/07/2008
• Published: 10/09/2008
• Est. Priority Date: 07/24/2003
• Status: Active Grant

• Current Assignee: Eric C. Carlson, Gwendolen C. Littlewort, Ian R. Fasel, Javier R. Movellan, John Hershey, Josh Susskind, Kenichi Hidai, Kenta Kawamoto, Kohtaro Sabe, Marian S. Bartlett
• Original Assignee: Eric C. Carlson, Gwendolen C. Littlewort, Ian R. Fasel, Javier R. Movellan, John Hershey, Josh Susskind, Kenichi Hidai, Kenta Kawamoto, Kohtaro Sabe, Marian S. Bartlett
• Inventors: Sabe, Kohtaro, Kawamoto, Kenta, Carlson, Eric C., Hidai, Kenichi, Hershey, John, Movellan, Javier R., Fasel, Ian R., Bartlett, Marian S., Littlewort, Gwendolen C., Susskind, Josh
• Granted Patent: US 7,587,069 B2
• Time in Patent Office: Days
• US Class Current: 382/100
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06N 3/004 Artificial life, i.e. computing arrangements simulating life
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/175 Static expression

Abstract

A facial expression recognition system that uses a face detection apparatus realizing efficient learning and high-speed detection processing based on ensemble learning when detecting an area representing a detection target and that is robust against shifts of face position included in images and capable of highly accurate expression recognition, and a learning method for the system, are provided. When learning data to be used by the face detection apparatus by Adaboost, processing to select high-performance weak hypotheses from all weak hypotheses, then generate new weak hypotheses from these high-performance weak hypotheses on the basis of statistical characteristics, and select one weak hypothesis having the highest discrimination performance from these weak hypotheses, is repeated to sequentially generate a weak hypothesis, and a final hypothesis is thus acquired. In detection, using an abort threshold value that has been learned in advance, whether provided data can be obviously judged as a non-face is determined every time one weak hypothesis outputs the result of discrimination. If it can be judged so, processing is aborted. A predetermined Gabor filter is selected from the detected face image by an Adaboost technique, and a support vector for only a feature quantity extracted by the selected filter is learned, thus performing expression recognition.

Claims

1-31. -31. (canceled)

32. A facial expression learning apparatus for learning data to be used by a facial expression recognition apparatus, the facial expression recognition apparatus being adapted for recognizing an expression of a provided face image by using an expression learning data set including plural face images representing specific expressions as recognition targets and plural face images representing expressions different from the specific expressions,

the facial expression learning apparatus comprising an expression learning unit for learning data to be used by the facial expression recognition apparatus, the facial expression recognition apparatus identifying the face images representing the specific expressions from provided face images on the basis of a face feature extracted from the expression learning data set by using a Gabor filter.

33. The facial expression learning apparatus as claimed in claim 32, wherein the expression learning unit learns a support vector for identifying a face image representing the specific expression on the basis of the face feature extracted from the expression learning data set by using the Gabor filter.

34. The facial expression learning apparatus as claimed in claim 32, wherein the expression learning unit has:

a weak hypothesis generation unit for repeating processing to generate a weak hypothesis for estimating whether a provided face image is of the specific expression or not on the basis of the result of filtering by one Gabor filter selected from plural Gabor filters;

a reliability calculation unit for calculating reliability of the weak hypothesis generated by the weak hypothesis generation unit on the basis of estimation performance of the weak hypothesis with respect to the expression learning data set; and

a data weighting update unit for updating data weighting set for the expression learning data set on the basis of the reliability, and

wherein the weak hypothesis generation unit repeats processing to generate the weak hypothesis while selecting one Gabor filter having the highest estimation performance with respect to the expression learning data set every time the data weighting is updated.

35. The facial expression learning apparatus as claimed in claim 32, wherein the expression learning unit has:

a weak hypothesis generation unit for repeating processing to generate a weak hypothesis for estimating whether a provided face image is of the specific expression or not on the basis of the result of filtering by one Gabor filter selected from plural Gabor filters;

a reliability calculation unit for calculating reliability of the weak hypothesis generated by the weak hypothesis generation unit on the basis of estimation performance of the weak hypothesis with respect to the expression learning data set;

a data weighting update unit for updating data weighting set for the expression learning data set on the basis of the reliability; and

a support vector learning unit for learning a support vector for identifying a face image representing the specific expression on the basis of the face feature extracted form the expression learning data set by a predetermined Gabor filter, and

wherein the weak hypothesis generation unit repeats processing to generate the weak hypothesis while selecting one Gabor filter having the highest estimation performance with respect to the expression learning data set every time the data weighting is updated, and

the support vector learning unit extracts the face feature by using the Gabor filter selected by the weak hypothesis generated by the weak hypothesis generation unit, and thus learns the support vector.

36. A facial expression learning method for learning data to be used by a facial expression recognition apparatus, the facial expression recognition apparatus being adapted for recognizing an expression of a provided face image by using an expression learning data set including plural face images representing specific expressions as recognition targets and plural face images representing expressions different from the specific expressions,

the facial expression learning method comprising an expression learning step of learning data to be used by the facial expression recognition apparatus, the facial expression recognition apparatus identifying the face images representing the specific expressions from provided face images on the basis of a face feature extracted from the expression learning data set by using a Gabor filter.

37. The facial expression learning method as claimed in claim 36, wherein the expression learning step includes learning a support vector for identifying a face image representing the specific expression on the basis of the face feature extracted from the expression learning data set by using the Gabor filter.

38. The facial expression learning method as claimed in claim 36, wherein the expression learning step includes:

a weak hypothesis generation step of repeating processing to generate a weak hypothesis for estimating whether a provided face image is of the specific expression or not on the basis of the result of filtering by one Gabor filter selected from plural Gabor filters;

a reliability calculation step of calculating reliability of the weak hypothesis generated at the weak hypothesis generation step on the basis of estimation performance of the weak hypothesis with respect to the expression learning data set; and

a data weighting update step of updating data weighting set for the expression learning data set on the basis of the reliability, and

wherein the weak hypothesis generation step includes generating the weak hypothesis while selecting one Gabor filter having the highest estimation performance with respect to the expression learning data set every time the data weighting is updated.

39. The facial expression learning method as claimed in claim 36, wherein the expression learning step includes:

a weak hypothesis generation step of repeating processing to generate a weak hypothesis for estimating whether a provided face image is of the specific expression or not on the basis of the result of filtering by one Gabor filter selected from plural Gabor filters;

a reliability calculation step of calculating reliability of the weak hypothesis generated at the weak hypothesis generation step on the basis of estimation performance of the weak hypothesis with respect to the expression learning data set;

a data weighting update step of updating data weighting set for the expression learning data set on the basis of the reliability; and

a support vector learning step of learning a support vector for identifying a face image representing the specific expression on the basis of the face feature extracted form the expression learning data set by a predetermined Gabor filter, and

wherein the weak hypothesis generation step includes generating the weak hypothesis while selecting one Gabor filter having the highest estimation performance with respect to the expression learning data set every time the data weighting is updated, and

the support vector learning step includes extracting the face feature by using the Gabor filter selected by the weak hypothesis generated by the weak hypothesis generation unit, and thus learning the support vector.

40. A facial expression recognition apparatus comprising:

a face feature extraction unit for filtering a provided face image by using a Gabor filter and thus extracting a face feature; and

an expression recognition unit for recognizing an expression of the provided face image on the basis of the face feature.

41. The facial expression recognition apparatus as claimed in claim 40, wherein the expression recognition unit has an expression identifier for identifying a face image showing a specific expression as a recognition target from the provided face image, and

the expression identifier is a support vector machine (SVM) for identifying whether the face feature extracted by the face feature extraction unit is the specific expression or not, using a support vector that has been learned in advance.

42. The facial expression recognition apparatus as claimed in claim 40, wherein the expression recognition unit has an expression identifier for identifying a face image showing a specific expression as a recognition target from the provided face image, and the expression identifier comprises:

an estimation result output unit including plural weak hypotheses; and

a discrimination unit for discriminating whether the provided face image shows the specific expression on the basis of the result of estimation outputted from the estimation result output unit;

wherein the estimation result output unit estimates and outputs whether the provided face image shows the specific expression or not on the basis of a feature quantity that has been learned in advance, and

the discrimination unit adds the product of the result of estimation of a weak hypothesis and reliability that has been learned on the basis of estimation performance of the weak hypothesis, every time one hypothesis outputs the result of estimation, and discriminating whether the face image shows the specific expression or not on the basis of the result of addition.

43. The facial expression recognition apparatus as claimed in claim 42, wherein the expression recognition unit has plural expression identifiers for identifying faces images showing specific expressions, and an expression decision unit for deciding the expression of the provided face image, and

the expression decision unit decides the expression of the provided face image on the basis of the result of addition of each expression identifier.

44. The facial expression recognition apparatus as claimed in claim 40, wherein the expression recognition unit has an expression identifier for identifying a face image showing a specific expression as a recognition target from the provided face image,

the expression identifier is a support vector machine (SVM) for identifying whether the face feature extracted by the face feature extraction unit is the specific expression or not, using a support vector that has been learned in advance, and

the support vector has been learned in advance on the basis of a face feature extracted from an expression learning data set including plural face images showing a specific expression as a recognition target and plural images showing expressions different from the specific expression to extract a face feature from the expression learning data set, by using all Gabor filters selected as weak hypotheses when repeatedly generating a weak hypothesis for estimating whether a provided face image shows the specific expression or not on the basis of the result of filtering of one Gabor filter selected form plural Gabor filters.

45. A facial expression recognition method comprising:

a face feature extraction step of filtering a provided face image by using a Gabor filter and thus extracting a face feature; and

an expression recognition step of recognizing an expression of the provided face image on the basis of the face feature.

46. The facial expression recognition method as claimed in claim 45, wherein the expression recognition step includes a specific expression identification step of identifying a face image showing a specific expression as a recognition target from the provided face image, and

at the specific expression identification step, whether the face feature extracted at the face feature extraction step is the specific expression or not is identified by a support vector machine (SVM) using a support vector that has been learned in advance.

47. The facial expression recognition method as claimed in claim 45, wherein the expression recognition step has a specific expression identification step of identifying a face image showing a specific expression as a recognition target from the provided face image, and

the specific expression identification step includes:

an estimation result output step of estimating whether the provided face image shows the specific expression or not by each plural weak hypotheses on the basis of a feature quantity that has been learned in advance; and

a discrimination step of discriminating whether the provided face image shows the specific expression on the basis of the result of estimation outputted from the estimation result output step;

wherein at the estimation result output step, the product of the result of estimation of a weak hypothesis and reliability that has been learned on the basis of estimation performance of the weak hypothesis is added every time one hypothesis outputs the result of estimation, and whether the face image shows the specific expression or not is discriminated on the basis of the result of addition.

48. The facial expression recognition method as claimed in claim 45, wherein the expression recognition step has a specific expression identification step of identifying a face image showing the specific expression from the provided face image,

the specific expression identification step is a step of identifying whether the face feature extracted at the face feature extraction step represents a face image showing the specific expression or not by a support vector machine (SVM) using a support vector that has been learned in advance, and

the support vector has been learned in advance on the basis of a face feature extracted from an expression learning data set including plural face images showing a specific expression as a recognition target and plural images showing expressions different from the specific expression to extract a face feature from the expression learning data set, by using all Gabor filters selected as weak hypotheses when repeatedly generating a weak hypothesis for estimating whether a provided face image shows the specific expression or not on the basis of the result of filtering of one Gabor filter selected form plural Gabor filters.

49-50. -50. (canceled)