Facial expression recognition systems and methods and computer program products thereof

• US 9,036,018 B2
• Filed: 05/26/2011
• Issued: 05/19/2015
• Est. Priority Date: 06/17/2010
• Status: Active Grant

• Current Assignee: Institute For Information Industry
• Original Assignee: Institute For Information Industry
• Inventors: Wang, Wen-Nan, Lai, Shang-Hong, Liao, Chia-Te, Tsai, Tzu-Chen, Chuang, Hui-Ju, Chang, Fang-kai
• Primary Examiner(s): AN, SHAWN S
• Application Number:
  US13/116,724
• Publication Number:
  US 20110310237A1
• Time in Patent Office: 1,454 Days
• Field of Search: 348/77, 345/420
• US Class Current: 348/77
• CPC Class Codes: A61B 1/00036 Means for power saving, e.g. sleeping mode
• CPC Class Codes: A61B 1/041 Capsule endoscopes for imaging
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

A facial expression recognition system includes a facial database, an image capture unit, an expression recognition data generation unit and an expression recognition unit. The facial database includes a plurality of expression information and expression features of optical flow field, wherein each of the expression features of optical flow field corresponds to one of the expression information. The image capture unit captures a plurality of facial images. The expression recognition data generation unit is coupled to the image capture unit and the facial database for receiving a first facial image and a second facial image from the image capture unit and calculating an expression feature of optical flow field between the first facial image and the second facial image corresponding to each of the expression information. The expression recognition unit is coupled to the expression recognition data generation unit for determining a facial expression corresponding to the first and second facial images according to the calculated expression feature of optical flow field for each of the expression information and the variation features in optical flow in the facial database.

Claims

1. A facial expression recognition method, comprising:

receiving a first facial image and a second facial image;

calculating an expression feature of optical flow field between the first facial image and the second facial image corresponding to each of a plurality of expression information respectively; and

determining a facial expression corresponding to the first and second facial images according to the calculated expression feature of optical flow field for each of the plurality of expression information and a plurality of the expression features of optical flow field in a facial database, wherein the facial database comprises the plurality of expression information and the plurality of expression features of optical flow field, and wherein each of the expression features of optical flow field corresponds to one of the plurality of expression information,

wherein the facial database further comprises a weight mask corresponding to each of the plurality of expression information and the step of calculating the expression feature of optical flow field further comprises;

generating the calculated expression feature of optical flow field according to the expression feature of optical flow field between the first facial image and the second facial image and the weight mask corresponding to each of the plurality of expression information.

2. The facial expression recognition method of claim 1, wherein the step of calculating the expression feature of optical flow field between the first facial image and the second facial image further performs a facial calibration step for adjusting the first facial image to the second facial image.

3. The facial expression recognition method of claim 2, wherein the facial calibration step further comprises:

estimating a motion parameter by utilizing a position of a facial feature point of the first facial image and the second facial image respectively; and

adjusting the first facial image to the second facial image by utilizing the calculated motion parameter.

4. The facial expression recognition method of claim 3, wherein the step of adjusting the first facial image to the second facial image further comprises:

utilizing a facial area in the second facial image to roughly estimate a position of a nose area thereof;

locating the nostrils by differentiating the nose area to search the local maximum of gradient computed from a nose image corresponding to the nose area;

utilizing the located nostrils to determine a nose portion of the second facial image; and

comparing the nose portion of the second facial image to a previously calculated nose portion of the first facial image to obtain the motion parameters of the second image that correspond to a head area of the first facial image and adjusting the second facial image to the first facial image according to the obtained motion parameters.

5. The facial expression recognition method of claim 3, wherein the motion parameters of the first and the second facial images comprise the scaling ratio, the moving direction and the rotation angle.

6. The facial expression recognition method of claim 1, further comprising:

determining a facial important area corresponding to each of the expression information; and

weighting the facial important area to generate a weight mask corresponding to each of the expression information.

7. The facial expression recognition method of claim 1, further comprising:

determining whether the calculated expression feature of optical flow field is similar to one of the plurality of expression features of optical flow field in the facial database; and

if the calculated variation in optical flow is similar to one of the plurality of expression features of optical flow field in the facial database, determining that the expression of the similar one of the expression features in the facial database is the facial expression corresponding to the first and the second facial images.

8. The facial expression recognition method of claim 1, wherein the step of determining the facial expression corresponding to the facial images is performed by utilizing a classifier to determine the corresponding facial expression from the expression information and the variation features in optical flow in the facial database.

9. A facial expression recognition system, comprising:

a facial database, including a plurality of expression information and a plurality of expression features of optical flow field, wherein each of the expression features of optical flow field corresponds to one of the plurality of expression information;

an image capture unit, capturing a plurality of facial images;

an expression recognition data generation unit coupled to the image capture unit and the facial database, receiving a first facial image and a second facial image from the image capture unit and calculating an expression feature of optical flow field between the first facial image and the second facial image corresponding to each of the plurality of expression information respectively; and

an expression recognition unit coupled to the expression recognition data generation unit, determining a facial expression corresponding to the first and second facial images according to the calculated expression feature of optical flow field for each of the plurality of expression information and the variation features in optical flow in the facial database,

wherein the facial database further comprises a weight mask corresponding to each of the plurality of expression information and the expression recognition data generation unit further generates the calculated expression feature of optical flow field according to the expression feature of optical flow field between the first facial image and the second facial image and the weight mask corresponding to each of the plurality of expression information.

10. The facial expression recognition system of claim 9, wherein the expression recognition data generation unit further comprises a calibration module for performing a facial calibration step to adjust the first facial image to the second facial image when calculating the expression features of optical flow field between the first facial image and the second facial image.

11. The facial expression recognition system of claim 10, wherein the calibration module further estimates a motion parameter by utilizing a position of a facial feature point of the first and the second facial images and adjusts the first facial image to the second facial image by utilizing the calculated motion parameter to perform the facial calibration.

12. The facial expression recognition system of claim 11, wherein the calibration module further utilizes a facial area in the second facial image to roughly estimate a position of a nose area thereof, locates the nostrils by differentiating the nose area to search the local maximum of a gradient nose corresponding to the nose area, utilizes the located nostrils to determine a nose portion of the second facial image and compares the nose portion of the second facial image to a previously calculated nose portion of the first facial image to obtain the motion parameters of the second image that correspond to a head area of the first facial image and adjusts the second facial image to the first facial image according to the obtained motion parameters.

13. The facial expression recognition system of claim 11, wherein the motion parameters of the first and the second facial images comprise the scaling ratio, the moving direction and the rotation angle.

14. The facial expression recognition system of claim 9, wherein the weight mask corresponding to each of the expression information is generated by weighting a facial important area corresponding to each of the expression information.

15. The facial expression recognition system of claim 9, wherein the expression recognition unit further determines whether the calculated expression feature of optical flow field is similar to one of the plurality of expression features of optical flow field in the facial database, and determines that the expression of the similar one of the expression features in the facial database is the facial expression corresponding to the first and the second facial images if the calculated variation in optical flow is similar to one of the plurality of expression features of optical flow field of the one of the expression features of the facial database.

16. The facial expression recognition system of claim 9, wherein the expression recognition unit is a classifier for determining the corresponding facial expression from the expression information and the variation features in optical flow in the facial database.

17. A facial expression recognition apparatus, comprising:

a shell;

a facial database disposed inside of the shell, including a plurality of expression information and a plurality of expression features of optical flow field, wherein each of the expression features of optical flow field corresponds to one of the plurality of expression information;

an image capture unit disposed inside of the shell, capturing a plurality of facial images;

an expression recognition data generation unit disposed inside of the shell and coupled to the image capture unit and the facial database, receiving a first facial image and a second facial image from the image capture unit and calculating an expression feature of optical flow field between the first facial image and the second facial image corresponding to each of the plurality of expression information respectively;

an expression recognition unit disposed inside of the shell and coupled to the expression recognition data generation unit, determining a facial expression corresponding to the first and second facial images according to the calculated expression feature of optical flow field for each of the plurality of expression information and the variation features in optical flow in the facial database; and

a display unit disposed outside of the shell and coupled to the expression recognition unit for displaying a result corresponding to the determined facial expression,

wherein the facial database further comprises a weight mask corresponding to each of the plurality of expression information and the expression recognition data generation unit further generates the calculated expression feature of optical flow field according to the expression feature of optical flow field between the first facial image and the second facial image and the weight mask corresponding to each of the plurality of expression information.

18. A non-transitory computer-readable storage medium storing a computer program, which, when executed, causes a device to perform a facial expression recognition method, and the method comprising:

receiving a first facial image and a second facial image;

calculating an expression feature of optical flow field between the first facial image and the second facial image corresponding to each of a plurality of expression information respectively; and

determining a facial expression corresponding to the first and second facial images according to the calculated expression feature of optical flow field for each of the plurality of expression information and a plurality of the expression features of optical flow field in a facial database, wherein the facial database comprises the plurality of expression information and the plurality of expression features of optical flow field, and wherein each of the expression features of optical flow field corresponds to one of the plurality of expression information,

wherein the facial database further comprises a weight mask corresponding to each of the plurality of expression information and the step of calculating the expression feature of optical flow field further comprises;

generating the calculated expression feature of optical flow field according to the expression feature of optical flow field between the first facial image and the second facial image and the weight mask corresponding to each of the plurality of expression information.