FACIAL ANALYSIS TO DETECT ASYMMETRIC EXPRESSIONS

• US 20140016860A1
• Filed: 09/19/2013
• Published: 01/16/2014
• Est. Priority Date: 06/07/2010
• Status: Active Grant

• Current Assignee: Affectiva, Inc. (Smart Eye AB)
• Original Assignee: Affectiva, Inc. (Smart Eye AB)
• Inventors: Senechal, Thibaud, el Kaliouby, Rana
• Granted Patent: US 10,108,852 B2
• Time in Patent Office: Days
• US Class Current: 382/159
• CPC Class Codes: A61B 5/02055 Simultaneously evaluating both cardiovascular condition and temperature
• CPC Class Codes: A61B 5/02405 Determining heart rate variability
• CPC Class Codes: A61B 5/0533 Measuring galvanic skin response
• CPC Class Codes: A61B 5/08 Detecting, measuring or recording devices for evaluating the respiratory organs A61B5/0205 takes precedence
• CPC Class Codes: A61B 5/11 Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb for measuring pulse A61B5/02 ;A61B5/1038 takes precedence; motion detection to correct for motion artifacts in physiological signals A61B5/721
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: G06Q 30/0242 Determining effectiveness of advertisements
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G16H 20/30 relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
• CPC Class Codes: G16H 20/70 relating to mental therapies, e.g. psychological therapy or autogenous training
• CPC Class Codes: G16H 30/20 for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16H 40/67 for remote operation

Abstract

A system and method for facial analysis to detect asymmetric expressions is disclosed. A series of facial images is collected, and an image from the series of images is evaluated with a classifier. The image is then flipped to create a flipped image. Then, the flipped image is evaluated with the classifier. The results of the evaluation of original image and the flipped image are compared. Asymmetric features such as a wink, a raised eyebrow, a smirk, or a wince are identified. These asymmetric features are associated with mental states such as skepticism, contempt, condescension, repugnance, disgust, disbelief, cynicism, pessimism, doubt, suspicion, and distrust.

Claims

1. A computer-implemented method for facial analysis comprising:

collecting a series of images of a face;

evaluating an image from the series of images with a classifier;

flipping the image to create a flipped image;

evaluating the flipped image with the classifier;

comparing results from the evaluating of the image and the evaluating the flipped image; and

identifying an asymmetrical expression on the face based on the comparing.

2. The method of claim 1 wherein the flipping includes taking a mirror image about a y-axis.

3. The method of claim 1 further comprising centering the face within the image.

4. The method of claim 1 further comprising compensating the face for a tilted head.

5. The method of claim 1 further comprising scoring asymmetry based on the comparing.

6. The method of claim 5 wherein the scoring provides a probability of a facial expression occurring.

7. The method of claim 1 further comprising detecting a fast onset of the asymmetrical expression from the series of images.

8. The method of claim 1 further comprising detecting a fast offset of the asymmetrical expression from the series of images.

9. The method of claim 8 further comprising filtering based on both the fast offset and a fast onset of the asymmetrical expression from the series of images.

10. The method of claim 1 further comprising inferring mental states based on the asymmetrical expression.

11. The method of claim 1 wherein the asymmetrical expression includes a smirk.

12. The method of claim 11 further comprising evaluating skepticism or contempt based on the smirk.

13. The method of claim 1 wherein the asymmetrical expression includes a wink.

14. The method of claim 1 wherein the asymmetrical expression includes a single brow rise.

15. The method of claim 1 wherein the asymmetrical expression includes a cheek puff.

16. The method of claim 1 wherein the asymmetrical expression is a unilateral expression.

17. The method of claim 1 wherein the series of images are from a video.

18. The method of claim 17 wherein the video is captured with a webcam.

19. The method of claim 1 further comprising training the classifier based on a receiver operating characteristic (ROC) curve.

20. The method of claim 19 wherein the training the classifier is further based on an area under curve (AUC) analysis.

21. The method of claim 1 wherein the classifier is based on one or more facial action units.

22. The method of claim 21 wherein the one or more facial action units includes a dimpler or a lip corner pull.

23. The method of claim 1 further comprising evaluating dynamic facial movements.

24. The method of claim 1 further comprising evaluating a sub-region of the face.

25. The method of claim 24 wherein the sub-region includes mouth or eyes.

26. The method of claim 1 further comprising using image descriptors to evaluate the image.

27. The method of claim 26 wherein the image descriptors include one or more of texture, edges, and color relating to a facial expression.

28. The method of claim 1 further comprising detecting landmarks on the face.

29. The method of claim 28 further comprising performing feature extraction based on the landmarks which were detected.

30. The method of claim 29 wherein the landmarks on the face include one or more from a group including eye corners, mouth corners, and eyebrows.

31. A computer program product embodied in a non-transitory computer readable medium for facial analysis, the computer program product comprising:

code for collecting a series of images of a face;

code for evaluating an image from the series of images with a classifier;

code for flipping the image to create a flipped image;

code for evaluating the flipped image with the classifier;

code for comparing results from the evaluating of the image and the evaluating the flipped image; and

code for identifying an asymmetrical expression on the face based on the comparing.

32. A computer system for facial analysis comprising:

a memory which stores instructions;

one or more processors coupled to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to;

collect a series of images of a face;

evaluate an image from the series of images with a classifier;

flip the image to create a flipped image;

evaluate the flipped image with the classifier;

compare results from the evaluating of the image and the evaluating the flipped image; and

identify an asymmetrical expression on the face based on the comparing.

33. A computer-implemented method for facial analysis comprising:

collecting a series of images of a face;

evaluating an image from the series of images;

identifying an asymmetrical expression on the face in the image; and

using the asymmetrical expression to evaluate a mental state.