Facial analysis to detect asymmetric expressions

• US 10,108,852 B2
• Filed: 09/19/2013
• Issued: 10/23/2018
• Est. Priority Date: 06/07/2010
• Status: Active Grant

• Current Assignee: Affectiva, Inc. (Smart Eye AB)
• Original Assignee: Affectiva, Inc. (Smart Eye AB)
• Inventors: Senechal, Thibaud, el Kaliouby, Rana
• Primary Examiner(s): Rudolph, Vincent
• Assistant Examiner(s): Azima, Shaghayegh
• Application Number:
  US14/031,907
• Publication Number:
  US 20140016860A1
• Time in Patent Office: 1,860 Days
• Field of Search: 382103, 382224, 382159
• CPC Class Codes: A61B 5/02055 Simultaneously evaluating both cardiovascular condition and temperature
• CPC Class Codes: A61B 5/02405 Determining heart rate variability
• CPC Class Codes: A61B 5/0533 Measuring galvanic skin response
• CPC Class Codes: A61B 5/08 Detecting, measuring or recording devices for evaluating the respiratory organs A61B5/0205 takes precedence
• CPC Class Codes: A61B 5/11 Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb for measuring pulse A61B5/02 ;A61B5/1038 takes precedence; motion detection to correct for motion artifacts in physiological signals A61B5/721
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: G06Q 30/0242 Determining effectiveness of advertisements
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G16H 20/30 relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
• CPC Class Codes: G16H 20/70 relating to mental therapies, e.g. psychological therapy or autogenous training
• CPC Class Codes: G16H 30/20 for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16H 40/67 for remote operation

Abstract

A system and method for facial analysis to detect asymmetric expressions is disclosed. A series of facial images is collected, and an image from the series of images is evaluated with a classifier. The image is then flipped to create a flipped image. Then, the flipped image is evaluated with the classifier. The results of the evaluation of original image and the flipped image are compared. Asymmetric features such as a wink, a raised eyebrow, a smirk, or a wince are identified. These asymmetric features are associated with mental states such as skepticism, contempt, condescension, repugnance, disgust, disbelief, cynicism, pessimism, doubt, suspicion, and distrust.

Claims

1. A computer-implemented method for facial analysis comprising:

collecting a series of images of a face;

evaluating an image from the series of images with a classifier;

flipping the image to create a flipped image;

evaluating the flipped image with the classifier;

comparing results from the evaluating of the image and the evaluating the flipped image;

identifying an asymmetrical expression on the face based on the comparing;

inferring mental states based on the asymmetrical expression wherein the mental states include a mental state of disbelief; and

scoring the asymmetrical expression based on the comparing, wherein the left and right sides of the face are different, and wherein the scoring provides a probability of a facial expression occurring, wherein the scoring includes;

providing a histogram computed from the image to a classifier to produce a first output;

providing a histogram computed from the flipped image to a classifier to produce a second output; and

computing a difference between the first output and the second output by subtracting the second output from the first output.

2. The method of claim 1 wherein the flipping includes taking a mirror image about a y-axis.

3. The method of claim 1 further comprising centering the face within the image.

4. The method of claim 1 further comprising compensating the face for a tilted head.

5. The method of claim 1 further comprising detecting a fast onset of the asymmetrical expression from the series of images.

6. The method of claim 1 further comprising detecting a fast offset of the asymmetrical expression from the series of images.

7. The method of claim 6 further comprising filtering based on both the fast offset and a fast onset of the asymmetrical expression from the series of images.

8. The method of claim 1 wherein the asymmetrical expression includes a smirk.

9. The method of claim 8 further comprising evaluating skepticism or contempt based on the smirk.

10. The method of claim 1 wherein the asymmetrical expression includes a wink.

11. The method of claim 1 wherein the asymmetrical expression includes a single brow rise.

12. The method of claim 1 wherein the asymmetrical expression includes a cheek puff.

13. The method of claim 1 wherein the asymmetrical expression is a unilateral expression.

14. The method of claim 1 wherein the series of images are from a video.

15. The method of claim 14 wherein the video is captured with a webcam.

16. The method of claim 1 further comprising training the classifier based on a receiver operating characteristic (ROC) curve.

17. The method of claim 16 wherein the training the classifier is further based on an area under curve (AUC) analysis.

18. The method of claim 1 wherein the classifier is based on one or more facial action units.

19. The method of claim 18 wherein the one or more facial action units includes a dimpler or a lip corner pull.

20. The method of claim 1 further comprising evaluating dynamic facial movements.

21. The method of claim 1 further comprising evaluating a sub-region of the face.

22. The method of claim 21 wherein the sub-region includes mouth or eyes.

23. The method of claim 1 further comprising using image descriptors to evaluate the image.

24. The method of claim 23 wherein the image descriptors include one or more of texture, edges, and color relating to a facial expression.

25. The method of claim 1 further comprising detecting landmarks on the face.

26. The method of claim 25 further comprising performing feature extraction based on the landmarks which were detected.

27. The method of claim 26 wherein the landmarks on the face include one or more from a group including eye corners, mouth corners, and eyebrows.

28. The method of claim 1 wherein the mental state of disbelief includes skepticism, contempt, condescension, repugnance, disgust, disbelief, cynicism, pessimism, doubt, suspicion, or distrust.

29. The method of claim 1 wherein the scoring includes:

providing a second histogram computed from the image to a classifier to produce a third output;

providing a second histogram computed from the flipped image to a classifier to produce a fourth output; and

computing a difference between the third output and the fourth output by subtracting the fourth output from the third output.

30. The method of claim 29 wherein the scoring includes determining a summation of differences by summing:

the difference between the first output and the second output; and

the difference between the third output and the fourth output.

31. The method of claim 30 wherein the scoring includes determining a likelihood of unilateral expression by determining the absolute value of the summation of differences.

32. The method of claim 1 wherein the histograms comprise histogram of oriented gradients (HOG) analysis.

33. A computer program product embodied in a non-transitory computer readable medium for facial analysis, the computer program product comprising code which causes one or more processors to perform operations of:

collecting a series of images of a face;

evaluating an image from the series of images with a classifier;

flipping the image to create a flipped image;

evaluating the flipped image with the classifier;

comparing results from the evaluating of the image and the evaluating the flipped image;

identifying an asymmetrical expression on the face based on the comparing;

inferring mental states based on the asymmetrical expression wherein the mental states include a mental state of disbelief; and

scoring the asymmetrical expression based on the comparing, wherein the left and right sides of the face are different, and wherein the scoring provides a probability of a facial expression occurring, wherein the scoring includes;

providing a histogram computed from the image to a classifier to produce a first output;

providing a histogram computed from the flipped image to a classifier to produce a second output; and

computing a difference between the first output and the second output by subtracting the second output from the first output.

34. A computer system for facial analysis comprising:

a memory which stores instructions;

one or more processors coupled to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to;

collect a series of images of a face;

evaluate an image from the series of images with a classifier;

flip the image to create a flipped image;

evaluate the flipped image with the classifier;

compare results from the evaluating of the image and the evaluating the flipped image;

identify an asymmetrical expression on the face based on the comparing;

infer mental states based on the asymmetrical expression wherein the mental states include a mental state of disbelief; and

score the asymmetrical expression based on the comparing, wherein the left and right sides of the face are different, and wherein the scoring provides a probability of a facial expression occurring, wherein the scoring includes;

providing a histogram computed from the image to a classifier to produce a first output;

providing a histogram computed from the flipped image to a classifier to produce a second output; and

compute a difference between the first output and the second output by subtracting the second output from the first output.

35. A computer-implemented method for facial analysis comprising:

collecting a series of images of a face;

evaluating an image from the series of images;

identifying an asymmetrical expression on the face in the image;

scoring the asymmetrical expression, wherein the left and right sides of the face are different, and wherein the scoring provides a probability of a facial expression occurring, wherein the scoring includes;

providing a histogram computed from the image to a classifier to produce a first output;

providing a histogram computed from a flipped image to a classifier to produce a second output; and

computing a difference between the first output and the second output by subtracting the second output from the first output; and

using the asymmetrical expression and score to evaluate a mental state wherein the mental state includes a mental state of disbelief.