MENTAL STATE EVENT DEFINITION GENERATION

• US 20150313530A1
• Filed: 07/10/2015
• Published: 11/05/2015
• Est. Priority Date: 08/16/2013
• Status: Abandoned Application

• Current Assignee: Affectiva, Inc. (Smart Eye AB)
• Original Assignee: Affectiva, Inc. (Smart Eye AB)
• Inventors: Kodra, Evan, el Kaliouby, Rana, Vandal, Thomas James
• Application Number:
  US14/796,419
• Publication Number:
  US 20150313530A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/6898 Portable consumer electronic devices, e.g. music players, telephones, tablet computers
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers
• CPC Class Codes: G06Q 30/0242 Determining effectiveness of advertisements
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G09B 5/06 with both visual and audible presentation of the material to be studied
• CPC Class Codes: G16H 20/70 relating to mental therapies, e.g. psychological therapy or autogenous training
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16H 40/67 for remote operation
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: G16H 50/70 for mining of medical data, e.g. analysing previous cases of other patients

Abstract

Analysis of mental states is provided based on videos of a plurality of people experiencing various situations such as media presentations. Videos of the plurality of people are captured and analyzed using classifiers. Facial expressions of the people in the captured video are clustered based on set criteria. A unique signature for the situation to which the people are being exposed is then determined based on the expression clustering. In certain scenarios, the clustering is augmented by self-report data from the people. In embodiments, the expression clustering is based on a combination of multiple facial expressions.

Claims

1. A computer-implemented method for analysis comprising:

obtaining a plurality of videos of people;

analyzing the plurality of videos using classifiers;

performing expression clustering based on the analyzing; and

determining a temporal signature for an event based on the expression clustering.

2. The method of claim 1 wherein the temporal signature includes a length.

3. The method of claim 2 wherein the length is computed based on detection of adjacent local minima of a facial expression probability curve.

4. The method of claim 1 wherein the temporal signature includes a peak intensity.

5. The method of claim 1 wherein the temporal signature includes a shape for an intensity transition from low intensity to a peak intensity.

6. The method of claim 1 wherein the temporal signature includes a shape for an intensity transition from a peak intensity to low intensity.

7. The method of claim 1 wherein the plurality of videos are of people who are viewing substantially identical situations that include viewing media.

8. The method of claim 7 wherein the media is oriented toward an emotion.

9. The method of claim 8 wherein the emotion includes one or more of humor, sadness, poignancy, and mirth.

11. The method of claim 1 wherein the temporal signature includes a peak intensity and a rise rate to the peak intensity.

12. The method of claim 11 further comprising filtering events having the peak intensity less than a predetermined threshold.

14. The method of claim 1 wherein the temporal signature includes a rise rate, a peak intensity, and a decay rate.

15. The method of claim 14 wherein the analyzing further comprises classifying a facial expression as belonging to a category of posed or spontaneous.

16. The method of claim 1 wherein a classifier, from the classifiers, is used on a mobile device where the plurality of videos are obtained with the mobile device.

17. The method of claim 1 wherein the expression clustering is for smiles, smirks, brow furrows, squints, lowered eyebrows, raised eyebrows, or attention.

18. The method of claim 1 wherein the expression clustering is for inner brow raiser, outer brow raiser, brow lowerer, upper lid raiser, cheek raiser, lid tightener, lips toward each other, nose wrinkle, upper lid raiser, nasolabial deepener, lip corner puller, sharp lip puller, dimpler, lip corner depressor, lower lip depressor, chin raiser, lip pucker, tongue show, lip stretcher, neck tightener, lip funneler, lip tightener, lips part, jaw drop, mouth stretch, lip suck, jaw thrust, jaw sideways, jaw clencher, lip bite, cheek blow, cheek puff, cheek suck, tongue bulge, lip wipe, nostril dilator, nostril compressor, glabella lowerer, inner eyebrow lowerer, eyes closed, eyebrow gatherer, blink, wink, head turn left, head turn right, head up, head down, head tilt left, head tilt right, head forward, head thrust forward, head back, head shake up and down, head shake side to side, head upward and to a side, eyes turn left, eyes left, eyes turn right, eyes right, eyes up, eyes down, walleye, cross-eye, upward rolling of eyes, clockwise upward rolling of eyes, counter-clockwise upward rolling of eyes, eyes positioned to look at other person, head and/or eyes look at other person, sniff, speech, swallow, chewing, shoulder shrug, head shake back and forth, head nod up and down, flash, partial flash, shiver/tremble, or fast up-down look.

19. The method of claim 1 wherein the expression clustering is for a combination of facial expressions.

20. The method of claim 1 further comprising using the temporal signature to infer a mental state where the mental state includes one or more of sadness, stress, anger, happiness, disgust, frustration, confusion, disappointment, hesitation, cognitive overload, focusing, engagement, attention, boredom, exploration, confidence, trust, delight, skepticism, doubt, satisfaction, excitement, laughter, calmness, and curiosity.

21. The method of claim 1 wherein the analyzing includes:

identifying a human face within a frame of a video selected from the plurality of videos;

defining a region of interest (ROI) in the frame that includes the identified human face;

extracting one or more histogram-of-oriented-gradients (HoG) features from the ROI; and

computing a set of facial metrics based on the one or more HoG features.

22. The method of claim 21 further comprising smoothing each metric from the set of facial metrics.

24. The method of claim 1 wherein the performing expression clustering comprises performing K-means clustering.

26. The method of claim 1 further comprising associating demographic information with each event.

27. The method of claim 26 wherein the demographic information includes country of residence.

28. The method of claim 27 further comprising generating an international event signature profile.

29. The method of claim 1 wherein the analyzing includes:

identifying multiple human faces within a frame of a video selected from the plurality of videos;

defining a region of interest (ROI) in the frame for each identified human face;

extracting one or more histogram-of-oriented-gradients (HoG) features from each ROI; and

computing a set of facial metrics based on the one or more HoG features for each of the multiple human faces.

10. (canceled)

13. (canceled)

23. (canceled)

25. (canceled)

30. A computer program product embodied in a non-transitory computer readable medium for analysis, the computer program product comprising:

code for obtaining a plurality of videos of people;

code for analyzing the plurality of videos using classifiers;

code for performing expression clustering based on the analyzing; and

code for determining a temporal signature for an event based on the expression clustering.

31. A computer system for analysis comprising:

a memory which stores instructions;

one or more processors attached to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to;

obtain a plurality of videos of people;

analyze the plurality of videos using classifiers;

perform expression clustering based on the analyzing; and

determine a temporal signature for an event based on the expression clustering.

32-33. -33. (canceled)