Personal emotional profile generation

• US 10,111,611 B2
• Filed: 07/10/2014
• Issued: 10/30/2018
• Est. Priority Date: 06/07/2010
• Status: Active Grant

• Current Assignee: Affectiva, Inc. (Smart Eye AB)
• Original Assignee: Affectiva, Inc. (Smart Eye AB)
• Inventors: el Kaliouby, Rana, England, Avril
• Primary Examiner(s): Tucker, Wesley J
• Application Number:
  US14/328,554
• Publication Number:
  US 20140323817A1
• Time in Patent Office: 1,573 Days
• CPC Class Codes: A61B 5/02055 Simultaneously evaluating both cardiovascular condition and temperature
• CPC Class Codes: A61B 5/02405 Determining heart rate variability
• CPC Class Codes: A61B 5/0533 Measuring galvanic skin response
• CPC Class Codes: A61B 5/08 Detecting, measuring or recording devices for evaluating the respiratory organs A61B5/0205 takes precedence
• CPC Class Codes: A61B 5/11 Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb for measuring pulse A61B5/02 ;A61B5/1038 takes precedence; motion detection to correct for motion artifacts in physiological signals A61B5/721
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/6824 Arm or wrist
• CPC Class Codes: G06Q 30/0271 Personalized advertisement
• CPC Class Codes: G06Q 30/0282 Rating or review of business operators or products
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G16H 20/40 relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 20/70 relating to mental therapies, e.g. psychological therapy or autogenous training
• CPC Class Codes: G16H 40/67 for remote operation

Abstract

The mental state of an individual is obtained in order to generate an emotional profile for the individual. The individual'"'"'s mental state is derived from an analysis of the individual'"'"'s facial and physiological information. The emotional profile of other individuals is correlated to the first individual for comparison. Various categories of emotional profiles are defined based upon the correlation. The emotional profile of the individual or group of individuals is rendered for display, used to provide feedback and to recommend activities for the individual, or provide information about the individual.

Claims

1. A computer-implemented method for mental state analysis comprising:

obtaining mental state data from an individual, wherein the mental state data is extracted, using one or more processors, from facial images of an individual captured as they respond to stimuli;

analyzing the mental state data extracted from facial images to produce mental state information;

correlating the mental state information of the individual with mental state information from a plurality of people, wherein the mental state information from the plurality of people is determined as a function of at least one historical environmental condition measured from ambient conditions affecting at least one of the plurality of people and stored as historical contextual information;

categorizing, using one or more processors, the individual with others from the plurality of people based on the correlating and a rate of change of a plurality of expressions of the individual, wherein the rate of change is evaluated during exposure to specific stimuli and wherein the categorizing is part of a profile;

determining at least one recent environmental condition by measuring an ambient condition affecting the individual during the exposure to specific stimuli and storing the measurement as recent contextual information; and

predicting a future reaction to a stimulus, based on the profile, the historical contextual information, and the recent contextual information.

2. The method of claim 1 wherein the profile summarizes mental state analysis.

3. The method of claim 1 wherein the profile comprises an emotographic profile of the individual.

4. The method of claim 1 wherein the categorizing includes evaluation of facial expressions for anger, sadness, happiness, or disgust.

5. The method of claim 4 further comprising pushing media content to the individual based on the categorizing.

6. The method of claim 1 wherein the profile includes demographic information.

7. The method of claim 1 further comprising extrapolating behavior based on the profile.

8. The method of claim 1 further comprising correlating a component of the profile to a purchase intent.

9. The method of claim 1 further comprising targeting advertisements based on the profile.

10. The method of claim 1 further comprising using the profile to predict a preference.

11. The method of claim 1 further comprising posting a representation of the profile to a social media site.

12. The method of claim 1 further comprising using the profile to categorize a product or service.

13. The method of claim 1 further comprising matching the individual with one or more people from the plurality of people, based on the categorizing.

14. The method of claim 1 wherein the analyzing of the mental state data is accomplished, at least in part, on a device doing collecting of the mental state data and wherein the device doing the collecting includes a mobile device.

15. The method of claim 1 wherein the obtaining is accomplished by collecting the mental state data over time.

16. The method of claim 1 wherein the obtaining is accomplished by collecting the mental state data with a plurality of devices.

17. The method of claim 1 further comprising developing norms for the individual.

18. The method of claim 17 further comprising using response differences in the individual, over an interval of time, from the norms for the individual to make a recommendation.

19. The method of claim 1 further comprising making a recommendation for digital media based on the categorizing.

20. The method of claim 1 further comprising developing norms for a demographic group.

21. The method of claim 20 further comprising using response differences for the individual from the norms for the demographic group to refine a profile.

22. The method of claim 1 further comprising performing emotional journaling based on the mental state information for the individual.

23. The method of claim 1 wherein the mental state data includes facial data, physiological data, or accelerometer data.

26. The method of claim 1 wherein the images capture physiological data.

27. The method of claim 26 further comprising extracting emotional state information from the captured physiological data.

28. The method of claim 3 further comprising periodically updating the emotographic profile of an individual as additional profile information is collected.

29. The method of claim 2 further comprising rendering, on a device of the individual, the profile and a mental state score, wherein the mental state score is based on mental state data collected over a period of time.

30. The method of claim 1 further comprising making activity recommendations based upon the profile.

31. The method of claim 30 wherein the activity recommendations are based on time of day.

32. The method of claim 30 wherein the activity recommendations are based upon a period of time during the day.

33. The method of claim 30 wherein the activity recommendations further comprise calendar date based scheduling.

34. The method of claim 1 wherein the at least one environmental condition is a temperature or humidity.

35. The method of claim 1 further comprising rendering an output comprising an emotional analysis of at least one of mental state information of the individual and mental state information from the plurality of people.

36. The method of claim 35 wherein the output comprises a graphical timeline depicting changes in the emotional state of at least one of the individual and the plurality of people.

37. The method of claim 35 wherein the output comprises a bar graph depicting relative estimated levels of at least two emotions over the course of a plurality of days.

38. The method of claim 1 wherein the analyzing includes inferring mental states.

24. A computer program product embodied in a non-transitory computer readable medium for mental state analysis, the computer program product comprising code which causes one or more processors to perform operations of:

obtaining mental state data from an individual, wherein the mental state data is extracted from facial images of an individual captured as they respond to stimuli;

analyzing the mental state data extracted from facial images to produce mental state information;

correlating the mental state information of the individual with mental state information from a plurality of people, wherein the mental state information from the plurality of people is determined as a function of at least one historical environmental condition measured from ambient conditions affecting at least one of the plurality of people and stored as historical contextual information;

categorizing the individual with others from the plurality of people based on the correlating and a rate of change of a plurality of expressions of the individual, wherein the rate of change is evaluated during exposure to specific stimuli and wherein the categorizing is part of a profile;

determining at least one recent environmental condition by measuring an ambient condition affecting the individual during the exposure to specific stimuli and storing the measurement as recent contextual information; and

predicting a future reaction to a stimulus, based on the profile, the historical contextual information, and the recent contextual information.

25. A computer system for mental state analysis comprising:

a memory which stores instructions;

one or more processors attached to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to;

obtain mental state data from an individual, wherein the mental state data is extracted from facial images of an individual captured as they respond to stimuli;

analyze the mental state data extracted from facial images to produce mental state information;

correlate the mental state information of the individual with mental state information from a plurality of people, wherein the mental state information from the plurality of people is determined as a function of at least one historical environmental condition measured from ambient conditions affecting at least one of the plurality of people and stored as historical contextual information;

categorize the individual with others from the plurality of people based on the correlating and a rate of change of a plurality of expressions of the individual, wherein the rate of change is evaluated during exposure to specific stimuli and wherein the categorizing is part of a profile;

determine at least one recent environmental condition by measuring an ambient condition affecting the individual during the exposure to specific stimuli and storing the measurement as recent contextual information; and

predict a future reaction to a stimulus, based on the profile, the historical contextual information, and the recent contextual information.