Collection of affect data from multiple mobile devices

• US 9,934,425 B2
• Filed: 12/30/2013
• Issued: 04/03/2018
• Est. Priority Date: 06/07/2010
• Status: Active Grant

• Current Assignee: Affectiva, Inc. (Smart Eye AB)
• Original Assignee: Affectiva, Inc. (Smart Eye AB)
• Inventors: el Kaliouby, Rana, Bender, Daniel Abraham, Kodra, Evan, Nowak, Oliver Ernst, Sadowsky, Richard Scott, Senechal, Thibaud, Turcot, Panu James
• Primary Examiner(s): Tucker, Wesley J
• Application Number:
  US14/144,413
• Publication Number:
  US 20140112540A1
• Time in Patent Office: 1,555 Days
• CPC Class Codes: A61B 5/02055 Simultaneously evaluating both cardiovascular condition and temperature
• CPC Class Codes: A61B 5/02405 Determining heart rate variability
• CPC Class Codes: A61B 5/0533 Measuring galvanic skin response
• CPC Class Codes: A61B 5/08 Detecting, measuring or recording devices for evaluating the respiratory organs A61B5/0205 takes precedence
• CPC Class Codes: A61B 5/11 Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb for measuring pulse A61B5/02 ;A61B5/1038 takes precedence; motion detection to correct for motion artifacts in physiological signals A61B5/721
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: G06Q 30/0271 Personalized advertisement
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G16H 20/40 relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 20/70 relating to mental therapies, e.g. psychological therapy or autogenous training
• CPC Class Codes: G16H 40/67 for remote operation
• CPC Class Codes: G16H 80/00 ICT specially adapted for facilitating communication between medical practitioners or patients, e.g. for collaborative diagnosis, therapy or health monitoring

Abstract

A user interacts with various pieces of technology to perform numerous tasks and activities. Reactions can be observed and mental states inferred from these performances. Multiple devices, including mobile devices, can observe and record or transmit a user'"'"'s mental state data. The mental state data collected from the multiple devices can be used to analyze the mental states of the user. The mental state data can be in the form of facial expressions, electrodermal activity, movements, or other detectable manifestations. Multiple cameras on the multiple devices can be usefully employed to collect facial data. An output can be rendered based on an analysis of the mental state data.

Claims

1. A computer-implemented method for mental state analysis comprising:

obtaining mental state data which is collected on an individual from multiple sources wherein the multiple sources include at least two sources of facial image data, wherein at least one of the facial image data sources collects facial image data intermittently while the individual is looking toward a camera, and wherein the multiple sources include at least one mobile device, including a cell phone, a tablet computer, or a wearable device;

aggregating, using one or more processors, the mental state data from the multiple sources;

obtaining analysis of the mental state data which is aggregated from the multiple sources, wherein the analysis results in mental state analysis;

interpolating the mental state analysis where the mental state data collected is intermittent, wherein the interpolating generates interpolated data for time periods during which mental state data was not collected from one or more of the multiple sources; and

rendering an output based on the analysis of the mental state data.

2. The method of claim 1 wherein the mental state data from multiple sources is tagged.

3. The method of claim 2 wherein the mental state data is tagged with an identity value for the individual.

4. The method of claim 3 wherein the mental state data is tagged with information on context in which the mental state data was collected.

5. The method of claim 2 further comprising partitioning the mental state data based on tagging.

6. The method of claim 1 wherein the at least two sources of the facial image data include one or more of a webcam, a phone camera, a tablet camera, a wearable camera, a room camera, a mobile device, a cell phone, a tablet computer, or a laptop computer.

7. The method of claim 1 wherein the at least one mobile device includes a forward facing camera.

8. The method of claim 1 wherein the analysis of the mental state data is obtained from a web service.

9. The method of claim 1 further comprising performing face detection to determine whether the individual is looking toward the camera.

10. The method of claim 9 further comprising filtering out faces of one or more other people to determine whether the individual is looking toward the camera.

11. The method of claim 1 further comprising interpolating mental state data when the mental state data collected is intermittent.

12. The method of claim 1 further comprising imputing additional mental state data for one or more periods where no mental state data was collected.

13. The method of claim 1 further comprising determining contextual information.

14. The method of claim 13 wherein the contextual information is based on one or more of skin temperature, accelerometer data, a photograph, an email, a text message, a phone log, or GPS information.

15. The method of claim 1 further comprising inferring mental states based on the mental state data which was collected.

16. The method of claim 15 wherein the mental states inferred include one or more of frustration, confusion, disappointment, hesitation, cognitive overload, focusing, being engaged, attending, boredom, exploration, confidence, trust, delight, or satisfaction.

17. The method of claim 16 further comprising populating a calendar based on the mental states which were inferred.

18. The method of claim 1 further comprising posting information based on the analysis to a social network page.

19. The method of claim 1 wherein the mental state data includes one or more of a group including physiological data, facial image data, or accelerometer data.

20. The method of claim 19 wherein the physiological data includes one or more of electrodermal activity, heart rate, heart rate variability, skin temperature, or respiration.

23. The method of claim 1 wherein the facial image data includes abstracted facial expression information.

24. The method of claim 1 wherein the facial image data is used to determine mental state information.

25. The method of claim 1 further comprising stitching mental state analysis from the mental state data from the multiple sources.

26. The method of claim 25 wherein the mental state analysis further comprises identifying a best view, from the multiple sources of facial image data, and stitching the mental state analysis together based on the best view that was identified for the mental state data that was collected intermittently.

27. The method of claim 5 wherein the partitioning of the mental state data is based on tags identifying a source of the intermittent mental state data.

28. The method of claim 2 further comprising analyzing the mental state data that was tagged to generate an emotigraphic profile for the individual.

29. The method of claim 5 further comprising separating the mental state data into two or more groups based on content of tags from the mental state data that was tagged.

30. The method of claim 29 further comprising reassembling the mental state data, based on the tags.

31. The method of claim 30 wherein the reassembling allows combining of images collected from multiple devices.

32. The method of claim 1 wherein the output is used to populate a calendar along with activities already included in the calendar.

33. The method of claim 1 wherein the interpolating generates interpolated data for a first one of the multiple sources for time periods during which mental state data was not collected from the first one of the multiple sources.

34. The method of claim 1 wherein the interpolating generates interpolated data for time periods during which mental state data was not collected from any of the multiple sources.

35. The method of claim 1 wherein the interpolating generates interpolated data as a function of mental state data collected from at least two of the multiple sources.

21. A computer program product embodied in a non-transitory computer readable medium for mental state analysis, the computer program product comprising code which causes one or more processors to perform operations of:

obtaining mental state data which is collected on an individual from multiple sources wherein the multiple sources include at least two sources of facial image data, wherein at least one of the facial image data sources collects facial image data intermittently while the individual is looking toward a camera, and wherein the multiple sources include at least one mobile device, including a cell phone, a tablet computer, or a wearable device;

aggregating the mental state data from the multiple sources;

obtaining analysis of the mental state data which is aggregated from multiple sources, wherein the analysis results in mental state analysis;

interpolating the mental state analysis where the mental state data collected is intermittent, wherein the interpolating generates interpolated data for time periods during which mental state data was not collected from one or more of the multiple sources; and

rendering an output based on the analysis of the mental state data.

22. A computer system for mental state analysis comprising:

a memory which stores instructions;

one or more processors coupled to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to;

obtain mental state data which is collected on an individual from multiple sources wherein the multiple sources include at least two sources of facial image data, wherein at least one of the facial image data sources collects facial image data intermittently while the individual is looking toward a camera, and wherein the multiple sources include at least one mobile device, including a cell phone, a tablet computer, or a wearable device;

aggregate the mental state data from the multiple sources;

obtain analysis of the mental state data which is aggregated from multiple sources, wherein the analysis results in mental state analysis;

interpolate the mental state analysis where the mental state data collected is intermittent, wherein the interpolation generates interpolated data for time periods during which mental state data was not collected from one or more of the multiple sources; and

render an output based on the analysis of the mental state data.