Sporadic collection of mobile affect data

• US 9,204,836 B2
• Filed: 10/26/2013
• Issued: 12/08/2015
• Est. Priority Date: 06/07/2010
• Status: Active Grant

• Current Assignee: Affectiva, Inc. (Smart Eye AB)
• Original Assignee: Affectiva, Inc. (Smart Eye AB)
• Inventors: Bender, Daniel, el Kaliouby, Rana, Kodra, Evan, Nowak, Oliver Ernst, Sadowsky, Richard Scott
• Primary Examiner(s): TUCKER, WESLEY J
• Application Number:
  US14/064,136
• Publication Number:
  US 20140051047A1
• Time in Patent Office: 773 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A61B 5/0022 Monitoring a patient using a global network, e.g. telephone networks, internet
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/02055 Simultaneously evaluating both cardiovascular condition and temperature
• CPC Class Codes: A61B 5/02405 Determining heart rate variability
• CPC Class Codes: A61B 5/0531 Measuring skin impedance
• CPC Class Codes: A61B 5/0533 Measuring galvanic skin response
• CPC Class Codes: A61B 5/08 Detecting, measuring or recording devices for evaluating the respiratory organs A61B5/0205 takes precedence
• CPC Class Codes: A61B 5/11 Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb for measuring pulse A61B5/02 ;A61B5/1038 takes precedence; motion detection to correct for motion artifacts in physiological signals A61B5/721
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/6898 Portable consumer electronic devices, e.g. music players, telephones, tablet computers
• CPC Class Codes: A61B 5/72 Signal processing specially adapted for physiological signals or for diagnostic purposes character recognition, recognising digital ink, document-oriented image-based pattern recognition G06V30/00; recognition or understanding of postal images G06V30/424
• CPC Class Codes: G06Q 30/0271 Personalized advertisement
• CPC Class Codes: G16H 20/70 relating to mental therapies, e.g. psychological therapy or autogenous training
• CPC Class Codes: G16H 40/67 for remote operation

Abstract

A user may react to an interaction by exhibiting a mental state. A camera or other monitoring device can be used to capture one or more manifestations of the user'"'"'s mental state, such as facial expressions, electrodermal activity, or movements. However, there may be conditions where the monitoring device is not able to detect the manifestation continually. Thus, various capabilities and implementations are described where the mental state data is collected on an intermittent basis, analyzed, and an output rendered based on the analysis of the mental state data.

Claims

1. A computer-implemented method for mental state analysis comprising:

collecting mental state data of an individual on an intermittent basis;

collecting other mental state data, including electrodermal activity data, from the individual on a continuous basis;

obtaining analysis of the mental state data on the individual, as a result of sending a request to a web service for the analysis where the analysis of the mental state data is received from the web service;

interpolating, using one or more processors, mental state analysis in between the collecting which is intermittent;

imputing additional mental state data where the mental state data is missing;

filtering out faces of one or more other people to determine when an individual is looking in a direction of a camera;

determining contextual information based on accelerometer data; and

rendering an output based on the analysis of the mental state data.

2. The method of claim 1 further comprising interpolating mental state data in between the collecting which is intermittent.

3. The method of claim 1 further comprising collecting other mental state data from the individual on a continuous basis.

4. The method of claim 1 wherein the collecting of the mental state data is accomplished with a mobile device.

5. The method of claim 1 wherein the collecting of the mental state data is accomplished for one type of mental state data.

6. The method of claim 5 wherein the one type of mental state data includes facial data.

7. The method of claim 6 wherein the facial data is collected with a webcam.

8. The method of claim 6 wherein the facial data is collected intermittently when the individual is looking in a direction of a camera.

9. The method of claim 8 further comprising performing face detection to determine when the individual is looking in the direction of the camera.

10. The method of claim 8 further comprising filtering out faces of one or more other people to determine when the individual is looking in the direction of the camera.

11. The method of claim 1 further comprising determining contextual information.

12. The method of claim 1 further comprising sending one or more of the mental state data, a subset of the mental state data, or an initial analysis of the mental state data to the web service.

13. The method of claim 1 wherein the mental state data includes one or more of smiles, laughter, smirks, or grimaces.

14. The method of claim 1 further comprising posting the analysis to a social network.

15. A computer program product embodied in a non-transitory computer readable medium for mental state analysis, the computer program product comprising:

code for collecting mental state data of an individual on an intermittent basis;

code for collecting other mental state data, including electrodermal activity data, from the individual on a continuous basis;

code for obtaining analysis of the mental state data on the individual, as a result of sending a request to a web service for the analysis where the analysis of the mental state data is received from the web service;

code for interpolating, using one or more processors, mental state analysis in between the collecting which is intermittent;

code for imputing additional mental state data where the mental state data is missing;

code for filtering out faces of one or more other people to determine when an individual is looking in a direction of a camera;

code for determining contextual information based on accelerometer data; and

code for rendering an output based on the analysis of the mental state data.

17. The computer program product of claim 15 further comprising code for interpolating mental state data in between the collecting which is intermittent.

18. The computer program product of claim 15 further comprising code for collecting other mental state data from the individual on a continuous basis.

19. The computer program product of claim 15 wherein the collecting of the mental state data accomplished for one type of mental state data.

20. The computer program product of claim 19 wherein the one type of mental state data includes facial data.

21. The computer program product of claim 15 wherein the mental state data includes one or more of smiles, laughter, smirks, or grimaces.

22. The computer program product of claim 15 further comprising code for posting the analysis to a social network.

16. A system for mental state analysis comprising:

a memory for storing instructions;

one or more processors attached to the memory wherein the one or more processors are configured to;

collect mental state data of an individual on an intermittent basis;

collect other mental state data, including electrodermal activity data, from the individual on a continuous basis;

obtain analysis of the mental state data on the individual, as a result of sending a request to a web service for the analysis where the analysis of the mental state data is received from the web service;

interpolate, using one or more processors, mental state analysis in between the collecting which is intermittent;

impute additional mental state data where the mental state data is missing;

filter out faces of one or more other people to determine when an individual is looking in a direction of a camera;

determine contextual information based on accelerometer data; and

render an output based on the analysis of the mental state data.