Sporadic collection with mobile affect data

• US 10,143,414 B2
• Filed: 12/07/2015
• Issued: 12/04/2018
• Est. Priority Date: 06/07/2010
• Status: Active Grant

• Current Assignee: Affectiva, Inc. (Smart Eye AB)
• Original Assignee: Affectiva, Inc. (Smart Eye AB)
• Inventors: el Kaliouby, Rana, Bender, Daniel, Kodra, Evan, Nowak, Oliver Ernst, Sadowsky, Richard Scott
• Primary Examiner(s): Tucker, Wesley J
• Application Number:
  US14/961,279
• Publication Number:
  US 20160081607A1
• Time in Patent Office: 1,093 Days
• CPC Class Codes: A61B 2562/0219 Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
• CPC Class Codes: A61B 5/0022 Monitoring a patient using a global network, e.g. telephone networks, internet
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/01 Measuring temperature of body parts ; Diagnostic temperature sensing, e.g. for malignant or inflamed tissue clinical contact thermometers G01K13/20
• CPC Class Codes: A61B 5/02055 Simultaneously evaluating both cardiovascular condition and temperature
• CPC Class Codes: A61B 5/02405 Determining heart rate variability
• CPC Class Codes: A61B 5/0531 Measuring skin impedance
• CPC Class Codes: A61B 5/0533 Measuring galvanic skin response
• CPC Class Codes: A61B 5/08 Detecting, measuring or recording devices for evaluating the respiratory organs A61B5/0205 takes precedence
• CPC Class Codes: A61B 5/11 Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb for measuring pulse A61B5/02 ;A61B5/1038 takes precedence; motion detection to correct for motion artifacts in physiological signals A61B5/721
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/6898 Portable consumer electronic devices, e.g. music players, telephones, tablet computers
• CPC Class Codes: A61B 5/72 Signal processing specially adapted for physiological signals or for diagnostic purposes character recognition, recognising digital ink, document-oriented image-based pattern recognition G06V30/00; recognition or understanding of postal images G06V30/424
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06Q 30/0271 Personalized advertisement
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G16H 20/40 : relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 20/70 : relating to mental therapies, e.g. psychological therapy or autogenous training
• CPC Class Codes: G16H 30/20 : for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 : for processing medical images, e.g. editing
• CPC Class Codes: G16H 40/63 : for local operation
• CPC Class Codes: G16H 40/67 : for remote operation
• CPC Class Codes: G16H 50/20 : for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

An individual can exhibit one or more mental states when reacting to a stimulus. A camera or other monitoring device can be used to collect, on an intermittent basis, mental state data including facial data. The mental state data can be interpolated between the intermittent collecting. The facial data can be obtained from a series of images of the individual where the images are captured intermittently. A second face can be identified, and the first face and the second face can be tracked.

Claims

1. A computer-implemented method for mental state analysis comprising:

collecting mental state data of an individual on an intermittent basis wherein the mental state data includes facial image data and the facial image data is collected intermittently across a plurality of devices;

imputing additional mental state data where the mental state data is missing, wherein the imputing is based on mental state data collected from other individuals associated with the individual;

interpolating, using one or more processors, mental state data in between the collecting which is intermittent;

obtaining analysis of the mental state data on the individual, wherein the analysis of the mental state data is received from a web service and includes analyzing the facial image data; and

rendering an output based on the analysis of the mental state data.

2. The method of claim 1 wherein the collecting mental state data is based on image analysis of the facial image data.

3. The method of claim 1 wherein the facial image data is obtained from a series of images of the individual.

4. The method of claim 3 further comprising identifying a second face from a second individual within the series of images.

5. The method of claim 4 further comprising tracking a face for the individual within the series of images.

6. The method of claim 5 further comprising tracking the second face within the series of images.

7. The method of claim 1 further comprising collecting other mental state data from the individual on a continuous basis.

8. The method of claim 7 wherein the other mental state data includes electrodermal activity data.

9. The method of claim 7 wherein the other mental state data includes audio voice data.

10. The method of claim 1 wherein the mental state data of an individual on an intermittent basis includes audio voice data.

11. The method of claim 1 wherein the obtaining analysis of the mental state data is based on sending a request to a web service for the analysis.

12. The method of claim 1 further comprising interpolating mental state analysis in between the collecting which is intermittent.

13. The method of claim 1 wherein the facial image data is collected intermittently when the individual is looking in a direction of a camera.

14. The method of claim 13 further comprising performing face detection to determine when the individual is looking in the direction of the camera.

15. The method of claim 14 wherein the face detection is based on image classifiers.

16. The method of claim 1 further comprising comparing the mental state data against a plurality of mental state event temporal signatures.

17. The method of claim 1 further comprising filtering out faces of one or more other people to determine when the individual is looking in a direction of a camera.

18. The method of claim 1 further comprising determining contextual information.

19. The method of claim 18 wherein the contextual information is based on one or more of skin temperature or accelerometer data.

20. The method of claim 18 wherein the contextual information is based on one or more of a photograph, an email, a text message, a phone log, or GPS information.

21. The method of claim 1 further comprising translating the mental state data into an emoji for representation of the individual.

22. The method of claim 1 further comprising interpolating mental state analysis in between the collecting which is intermittent;

collecting other mental state data, including electrodermal activity data, from the individual on a continuous basis;

imputing additional mental state data where the mental state data is missing;

filtering out faces of one or more other people to determine when an individual is looking in a direction of a camera;

determining contextual information based on accelerometer data; and

sending a request to a web service for the analysis where the analysis of the mental state data is received from a web service.

23. The method of claim 1 further comprising sending one or more of the mental state data, a subset of the mental state data, or an initial analysis of the mental state data to the web service.

24. The method of claim 23 wherein the sending is accomplished on a periodic basis.

25. The method of claim 1 further comprising inferring mental states based on the mental state data which was collected.

28. The method of claim 1 wherein the other individuals are geographically nearby the individual.

29. The method of claim 1 wherein the other individuals are connected with the individual on a computerized social network.

26. A computer program product embodied in a non-transitory computer readable medium for mental state analysis, the computer program product comprising code which causes one or more processors to perform operations of:

collecting mental state data of an individual on an intermittent basis wherein the mental state data includes facial image data and the facial image data is collected intermittently across a plurality of devices;

imputing additional mental state data where the mental state data is missing, wherein the imputing is based on mental state data collected from other individuals associated with the individual;

interpolating mental state data in between the collecting which is intermittent;

obtaining analysis of the mental state data on the individual, wherein the analysis of the mental state data is received from a web service and includes analyzing the facial image data; and

rendering an output based on the analysis of the mental state data.

27. A system for mental state analysis comprising:

a memory for storing instructions;

one or more processors attached to the memory wherein the one or more processors are configured to;

collect mental state data of an individual on an intermittent basis wherein the mental state data includes facial image data and the facial image data is collected intermittently across a plurality of devices;

impute additional mental state data where the mental state data is missing, wherein the additional mental state data is based on mental state data collected from other individuals associated with the individual;

interpolate mental state data in between the collecting which is intermittent;

obtain analysis of the mental state data on the individual, wherein the analysis of the mental state data is received from a web service and includes analyzing the facial image data; and

render an output based on the analysis of the mental state data.