Mental state data tagging for data collected from multiple sources

• US 9,646,046 B2
• Filed: 03/15/2014
• Issued: 05/09/2017
• Est. Priority Date: 06/07/2010
• Status: Active Grant

• Current Assignee: Affectiva, Inc. (Smart Eye AB)
• Original Assignee: Affectiva, Inc. (Smart Eye AB)
• Inventors: Sadowsky, Richard Scott, el Kaliouby, Rana
• Primary Examiner(s): TUCKER, WESLEY J
• Application Number:
  US14/214,704
• Publication Number:
  US 20140201207A1
• Time in Patent Office: 1,151 Days
• CPC Class Codes: A61B 5/02055 Simultaneously evaluating both cardiovascular condition and temperature
• CPC Class Codes: A61B 5/02405 Determining heart rate variability
• CPC Class Codes: A61B 5/0533 Measuring galvanic skin response
• CPC Class Codes: A61B 5/08 Detecting, measuring or recording devices for evaluating the respiratory organs A61B5/0205 takes precedence
• CPC Class Codes: A61B 5/11 Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb for measuring pulse A61B5/02 ;A61B5/1038 takes precedence; motion detection to correct for motion artifacts in physiological signals A61B5/721
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: G06F 16/24 Querying
• CPC Class Codes: G06Q 30/0271 Personalized advertisement
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G16H 20/70 relating to mental therapies, e.g. psychological therapy or autogenous training
• CPC Class Codes: G16H 40/67 for remote operation
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

Mental state data useful for determining mental state information on an individual, such as video of an individual'"'"'s face, is captured. Additional data that is helpful in determining the mental state information, such as contextual information, is also determined. The data and additional data allows interpretation of individual mental state information. The additional data is tagged to the mental state data and at least some of the mental state data, along with the tagged data, can be sent to a web service where it is used to produce further mental state information.

Claims

1. A computer-implemented method for mental state analysis comprising:

receiving two or more portions of mental state data tagged with additional information, wherein the two or more portions of mental state data come from a plurality of sources of facial data, wherein the mental state data collected is intermittent, and wherein the plurality of sources includes at least one mobile device, including a cell phone, a tablet computer, a wearable device, or a laptop computer;

interpolating the intermittent mental state data, wherein the interpolating is based on the additional information that was tagged;

selecting one or more portions of the received two or more portions of mental state data based on the additional information that was tagged, wherein the one or more selected portions of mental state data are selected based, at least in part, on tags identifying a particular context; and

analyzing, using one or more processors, the one or more selected portions of mental state data to generate mental state information wherein a result from the analyzing is used to render an output on mental states.

2. The method of claim 1 wherein the plurality of sources of facial data include one or more of a webcam, a phone camera, a tablet camera, or a wearable camera.

3. The method of claim 1 wherein the one or more selected portions of mental state data are selected based, at least in part, on tags identifying a particular individual.

4. The method of claim 3 wherein the one or more selected portions of mental state data are selected based, at least in part, on tags identifying one or more contexts.

5. The method of claim 3 wherein the one or more selected portions of mental state data are selected to include tags identifying at least two different timestamps.

6. The method of claim 1 further comprising the output used for rendering being sent to another computer and the other computer provides the rendering.

25. The method of claim 1 further comprising imputing additional mental state data where the mental state data is missing.

26. The method of claim 1 further comprising associating interpolated data with the mental state data that is collected on an intermittent basis.

27. The method of claim 1 where the mental state data is intermittent due to an image collection being lost.

28. The method of claim 1 wherein the sources of facial data include a wearable camera which includes glasses worn by an observer.

29. The method of claim 1 wherein mental state data is collected from multiple devices while a user is performing a task using an electronic display during a portion of time.

30. The method of claim 29 wherein the multiple devices include a tablet computer and a cell phone.

31. The method of claim 1 wherein contextual data is collected simultaneously with the mental state data.

32. The method of claim 31 wherein the contextual data includes location data, environmental information, or time data.

7. A computer-implemented method for mental state analysis comprising:

capturing mental state data on an individual from a first source that includes facial information, wherein the mental state data collected is intermittent;

capturing mental state data on the individual from at least a second source that includes facial data, wherein the at least a second source comprises a mobile device, including a cell phone, a tablet computer, a wearable device, or a laptop computer;

determining additional data about the mental state data wherein the additional data provides information about mental states and wherein the additional data includes information about a context as the mental state data was collected;

tagging the additional data to the mental state data;

interpolating the intermittent mental state data, wherein the interpolating is based on the additional data that was tagged; and

sending at least a portion of the mental state data tagged with the additional data to a web service.

8. The method of claim 7 further comprising analyzing the mental state data to produce mental state information.

9. The method of claim 8, further comprising using the additional data in conjunction with the mental state data to produce the mental state information.

10. The method of claim 7 further comprising obtaining mental state data from a second source.

11. The method of claim 10 wherein the mental state data from the second source includes facial information.

12. The method of claim 11 wherein the mental state data from the second source includes biosensor information.

13. The method of claim 7 wherein the additional data includes information on a source that collected the mental state data.

14. The method of claim 7 wherein the context comprises a time.

15. The method of claim 7 wherein the context comprises a location.

16. The method of claim 15 wherein the location is determined using GPS.

17. The method of claim 7 wherein the context comprises environmental information.

18. The method of claim 7 wherein the context comprises an identity of another person in proximity of the individual.

19. The method of claim 7 wherein the context comprises an activity performed by the individual.

20. The method of claim 7 wherein the additional data includes information about an identity of the individual.

21. The method of claim 20 wherein the information about the identity of the individual is determined using face recognition.

22. The method of claim 7 wherein the portion of the mental state data is determined based on face recognition.

23. The method of claim 7 wherein the additional data includes human coded information on the individual.

33. The method of claim 7 further comprising locating pertinent mental state data based on the tagging.

34. The method of claim 33 wherein the pertinent mental state data is identified for retrieval from a database.

35. The method of claim 33 further comprising sending tagged mental state data over the internet to cloud or web-based storage or web-based services for remote use and using the tags locally on a machine where the mental state data was collected.

24. A computer program product embodied in a non-transitory computer readable medium for mental state analysis, the computer program product comprising code which causes one or more processors to perform operations of:

capturing mental state data on an individual from a first source that includes facial information, wherein the mental state data collected is intermittent;

capturing mental state data on the individual from at least a second source that includes facial data, wherein the at least a second source comprises a mobile device, including a cell phone, a tablet computer, a wearable device, or a laptop computer;

determining additional data about the mental state data wherein the additional data provides information about mental states and wherein the additional data includes information about a context as the mental state data was collected;

tagging the additional data to the mental state data;

interpolating the intermittent mental state data, wherein the interpolating is based on the additional data that was tagged; and

sending at least a portion of the mental state data tagged with the additional data to a web service.