MENTAL STATE DATA TAGGING FOR DATA COLLECTED FROM MULTIPLE SOURCES

• US 20140201207A1
• Filed: 03/15/2014
• Published: 07/17/2014
• Est. Priority Date: 06/07/2010
• Status: Active Grant

• Current Assignee: Affectiva, Inc. (Smart Eye AB)
• Original Assignee: Affectiva, Inc. (Smart Eye AB)
• Inventors: Sadowsky, Richard Scott, el Kaliouby, Rana
• Granted Patent: US 9,646,046 B2
• Time in Patent Office: Days
• US Class Current: 707/736
• CPC Class Codes: A61B 5/02055 Simultaneously evaluating both cardiovascular condition and temperature
• CPC Class Codes: A61B 5/02405 Determining heart rate variability
• CPC Class Codes: A61B 5/0533 Measuring galvanic skin response
• CPC Class Codes: A61B 5/08 Detecting, measuring or recording devices for evaluating the respiratory organs A61B5/0205 takes precedence
• CPC Class Codes: A61B 5/11 Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb for measuring pulse A61B5/02 ;A61B5/1038 takes precedence; motion detection to correct for motion artifacts in physiological signals A61B5/721
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: G06F 16/24 Querying
• CPC Class Codes: G06Q 30/0271 Personalized advertisement
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G16H 20/70 relating to mental therapies, e.g. psychological therapy or autogenous training
• CPC Class Codes: G16H 40/67 for remote operation
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

Mental state data useful for determining mental state information on an individual, such as video of an individual'"'"'s face, is captured. Additional data that is helpful in determining the mental state information, such as contextual information, is also determined. The data and additional data allows interpretation of individual mental state information. The additional data is tagged to the mental state data and at least some of the mental state data, along with the tagged data, can be sent to a web service where it is used to produce further mental state information.

Claims

1. A computer-implemented method for mental state analysis comprising:

receiving two or more portions of mental state data tagged with additional information;

selecting one or more portions of the received two or more portions of mental state data based on the additional information that was tagged; and

analyzing the one or more selected portions of mental state data to generate mental state information wherein a result from the analyzing is used to render an output on mental states.

2. The method of claim 1 wherein the received two or more portions come from a plurality of sources of facial data.

3. The method of claim 2 wherein the plurality of sources of facial data include one or more of a webcam, a phone camera, a tablet camera, a wearable camera.

4. The method of claim 2 wherein the plurality of sources includes at least one mobile device.

5. The method of claim 4 wherein the at least one mobile device includes a forward facing camera.

6. The method of claim 1 wherein the one or more selected portions of mental state data are selected based, at least in part, on tags identifying a particular individual.

7. The method of claim 6 wherein the one or more selected portions of mental state data are selected based, at least in part, on tags identifying one or more contexts.

8. The method of claim 6 wherein the one or more selected portions of mental state data are selected to include tags identifying at least two different timestamps.

9. The method of claim 1 wherein the one or more selected portions of mental state data are selected based, at least in part, on tags identifying a particular context.

12. The method of claim 1 further comprising the output used for rendering being sent to another computer and the other computer provides the rendering.

10-11. -11. (canceled)

13. (canceled)

14. A computer-implemented method for mental state analysis comprising:

capturing mental state data on an individual from a first source;

determining additional data about the mental state data wherein the additional data provides information about mental states;

tagging the additional data to the mental state data; and

sending at least a portion of the mental state data tagged with the additional data to a web service.

15. The method of claim 14 wherein the mental state data from the first source includes facial information.

16. The method of claim 14 further comprising analyzing the mental state data to produce mental state information.

17. The method of claim 16, further comprising using the additional data in conjunction with the mental state data to produce the mental state information.

18. The method of claim 14 further comprising obtaining mental state data from a second source.

19. The method of claim 18 wherein the mental state data from the second source includes facial information.

20. The method of claim 19 wherein the mental state data from the second source includes biosensor information.

21. The method of claim 14 wherein the additional data includes information on a source that collected the mental state data.

22. The method of claim 14 wherein the additional data includes information about a context as the mental state data was collected.

23. The method of claim 22 wherein the context comprises a time.

24. The method of claim 22 wherein the context comprises a location.

25. The method of claim 24 wherein the location is determined using GPS.

28. The method of claim 22 wherein the context comprises environmental information.

30. The method of claim 22 wherein the context comprises an identity of another person in proximity of the individual.

31. The method of claim 22 wherein the context comprises an activity performed by the individual.

33. The method of claim 14 wherein the additional data includes information about an identity of the individual.

36. The method of claim 33 wherein the information about the identity of the individual is determined using face recognition.

38. The method of claim 14 wherein the portion of the mental state data is determined based on face recognition.

39. The method of claim 14 wherein the additional data includes human coded information on the individual.

26-27. -27. (canceled)

29. (canceled)

32. (canceled)

34-35. -35. (canceled)

37. (canceled)

40-41. -41. (canceled)

42. A computer program product embodied in a non-transitory computer readable medium for mental state analysis, the computer program product comprising:

code for capturing mental state data on an individual from a first source;

code for determining additional data about the mental state data wherein the additional data provides information about mental states;

code for tagging the additional data to the mental state data; and

code for sending at least a portion of the mental state data tagged with the additional data to a web service.

43. (canceled)