MENTAL STATE ANALYSIS USING HEART RATE COLLECTION BASED ON VIDEO IMAGERY

• US 20140200416A1
• Filed: 03/15/2014
• Published: 07/17/2014
• Est. Priority Date: 06/07/2010
• Status: Active Grant

• Current Assignee: Affectiva, Inc. (Smart Eye AB)
• Original Assignee: Affectiva, Inc. (Smart Eye AB)
• Inventors: el Kaliouby, Rana, Haering, Niels, Kashef, Youssef, Osman, Ahmed Adel, Bhatkar, Viprali
• Granted Patent: US 9,642,536 B2
• Time in Patent Office: Days
• US Class Current: 600/301
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/02055 Simultaneously evaluating both cardiovascular condition and temperature
• CPC Class Codes: A61B 5/02405 Determining heart rate variability
• CPC Class Codes: A61B 5/1114 Tracking parts of the body
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/441 Skin evaluation, e.g. for skin disorder diagnosis
• CPC Class Codes: G06Q 30/0242 Determining effectiveness of advertisements
• CPC Class Codes: G06Q 30/0271 Personalized advertisement
• CPC Class Codes: G16H 20/70 relating to mental therapies, e.g. psychological therapy or autogenous training
• CPC Class Codes: G16H 40/67 for remote operation
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

Video of one or more people is obtained and analyzed. Heart rate information is determined from the video and the heart rate information is used in mental state analysis. The heart rate information and resulting mental state analysis are correlated to stimuli, such as digital media which is consumed or with which a person interacts. The heart rate information is used to infer mental states. The mental state analysis, based on the heart rate information, can be used to optimize digital media or modify a digital game.

Claims

1. A computer-implemented method for mental state analysis comprising:

obtaining video of an individual;

analyzing the video to determine heart rate information; and

inferring mental states of the individual based on the heart rate information.

2. The method of claim 1 further comprising analyzing a media presentation based on the mental states, which were inferred.

3. The method of claim 2 wherein the analyzing the media presentation includes evaluating advertisement effectiveness.

4. The method of claim 2 wherein the analyzing the media presentation includes optimizing the media presentation.

6. The method of claim 1 wherein the heart rate information is correlated to a stimulus that the individual is encountering.

7. The method of claim 6 wherein the stimulus pertains to a media presentation or is based on a game.

8. The method of claim 7 wherein the game is modified based on the heart rate information.

10. The method of claim 1 wherein the analysis includes evaluation of phasic and tonic heart rate responses.

11. The method of claim 1 further comprising aggregating the heart rate information for the individual with other people.

12. The method of claim 1 further comprising aggregating the mental states for the individual with other people.

13. The method of claim 1 wherein learning about heart rate information is included as part of the analyzing.

14. The method of claim 1 wherein the inferring includes determining arousal, attention, or valence.

15. The method of claim 1 wherein the analyzing includes calculating blood volume pulse.

16. The method of claim 1 wherein the inferring factors in a time lag between a stimulus and the heart rate information.

17. The method of claim 1 wherein the analyzing factors in an occlusion of part of a face for the individual.

19. The method of claim 1 further comprising determining contextual information.

24. The method of claim 1 wherein the analyzing includes identifying a location of a face of the individual in a portion of the video.

25. The method of claim 24 further comprising establishing a region of interest including the face, separating pixels in the region of interest into at least two channel values and combining to form raw traces, transforming and decomposing the raw traces into at least one independent source signal, and processing the at least one independent source signal to obtain the heart rate information.

26. The method of claim 25 wherein the heart rate information includes heart rate and the heart rate is determined based on changes in an amount of reflected light.

27. The method of claim 1 wherein the video includes a plurality of other people.

28. The method of claim 27 further comprising identifying locations for faces of the plurality of other people and analyzing the video to determine heart rate information on the plurality of other people.

29. The method of claim 28 further comprising inferring mental states of the plurality of other people based on the heart rate information on the plurality of other people.

30. The method of claim 1 further comprising obtaining biosensor data for the individual.

31. The method of claim 30 wherein the biosensor data augments the heart rate information.

32. The method of claim 30 wherein the biosensor data includes one or more of electrodermal activity, heart rate, heart rate variability, skin temperature, or respiration.

34. The method of claim 1 further comprising collecting facial data based on the video.

35. The method of claim 34 wherein the facial data includes facial movements.

36. The method of claim 35 wherein the inferring is based on the facial data.

37. The method of claim 35 wherein the facial data is used in combination with the heart rate information.

38. The method of claim 1 wherein the mental states include one or more of frustration, confusion, disappointment, hesitation, cognitive overload, focusing, engagement, attention, boredom, exploration, confidence, trust, delight, disgust, skepticism, doubt, satisfaction, excitement, laughter, calmness, stress, sentimentality, and curiosity.

5. (canceled)

9. (canceled)

18. (canceled)

20-23. -23. (canceled)

33. (canceled)

39. A computer program product embodied in a non-transitory computer readable medium for mental state analysis, the computer program product comprising:

code for obtaining video of an individual;

code for analyzing the video to determine heart rate information; and

code for inferring mental states of the individual based on the heart rate information.

40. A computer system for mental state analysis comprising:

a memory which stores instructions;

one or more processors attached to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to;

obtain video of an individual;

analyze the video to determine heart rate information; and

infer mental states of the individual based on the heart rate information.