Mental state mood analysis using heart rate collection based on video imagery

• US 10,517,521 B2
• Filed: 05/08/2017
• Issued: 12/31/2019
• Est. Priority Date: 06/07/2010
• Status: Active Grant

• Current Assignee: Affectiva, Inc. (Smart Eye AB)
• Original Assignee: Affectiva, Inc. (Smart Eye AB)
• Inventors: el Kaliouby, Rana, Bhatkar, Viprali, Haering, Niels, Kashef, Youssef, Osman, Ahmed Adel
• Primary Examiner(s): Nakhjavan, Shervin K
• Application Number:
  US15/589,959
• Publication Number:
  US 20170238860A1
• Time in Patent Office: 967 Days
• Field of Search: None
• CPC Class Codes: A61B 2562/0233 Special features of optical sensors or probes classified in A61B5/00
• CPC Class Codes: A61B 2576/00 Medical imaging apparatus involving image processing or analysis A61B1/00009, A61B6/52 and A61B8/52 take precedence; image processing per se G06T; details of algorithms for analysing medical images G06T7/0012
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/02055 Simultaneously evaluating both cardiovascular condition and temperature
• CPC Class Codes: A61B 5/02405 Determining heart rate variability
• CPC Class Codes: A61B 5/02416 using photoplethysmograph signals, e.g. generated by infrared radiation A61B5/14552 takes precedence
• CPC Class Codes: A61B 5/0261 using optical means, e.g. infrared light
• CPC Class Codes: A61B 5/0295 using plethysmography, i.e. measuring the variations in the volume of a body part as modified by the circulation of blood therethrough, e.g. impedance plethysmography
• CPC Class Codes: A61B 5/0531 Measuring skin impedance
• CPC Class Codes: A61B 5/08 Detecting, measuring or recording devices for evaluating the respiratory organs A61B5/0205 takes precedence
• CPC Class Codes: A61B 5/1114 Tracking parts of the body
• CPC Class Codes: A61B 5/1176 Recognition of faces
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/441 Skin evaluation, e.g. for skin disorder diagnosis
• CPC Class Codes: A61B 5/7246 using correlation, e.g. template matching or determination of similarity
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: A61B 5/7278 Artificial waveform generation or derivation, e.g. synthesising signals from measured signals A61B5/327 takes precedence
• CPC Class Codes: G06Q 30/0242 Determining effectiveness of advertisements
• CPC Class Codes: G06Q 30/0271 Personalized advertisement
• CPC Class Codes: G16H 20/70 relating to mental therapies, e.g. psychological therapy or autogenous training
• CPC Class Codes: G16H 30/20 : for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 : for processing medical images, e.g. editing
• CPC Class Codes: G16H 50/20 : for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: G16H 50/30 : for calculating health indices; for individual health risk assessment

Abstract

Video of one or more people is obtained and analyzed. Heart rate information is determined from the video. The heart rate information is used in mental state analysis. The heart rate information and resulting mental state analysis are correlated to stimuli, such as digital media, which is consumed or with which a person interacts. The heart rate information is used to infer mental states. The inferred mental states are used to output a mood measurement. The mental state analysis, based on the heart rate information, is used to optimize digital media or modify a digital game. Training is employed in the analysis. Machine learning is engaged to facilitate the training.

Claims

1. A computer-implemented method for mental state analysis comprising:

obtaining video of an individual;

analyzing the video to determine heart rate information, wherein the analyzing includes;

identifying a face of the individual in a portion of the video;

separating pixels from the video of the individual, into at least a green pixel temporal intensity trace;

training a statistical classifier, wherein the training is learned from a data set consisting of human blood volume pulse synchronized face videos; and

recognizing a pulse, from the video of the individual, using the statistical classifier, by learning patterns of variability in the mean of the pixel temporal intensity trace;

correlating the heart rate information to a stimulus that the individual is encountering; and

inferring, using one or more processors, mental states of the individual based on the heart rate information.

2. The method of claim 1 further comprising outputting a mood measurement based on the mental states which were inferred.

3. The method of claim 1 further comprising analyzing a media presentation based on the mental states which were inferred.

4. The method of claim 3 wherein the analyzing a media presentation is further based on learning about heart rate information.

5. The method of claim 4 wherein the learning further comprises factoring in one or more previous frames of data of the video.

6. The method of claim 5 further comprising capturing heart rate signal fluctuations in the video due to blood flow.

7. The method of claim 1 further comprising aggregating the heart rate information for the individual with other people.

8. The method of claim 1 further comprising aggregating the mental states for the individual with other people.

9. The method of claim 1 wherein learning about heart rate information is included as part of the analyzing.

10. The method of claim 1 wherein the inferring includes determining arousal, attention, or valence.

11. The method of claim 1 wherein the analyzing includes calculating blood volume pulse.

12. The method of claim 1 wherein the analyzing factors in an occlusion of part of a face for the individual.

13. The method of claim 1 further comprising determining contextual information.

14. The method of claim 1 further comprising evaluating a temporal signature for the mental states.

15. The method of claim 14 further comprising using the temporal signature to infer additional mental states.

16. The method of claim 1 wherein the inferring mental states further comprises analyzing an emotional mood of the individual.

17. The method of claim 16 wherein the analyzing the emotional mood of the individual is used to enable the individual to track emotional health.

18. The method of claim 1 further comprising performing unsupervised learning, as part of the training, for pulse recognition.

19. The method of claim 1 further comprising learning image descriptors, as part of the training, for emotional content.

20. The method of claim 1 further comprising establishing a region of interest including the face, separating pixels in the region of interest into at least two channel values and combining to form raw traces, transforming and decomposing the raw traces into at least one independent source signal, and processing the at least one independent source signal to obtain the heart rate information.

21. The method of claim 20 wherein the heart rate information includes heart rate and the heart rate is determined based on changes in an amount of reflected light.

22. The method of claim 1 wherein the video includes a plurality of other people.

23. The method of claim 22 further comprising identifying faces of the plurality of other people and analyzing the video to determine heart rate information on the plurality of other people.

24. The method of claim 23 further comprising inferring mental states of the plurality of other people based on the heart rate information on the plurality of other people.

25. The method of claim 1 further comprising obtaining biosensor data for the individual.

26. The method of claim 25 wherein the biosensor data augments the heart rate information.

27. The method of claim 25 wherein the biosensor data includes one or more of electrodermal activity, heart rate, heart rate variability, skin temperature, or respiration.

28. A computer program product embodied in a non-transitory computer readable medium for mental state analysis, the computer program product comprising code which causes one or more processors to perform operations of:

obtaining video of an individual;

analyzing the video to determine heart rate information, wherein the analyzing includes;

identifying a face of the individual in a portion of the video;

separating pixels from the video of the individual, into at least a green pixel temporal intensity trace;

training a statistical classifier, wherein the training is learned from a data set consisting of human blood volume pulse synchronized face videos; and

recognizing a pulse, from the video of the individual, using the statistical classifier, by learning patterns of variability in the mean of the pixel temporal intensity trace;

correlating the heart rate information to a stimulus that the individual is encountering; and

inferring mental states of the individual based on the heart rate information.

29. A computer system for mental state analysis comprising:

a memory which stores instructions;

one or more processors attached to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to;

obtain video of an individual;

analyze the video to determine heart rate information, wherein analyzing includes;

identifying a face of the individual in a portion of the video;

separating pixels from the video of the individual into at least a green pixel temporal intensity trace;

training a statistical classifier, wherein the training is learned from a data set consisting of human blood volume pulse synchronized face videos; and

recognizing a pulse, from the video of the individual, using the statistical classifier, by learning patterns of variability in the mean of the pixel temporal intensity trace;

correlate the heart rate information to a stimulus that the individual is encountering; and

infer mental states of the individual based on the heart rate information.