Mental state analysis using blink rate

• US 9,723,992 B2
• Filed: 03/15/2014
• Issued: 08/08/2017
• Est. Priority Date: 06/07/2010
• Status: Active Grant

• Current Assignee: Affectiva, Inc. (Smart Eye AB)
• Original Assignee: Affectiva, Inc. (Smart Eye AB)
• Inventors: Senechal, Thibaud, el Kaliouby, Rana, Haering, Niels
• Primary Examiner(s): TUCKER, WESLEY J
• Application Number:
  US14/214,918
• Publication Number:
  US 20140200417A1
• Time in Patent Office: 1,242 Days
• Field of Search: 382118
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/02055 Simultaneously evaluating both cardiovascular condition and temperature
• CPC Class Codes: A61B 5/02405 Determining heart rate variability
• CPC Class Codes: A61B 5/08 Detecting, measuring or recording devices for evaluating the respiratory organs A61B5/0205 takes precedence
• CPC Class Codes: A61B 5/1114 Tracking parts of the body
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/441 Skin evaluation, e.g. for skin disorder diagnosis
• CPC Class Codes: G06Q 30/0271 Personalized advertisement
• CPC Class Codes: G16H 20/70 relating to mental therapies, e.g. psychological therapy or autogenous training
• CPC Class Codes: G16H 40/67 for remote operation
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

Mental state analysis is performed by obtaining video of an individual as the individual interacts with a computer, either by performing various operations or by consuming a media presentation. The video is analyzed to determine eye-blink information on the individual, such as eye-blink rate or eye-blink duration. A mental state of the individual is then inferred based on the eye blink information. The blink-rate information and associated mental states can be used to modify an advertisement, a media presentation, or a digital game.

Claims

1. A computer-implemented method for mental state analysis comprising:

obtaining video of an individual;

analyzing, using one or more processors, the video to detect a blink event based on a classifier for a blink that was determined wherein the blink event is determined by;

locating a portion of a face with eyes;

performing temporal analysis on the portion of the face; and

identifying that the eyes are closed for a frame in the video using the temporal analysis;

evaluating a blink duration of the individual for the blink event;

determining blink-rate information using the blink event and one or more other blink events;

compensating the blink-rate information for a context;

evaluating blinking for a group of people of which the individual is a part;

evaluating an average blink duration for the group of people;

determining a difference in blinking between the individual and a remainder of the group; and

inferring mental states of the individual for the blink event, wherein the mental states are based on the blink event, the blink duration of the individual, the average blink duration for the group of people, and the blink-rate information that was compensated.

2. The method of claim 1 wherein the analyzing filters out single eye winks.

3. The method of claim 1 wherein the analyzing filters out looking down by the individual.

4. The method of claim 1 wherein the context includes one or more of screen brightness, environment brightness, gender, and demographics.

5. The method of claim 1 further comprising aggregating the blink-rate information for the individual with blink-rate information for a plurality of other people.

6. The method of claim 1 further comprising correlating the blink-rate information with activities performed by the individual.

7. The method of claim 1 wherein the blink-rate information is correlated to a stimulus that the individual is encountering.

8. The method of claim 1 further comprising determining the classifier for a blink.

9. The method of claim 1 wherein the video is obtained from multiple sources.

10. The method of claim 9 wherein at least one of the multiple sources is a mobile device.

11. The method of claim 1 wherein the video is collected sporadically.

12. The method of claim 1 further comprising determining context for the individual.

13. The method of claim 1 wherein the inferring is performed by a web service.

14. The method of claim 1 wherein the inferring of mental states includes one or more of attention, concentration, boredom, or fatigue.

15. The method of claim 1 wherein the inferring of mental states includes one or more mental states of frustration, confusion, disappointment, hesitation, cognitive overload, focusing, engagement, exploration, confidence, trust, delight, disgust, skepticism, doubt, satisfaction, excitement, laughter, calmness, stress, anger, happiness, and curiosity.

16. The method of claim 1 wherein a higher blink rate indicates a mental state of being focused.

17. The method of claim 1 wherein the inferring includes evaluation of an impaired state.

18. The method of claim 1 further comprising analyzing the video for facial data or physiological data.

19. The method of claim 18 wherein the physiological data includes one or more of heart rate, heart rate variability, blink rate, skin temperature, and respiration.

20. The method of claim 1 further comprising collecting physiological data or accelerometer data with a biosensor.

21. The method of claim 20 wherein the physiological data includes one or more of heart rate, heart rate variability, electrodermal activity, skin temperature, and respiration.

22. The method of claim 1 further comprising correlating the mental states, which were inferred, to a context for the individual.

23. The method of claim 1 wherein the mental states, which were inferred, are used for advertisement selection.

24. The method of claim 1 wherein the mental states, which were inferred, are used to modify a game.

25. The method of claim 1 wherein the mental states, which were inferred, are used to modify a media presentation.

28. The method of claim 1 further comprising training a static classifier to determine when the eyes blink.

29. The method of claim 28 wherein the static classifier identifies that the eyes are open.

30. The method of claim 28 wherein the static classifier identifies that the eyes are closed.

31. The method of claim 1 further comprising evaluating synchronicity of blinking for the group.

32. The method of claim 1 wherein the difference is used to determine a mental state for the individual.

33. The method of claim 32 wherein the mental state includes lacking attention.

34. The method of claim 1 wherein eye closures, which are longer than blinks, are filtered.

35. The method of claim 28 wherein the training is done offline using a plurality of images or videos.

36. The method of claim 28 wherein the classifier is further configured to do filtering and differentiation.

37. The method of claim 36 wherein the filtering and differentiation is for winks, looking down, or eye closures.

38. The method of claim 1 further comprising downloading the classifier from a database for use in the blink analysis.

26. A computer program product embodied in a non-transitory computer readable medium for mental state analysis, the computer program product comprising code which causes one or more processors to perform operations of:

obtaining video of an individual;

analyzing the video to determine a blink event based on a classifier for a blink that was determined wherein the blink event is determined by;

locating a portion of a face with eyes;

performing temporal analysis on the portion of the face; and

identifying that the eyes are closed for a frame in the video using the temporal analysis;

evaluating a blink duration of the individual for the blink event;

determining blink-rate information using the blink event and one or more other blink events;

compensating the blink-rate information for a context;

evaluating blinking for a group of people of which the individual is a part;

evaluating an average blink duration for the group of people;

determining a difference in blinking between the individual and a remainder of the group; and

inferring mental states of the individual for the blink event, wherein the mental states are based on the blink event, the blink duration of the individual, the average blink duration for the group of people, and the blink-rate information that was compensated.

27. A computer system for mental state analysis comprising:

a memory which stores instructions;

one or more processors attached to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to;

obtain video of an individual;

analyze the video to determine a blink event based on a classifier for a blink that was determined wherein the blink event is determined by;

locating a portion of a face with eyes;

performing temporal analysis on the portion of the face; and

identifying that the eyes are closed for a frame in the video using the temporal analysis;

evaluate a blink duration of the individual for the blink event;

determine blink-rate information using the blink event and one or more other blink events;

compensate the blink-rate information for a context;

evaluate blinking for a group of people of which the individual is a part;

evaluate an average blink duration for the group of people;

determine a difference in blinking between the individual and a remainder of the group; and

infer mental states of the individual for the blink event, wherein the mental states are based on the blink event, the blink duration of the individual, the average blink duration for the group of people, and the blink-rate information that was compensated.