Mental state analysis using blink rate for vehicles

• US 10,074,024 B2
• Filed: 08/07/2017
• Issued: 09/11/2018
• Est. Priority Date: 06/07/2010
• Status: Active Grant

• Current Assignee: Affectiva, Inc. (Smart Eye AB)
• Original Assignee: Affectiva, Inc. (Smart Eye AB)
• Inventors: el Kaliouby, Rana, Mahmoud, Abdelrahman N., Mavadati, Seyedmohammad, Turcot, Panu James
• Primary Examiner(s): Wu, Zhen Y
• Application Number:
  US15/670,791
• Publication Number:
  US 20170337438A1
• Time in Patent Office: 400 Days
• Field of Search: 340439
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/02055 Simultaneously evaluating both cardiovascular condition and temperature
• CPC Class Codes: A61B 5/1103 Detecting eye twinkling
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/18 for vehicle drivers or machine operators
• CPC Class Codes: A61B 5/6893 Cars
• CPC Class Codes: B60K 28/06 responsive to incapacity of driver
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: G06Q 30/0271 Personalized advertisement
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30268 Vehicle interior
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G16H 20/70 relating to mental therapies, e.g. psychological therapy or autogenous training
• CPC Class Codes: G16H 40/67 for remote operation
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: G16Z 99/00 Subject matter not provided for in other main groups of this subclass

Abstract

Mental state analysis is performed by obtaining video of an individual as the individual interacts with a computer, either by performing various operations, such as driving a vehicle or being a passenger in a vehicle, or by consuming a media presentation. The video is analyzed to determine eye-blink information on the individual, such as eye-blink rate or eye-blink duration. The blink-rate information is compensated for a context. Blinking for a group of people of which the individual is a part is evaluated. Mental states of the individual are inferred for the blink event based on the blink event, the blink duration of the individual, the difference in blinking between the individual and the remainder of the group, and the blink-rate information that was compensated. The blink-rate information and associated mental states can be used to modify an advertisement, a media presentation, a digital game, or vehicle controls.

Claims

1. A computer-implemented method for mental state analysis comprising:

obtaining video of an individual with an image capture device;

analyzing, using one or more processors, the video to detect a blink event based on a classifier for a blink that was determined wherein the blink event is determined by identifying that eyes of the individual are closed for a frame in the video using temporal analysis;

evaluating, using the one or more processors, a blink duration of the individual for the blink event;

determining, using the one or more processors, blink-rate information using the blink event and one or more other blink events;

compensating, using the one or more processors, the blink-rate information for a context;

evaluating blinking for a group of people of which the individual is a part;

determining a difference in blinking between the individual and a remainder of the group; and

inferring, using the one or more processors, mental states of the individual for the blink event, wherein the mental states are based on the blink event, the blink duration of the individual, the difference in blinking between the individual and the remainder of the group, and the blink-rate information that was compensated.

2. The method of claim 1 wherein the video of the individual is obtained from within a vehicle.

3. The method of claim 2 wherein the individual is a passenger in the vehicle.

4. The method of claim 2 wherein the individual is a driver of the vehicle.

5. The method of claim 2 further comprising manipulating the vehicle based on the mental states that were inferred.

6. The method of claim 5 wherein the manipulating the vehicle includes recommending action.

7. The method of claim 5 wherein the manipulating the vehicle includes initiating a locking out operation, recommending a break for an occupant, recommending a different route, recommending how far to drive, controlling the vehicle in response to traffic, adjusting seats, adjusting mirrors, adjusting climate control, adjusting lighting, adjusting music, generating audio stimuli, activating a braking system, or activating steering control.

8. The method of claim 1 wherein the mental states that were inferred indicate one or more of attention, concentration, boredom, fatigue, or cognitive load for the individual.

9. The method of claim 1 wherein the mental states that were inferred indicate drowsiness for the individual.

10. The method of claim 1 further comprising locating a portion of a face with eyes.

11. The method of claim 10 further comprising performing temporal analysis on the portion of the face.

12. The method of claim 11 wherein the identifying that the eyes are closed is based on the performing temporal analysis on the portion of the face.

13. The method of claim 1 further comprising determining a difference in blinking by the individual and typical blinking for the individual.

14. The method of claim 13 wherein the inferring mental states of the individual is further based on the difference in blinking by the individual and typical blinking for the individual.

15. The method of claim 1 wherein the analyzing filters out single eye winks.

16. The method of claim 1 wherein the analyzing filters out looking down by the individual.

17. The method of claim 1 further comprising evaluating average blink duration.

18. The method of claim 1 further comprising aggregating the blink-rate information for the individual with blink-rate information for a plurality of other people.

19. The method of claim 1 further comprising correlating the blink-rate information with activities performed by the individual.

20. The method of claim 1 wherein the blink-rate information is correlated to a stimulus that the individual is encountering.

21. The method of claim 1 wherein the video is obtained from multiple sources.

22. The method of claim 21 wherein at least one of the multiple sources is a mobile device.

23. The method of claim 1 wherein the video is collected sporadically.

24. The method of claim 1 wherein the inferring includes evaluation of an impaired state.

25. The method of claim 1 further comprising collecting physiological data or accelerometer data with a biosensor.

26. The method of claim 1 further comprising correlating the mental states, which were inferred, to a context for the individual.

27. The method of claim 1 wherein the mental states, which were inferred, are used to modify a media presentation.

28. A computer program product embodied in a non-transitory computer readable medium for mental state analysis, the computer program product comprising code which causes one or more processors to perform operations of:

obtaining video of an individual with an image capture device;

analyzing, using one or more processors, the video to detect a blink event based on a classifier for a blink that was determined wherein the blink event is determined by identifying that eyes of the individual are closed for a frame in the video using temporal analysis;

evaluating, using the one or more processors, a blink duration of the individual for the blink event;

determining, using the one or more processors, blink-rate information using the blink event and one or more other blink events;

compensating, using the one or more processors, the blink-rate information for a context;

evaluating blinking for a group of people of which the individual is a part;

determining a difference in blinking between the individual and a remainder of the group; and

inferring, using the one or more processors, mental states of the individual for the blink event, wherein the mental states are based on the blink event, the blink duration of the individual, the difference in blinking between the individual and the remainder of the group, and the blink-rate information that was compensated.

29. A computer system for mental state analysis comprising:

a memory which stores instructions;

one or more processors attached to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to;

obtain video of an individual with an image capture device;

analyze the video to detect a blink event based on a classifier for a blink that was determined wherein the blink event is determined by identifying that eyes of the individual are closed for a frame in the video using temporal analysis;

evaluate a blink duration of the individual for the blink event;

determine blink-rate information using the blink event and one or more other blink events;

compensate the blink-rate information for a context;

evaluate blinking for a group of people of which the individual is a part;

determine a difference in blinking between the individual and a remainder of the group; and

infer mental states of the individual for the blink event, wherein the mental states are based on the blink event, the blink duration of the individual, the difference in blinking between the individual and the remainder of the group, and the blink-rate information that was compensated.