Advanced vehicle operator intelligence system

• US 9,908,530 B1
• Filed: 08/05/2016
• Issued: 03/06/2018
• Est. Priority Date: 04/17/2014
• Status: Active Grant

• Current Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Original Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Inventors: Fields, Brian Mark, Roberson, Steve, Chan, Aaron Scott, Sanchez, Kenneth Jason, Jon Donovan, Matthew Christopher
• Primary Examiner(s): Bee, Andrew
• Application Number:
  US15/229,926
• Time in Patent Office: 578 Days
• CPC Class Codes: B60K 28/066 actuating a signalling device B60K28/063 takes precedence
• CPC Class Codes: B60W 2040/0818 Inactivity or incapacity of driver
• CPC Class Codes: B60W 2040/0872 Driver physiology
• CPC Class Codes: B60W 2040/089 Driver voice
• CPC Class Codes: B60W 2520/00 Input parameters relating to overall vehicle dynamics
• CPC Class Codes: B60W 2520/105 Longitudinal acceleration
• CPC Class Codes: B60W 2520/125 Lateral acceleration
• CPC Class Codes: B60W 2540/21 Voice
• CPC Class Codes: B60W 2540/22 Psychological state; Stress level or workload
• CPC Class Codes: B60W 2540/221 Physiology, e.g. weight, heartbeat, health or special needs
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2554/801 Lateral distance
• CPC Class Codes: B60W 2555/20 Ambient conditions, e.g. wind or rain
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: B61L 15/009 On-board display devices
• CPC Class Codes: G06Q 40/08 Insurance
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G08B 21/02 Alarms for ensuring the safety of persons
• CPC Class Codes: G08B 21/06 indicating a condition of sleep, e.g. anti-dozing alarms
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/306 : User profiles
• CPC Class Codes: H04W 4/38 : for collecting sensor information
• CPC Class Codes: H04W 4/48 : for in-vehicle communication

Abstract

The method, system, and computer-readable medium cause the monitoring of a vehicle operator during the course of vehicle operation to determine whether the vehicle operator is impaired and causes a mitigating response when an impairment is determined to exist. The vehicle operator, the environment surrounding the vehicle, or forces acting on the vehicle may be monitored using a variety of sensors, including optical sensors, accelerometers, or biometric sensors (e.g., skin conductivity, heart rate, or voice modulation). When the vehicle operator is determined to be impaired, an alert or other mitigating response is implemented, based on the sensor data. In some embodiments, mitigating actions may be taken to avoid vehicle operator impairment. In further embodiments, a training period may be used to generate a profile for the vehicle operator. In yet further embodiments, information regarding multiple types of impairments may be used to determine and implement an appropriate mitigating response.

Claims

1. A computer-implemented method for monitoring a vehicle operator during operation of a vehicle, comprising:

receiving, at one or more processors, sensor data from one or more sensors regarding the vehicle operator, including data related to at least one of the following;

alertness of the vehicle operator, distractedness of the vehicle operator, emotional state of the vehicle operator, or vehicle operator workload due to traffic conditions;

determining, by one or more processors and using the received sensor data, first and second impairment scores associated with first and second vehicle operator states, including at least two of the following;

alertness of the vehicle operator, distractedness of the vehicle operator, anxiety of the vehicle operator, agitation of the vehicle operator, aggression of the vehicle operator, or health of the vehicle operator;

determining, by one or more processors, the vehicle operator is impaired with respect to the first vehicle operator state when the first impairment score exceeds a first threshold value associated with the first impairment score;

determining, by one or more processors, the second impairment score associated with the second vehicle operator state is elevated but does not exceed a second threshold value associated with the second impairment score;

determining, by one or more processors, one or more predicted reactions of the vehicle operator to one or more potential responses based upon the first and second impairment scores;

selecting, by one or more processors, a response to implement based upon the one or more predicted reactions of the vehicle operator to the one or more potential responses; and

causing the selected response to be implemented.

2. The computer-implemented method of claim 1, further comprising:

receiving, at one or more processors, additional sensor data from one or more sensors regarding the vehicle, including data related to at least one of the following;

traffic conditions, environmental conditions, or the movement of the vehicle;

wherein the one or more predicted reactions of the vehicle operator are further determined based upon the received additional sensor data.

3. The computer-implemented method of claim 1, further comprising:

receiving, at one or more processors, a user profile associated with the vehicle operator;

wherein the one or more predicted reactions of the vehicle operator are further determined based upon the user profile.

4. The computer-implemented method of claim 3, wherein the user profile includes information regarding reactions of the vehicle operator to stimuli associated with the one or more potential responses.

5. The computer-implemented method of claim 1, wherein the response is selected based upon a prediction of the ability of the response to mitigate the impairment of the vehicle operator.

6. The computer-implemented method of claim 1, wherein determining the first and second impairment scores includes:

calculating a plurality of impairment sub-scores based upon the received sensor data; and

calculating the first and second impairment scores as weighted combinations of the plurality of impairment sub-scores.

7. The computer-implemented method of claim 1, further comprising:

comparing, by one or more processors, the first and second impairment scores of the vehicle operator with impairment scores of a plurality of other vehicle operators; and

adjusting an insurance policy associated with the vehicle operator based at least in part upon the comparison.

8. The computer-implemented method of claim 1, wherein the received sensor data regarding the vehicle operator includes one or more of the following types of physiological sensor data regarding the vehicle operator:

a heart rate, heart rate variability data, a grip pressure, electrodermal activity data, a telematics driving score, a body temperature, an arm movement, a head movement, a vocal amplitude, a vocal frequency, a vocal pattern, a gaze direction, a gaze duration, a head direction, an eyelid opening, a blink rate, pupillometry data, a blood pressure, electroencephalographic data, a respiration rate, a respiration pattern, a galvanic skin response, functional near infrared optical brain imaging data, functional magnetic resonance imaging data, or electromyographic data.

9. A computer system for monitoring a vehicle operator during operation of a vehicle, comprising:

one or more processors;

one or more sensors; and

a program memory coupled to the one or more processors and storing executable instructions that, when executed by the one or more processors, cause the computer system to;

receive sensor data from one or more sensors regarding the vehicle operator, including data related to at least one of the following;

alertness of the vehicle operator, distractedness of the vehicle operator, emotional state of the vehicle operator, or vehicle operator workload due to traffic conditions;

determine, using the received sensor data, first and second impairment scores associated with first and second vehicle operator states, including at least two of the following;

alertness of the vehicle operator, distractedness of the vehicle operator, anxiety of the vehicle operator, agitation of the vehicle operator, aggression of the vehicle operator, or health of the vehicle operator;

determine the vehicle operator is impaired with respect to the first vehicle operator state when the first impairment score exceeds a first threshold value associated with the first impairment score;

determine the second impairment score associated with the second vehicle operator state is elevated but does not exceed a second threshold value associated with the second impairment score;

determine one or more predicted reactions of the vehicle operator to one or more potential responses based upon the first and second impairment scores;

select a response to implement based upon the one or more predicted reactions of the vehicle operator to the one or more potential responses; and

cause the selected response to be implemented.

10. The computer system of claim 9, wherein the program memory further stores executable instructions that, when executed by the one or more processors, cause the computer system to:

receive additional sensor data from one or more sensors regarding the vehicle, including data related to at least one of the following;

traffic conditions, environmental conditions, or the movement of the vehicle;

wherein the one or more predicted reactions of the vehicle operator are further determined based upon the received additional sensor data.

11. The computer system of claim 9, wherein the program memory further stores executable instructions that, when executed by the one or more processors, cause the computer system to:

receive a user profile associated with the vehicle operator, the user profile including information regarding reactions of the vehicle operator to stimuli associated with the one or more potential responses;

wherein the one or more predicted reactions of the vehicle operator are further determined based upon the user profile.

12. The computer system of claim 9, wherein the response is selected based upon a prediction of the ability of the response to mitigate the impairment of the vehicle operator.

13. The computer system of claim 9, wherein determining the first and second impairment scores includes:

calculating a plurality of impairment sub-scores based upon the received sensor data; and

calculating the first and second impairment scores as weighted combinations of the plurality of impairment sub-scores.

14. The computer system of claim 9, wherein the program memory further stores executable instructions that, when executed by the one or more processors, cause the computer system to:

compare the first and second impairment scores of the vehicle operator with impairment scores of a plurality of other vehicle operators; and

adjust an insurance policy associated with the vehicle operator based at least in part upon the comparison.

15. The computer system of claim 9, wherein the received sensor data regarding the vehicle operator includes one or more of the following types of physiological sensor data regarding the vehicle operator:

a heart rate, heart rate variability data, a grip pressure, electrodermal activity data, a telematics driving score, a body temperature, an arm movement, a head movement, a vocal amplitude, a vocal frequency, a vocal pattern, a gaze direction, a gaze duration, a head direction, an eyelid opening, a blink rate, pupillometry data, a blood pressure, electroencephalographic data, a respiration rate, a respiration pattern, a galvanic skin response, functional near infrared optical brain imaging data, functional magnetic resonance imaging data, or electromyographic data.

16. A tangible, non-transitory computer-readable medium storing instructions for monitoring a vehicle operator during operation of a vehicle that, when executed by one or more processors of a computer system, cause the computer system to:

receive sensor data from one or more sensors regarding the vehicle operator, including data related to at least one of the following;

alertness of the vehicle operator, distractedness of the vehicle operator, emotional state of the vehicle operator, or vehicle operator workload due to traffic conditions;

determine, using the received sensor data, first and second impairment scores associated with first and second vehicle operator states, including at least two of the following;

alertness of the vehicle operator, distractedness of the vehicle operator, anxiety of the vehicle operator, agitation of the vehicle operator, aggression of the vehicle operator, or health of the vehicle operator;

determine the vehicle operator is impaired with respect to the first vehicle operator state when the first impairment score exceeds a first threshold value associated with the first impairment score;

determine the second impairment score associated with the second vehicle operator state is elevated but does not exceed a second threshold value associated with the second impairment score;

determine one or more predicted reactions of the vehicle operator to one or more potential responses based upon the first and second impairment scores;

select a response to implement based upon the one or more predicted reactions of the vehicle operator to the one or more potential responses; and

cause the selected response to be implemented.

17. The tangible, non-transitory computer-readable medium of claim 16, further including instructions that, when executed by the one or more processors of the computer system, cause the computer system to:

receive additional sensor data from one or more sensors regarding the vehicle, including data related to at least one of the following;

traffic conditions, environmental conditions, or the movement of the vehicle;

wherein the one or more predicted reactions of the vehicle operator are further determined based upon the received additional sensor data.

18. The tangible, non-transitory computer-readable medium of claim 16, further including instructions that, when executed by the one or more processors of the computer system, cause the computer system to:

receive a user profile associated with the vehicle operator, the user profile including information regarding reactions of the vehicle operator to stimuli associated with the one or more potential responses;

wherein the one or more predicted reactions of the vehicle operator are further determined based upon the user profile.

19. The tangible, non-transitory computer-readable medium of claim 16, wherein the response is selected based upon a prediction of the ability of the response to mitigate the impairment of the vehicle operator.

20. The tangible, non-transitory computer-readable medium of claim 16, wherein determining the first and second impairment scores includes:

calculating a plurality of impairment sub-scores based upon the received sensor data; and

calculating the first and second impairment scores as weighted combinations of the plurality of impairment sub-scores.