Advanced vehicle operator intelligence system

• US 9,440,657 B1
• Filed: 09/17/2015
• Issued: 09/13/2016
• Est. Priority Date: 04/17/2014
• Status: Active Grant

• Current Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Original Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Inventors: Fields, Brian Mark, Roberson, Steve, Chan, Aaron Scott, Sanchez, Kenneth Jason, Donovan, Matthew Christopher Jon
• Primary Examiner(s): BEE, ANDREW W.
• Application Number:
  US14/857,242
• Time in Patent Office: 362 Days
• US Class Current: 1/1
• CPC Class Codes: B60K 28/066 actuating a signalling device B60K28/063 takes precedence
• CPC Class Codes: B60W 2040/0818 Inactivity or incapacity of driver
• CPC Class Codes: B60W 2040/0872 Driver physiology
• CPC Class Codes: B60W 2040/089 Driver voice
• CPC Class Codes: B60W 2520/00 Input parameters relating to overall vehicle dynamics
• CPC Class Codes: B60W 2520/105 Longitudinal acceleration
• CPC Class Codes: B60W 2520/125 Lateral acceleration
• CPC Class Codes: B60W 2540/21 Voice
• CPC Class Codes: B60W 2540/22 Psychological state; Stress level or workload
• CPC Class Codes: B60W 2540/221 Physiology, e.g. weight, heartbeat, health or special needs
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2554/801 Lateral distance
• CPC Class Codes: B60W 2555/20 Ambient conditions, e.g. wind or rain
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: B61L 15/009 On-board display devices
• CPC Class Codes: G06Q 40/08 Insurance
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G08B 21/02 Alarms for ensuring the safety of persons
• CPC Class Codes: G08B 21/06 indicating a condition of sleep, e.g. anti-dozing alarms
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/306 : User profiles
• CPC Class Codes: H04W 4/38 : for collecting sensor information
• CPC Class Codes: H04W 4/48 : for in-vehicle communication

Abstract

The method, system, and computer-readable medium cause the monitoring of a vehicle operator during the course of vehicle operation to determine whether the vehicle operator is impaired and causes a mitigating response when an impairment is determined to exist. The vehicle operator, the environment surrounding the vehicle, or forces acting on the vehicle may be monitored using a variety of sensors, including optical sensors, accelerometers, or biometric sensors (e.g., skin conductivity, heart rate, or voice modulation). When the vehicle operator is determined to be impaired, an alert or other mitigating response is implemented, based on the sensor data. In some embodiments, mitigating actions may be taken to avoid vehicle operator impairment. In further embodiments, a training period may be used to generate a profile for the vehicle operator.

Claims

1. A computer-implemented method for monitoring a vehicle operator during operation of a vehicle, comprising:

(i) during a training period;

receiving, at one or more processors, first sensor data from one or more sensors regarding at least one of the following;

alertness of the vehicle operator, distractedness of the vehicle operator, emotional state of the vehicle operator, traffic conditions, environmental conditions, or the movement of the vehicle;

determining, by one or more processors, that at least one response from a plurality of potential responses should be implemented based upon the received first sensor data;

selecting, by one or more processors, a first response from the plurality of potential responses;

causing the first response to be implemented;

receiving, at one or more processors, second sensor data from the one or more sensors following the commencement of implementation of the first response; and

recording in a user profile associated with the vehicle operator an indication of an effect of the first response on the vehicle operator, wherein the indication of the effect of the first response is based at least in part upon a reaction of the vehicle operator to the first response based upon the received second sensor data, and wherein recording in the user profile the indication of the effect of the first response on the vehicle operator includes generating the user profile based upon the received first and second sensor data; and

(ii) during a monitoring period following the training period;

receiving, at one or more processors, third sensor data from the one or more sensors regarding at least one of the following;

alertness of the vehicle operator, distractedness of the vehicle operator, emotional state of the vehicle operator, traffic conditions, environmental conditions, or the movement of the vehicle;

determining, by one or more processors, the vehicle operator is impaired, based upon the received third sensor data, with respect to at least one of the following vehicle operator states;

alertness of the vehicle operator, distractedness of the vehicle operator, anxiety of the vehicle operator, agitation of the vehicle operator, aggression of the vehicle operator, or health of the vehicle operator;

selecting, by one or more processors, a second response from the plurality of potential responses based upon the third sensor data and the user profile by;

determining two or more predicted reactions of the vehicle operator to two or more of the plurality of potential responses based upon the third sensor data and the user profile;

comparing the two or more predicted reactions of the vehicle operator to determine the most effective response from the two or more of the plurality of potential responses; and

selecting the most effective response from the two or more of the plurality of potential responses; and

causing the second response to be implemented.

2. The computer-implemented method of claim 1, wherein recording in the user profile the indication of the effect of the first response on the vehicle operator includes recording an operator reaction to the first response.

3. The computer-implemented method of claim 1, wherein:

the first response includes one or more stimuli to be presented to the vehicle operator, and

the indication of the effect of the first response on the vehicle operator includes information regarding one or more reactions of the vehicle operator to the one or more stimuli.

4. The computer-implemented method of claim 1, wherein recording in the user profile the indication of the effect of the first response on the vehicle operator further includes:

calculating a plurality of impairment scores based upon the received first sensor data,

calculating a corresponding plurality of impairment scores based upon the received second sensor data, and

recording an indication of the change between the plurality of impairment scores based upon the received first sensor data and the corresponding plurality of impairment scores based upon the received second sensor data.

5. The computer-implemented method of claim 4, further comprising:

calculating, by one or more processors, one or more total impairment scores from the plurality of impairment scores based upon the received first sensor data or the corresponding plurality of impairment scores based upon the received second sensor data, and

comparing, by one or more processors, the one or more total impairment scores with total impairment scores of other vehicle operators to adjust a risk level used in rating and underwriting an insurance policy of the vehicle operator.

6. The computer-implemented method of claim 1, wherein:

the received second sensor data includes one or more of the following types of physiological sensor data regarding the vehicle operator;

a heart rate, heart rate variability data, a grip pressure, electrodermal activity data, a telematics driving score, a body temperature, an arm movement, a head movement, a vocal amplitude, a vocal frequency, a vocal pattern, a gaze direction, a gaze duration, a head direction, an eyelid opening, a blink rate, pupillometry data, a blood pressure, electroencephalographic data, a respiration rate, a respiration pattern, a galvanic skin response, functional near infrared optical brain imaging data, functional magnetic resonance imaging data, or electromyographic data.

7. A computer system for monitoring a vehicle operator during operation of a vehicle, comprising:

one or more processors;

one or more sensors; and

a program memory coupled to the one or more processors and storing executable instructions that, when executed by the one or more processors, cause the computer system to;

(i) during a training period;

receive first sensor data from one or more sensors regarding at least one of the following;

alertness of the vehicle operator, distractedness of the vehicle operator, emotional state of the vehicle operator, traffic conditions, environmental conditions, or the movement of the vehicle;

determine that at least one response from a plurality of potential responses should be implemented based upon the received first sensor data;

select a first response from the plurality of potential responses;

cause the first response to be implemented;

receive second sensor data from the one or more sensors following the commencement of implementation of the first response; and

record in a user profile associated with the vehicle operator an indication of an effect of the first response on the vehicle operator, wherein the indication of the effect of the first response is based at least in part upon a reaction of the vehicle operator to the first response based upon the received second sensor data, and wherein recording in the user profile the indication of the effect of the first response on the vehicle operator includes generating the user profile based upon the received first and second sensor data; and

(ii) during a monitoring period following the training period;

receive third sensor data from the one or more sensors regarding at least one of the following;

alertness of the vehicle operator, distractedness of the vehicle operator, emotional state of the vehicle operator, traffic conditions, environmental conditions, or the movement of the vehicle;

determine the vehicle operator is impaired, based upon the received third sensor data, with respect to at least one of the following vehicle operator states;

alertness of the vehicle operator, distractedness of the vehicle operator, anxiety of the vehicle operator, agitation of the vehicle operator, aggression of the vehicle operator, or health of the vehicle operator;

select a second response from the plurality of potential responses based upon the third sensor data and the user profile by;

determining two or more predicted reactions of the vehicle operator to two or more of the plurality of potential responses based upon the third sensor data and the user profile;

comparing the two or more predicted reactions of the vehicle operator to determine the most effective response from the two or more of the plurality of potential responses; and

selecting the most effective response from the two or more of the plurality of potential responses; and

causing the second response to be implemented.

8. The computer system of claim 7, wherein recording in the user profile the indication of the effect of the first response on the vehicle operator includes recording an operator reaction to the first response.

9. The computer system of claim 7, wherein:

the first response includes one or more stimuli to be presented to the vehicle operator, and

the indication of the effect of the first response on the vehicle operator includes information regarding one or more reactions of the vehicle operator to the one or more stimuli.

10. The computer system of claim 7, wherein the executable instructions that cause the computer system to record in the user profile the indication of the effect of the first response on the vehicle operator further cause the computer system to:

calculate a plurality of impairment scores based upon the received first sensor data,

calculate a corresponding plurality of impairment scores based upon the received second sensor data, and

record an indication of the change between the plurality of impairment scores based upon the received first sensor data and the corresponding plurality of impairment scores based upon the received second sensor data.

11. The computer system of claim 10, wherein the program memory further stores executable instructions that, when executed by the computer system, cause the computer system to:

calculate one or more total impairment scores from the plurality of impairment scores based upon the received first sensor data or the corresponding plurality of impairment scores based upon the received second sensor data, and

compare the one or more total impairment scores with total impairment scores of other vehicle operators to adjust a risk level used in rating and underwriting an insurance policy of the vehicle operator.

12. The computer system of claim 7, wherein:

the received second sensor data includes one or more of the following types of physiological sensor data regarding the vehicle operator;

a heart rate, heart rate variability data, a grip pressure, electrodermal activity data, a telematics driving score, a body temperature, an arm movement, a head movement, a vocal amplitude, a vocal frequency, a vocal pattern, a gaze direction, a gaze duration, a head direction, an eyelid opening, a blink rate, pupillometry data, a blood pressure, electroencephalographic data, a respiration rate, a respiration pattern, a galvanic skin response, functional near infrared optical brain imaging data, functional magnetic resonance imaging data, or electromyographic data.

13. A tangible, non-transitory computer-readable medium storing instructions for monitoring a vehicle operator during operation of a vehicle that, when executed by one or more processors of a computer system, cause the computer system to:

(i) during a training period;

receive first sensor data from one or more sensors regarding at least one of the following;

alertness of the vehicle operator, distractedness of the vehicle operator, emotional state of the vehicle operator, traffic conditions, environmental conditions, or the movement of the vehicle;

determine that at least one response from a plurality of potential responses should be implemented based upon the received first sensor data;

select a first response from the plurality of potential responses;

cause the first response to be implemented;

receive second sensor data from the one or more sensors following the commencement of implementation of the first response; and

record in a user profile associated with the vehicle operator an indication of an effect of the first response on the vehicle operator, wherein the indication of the effect of the first response is based at least in part upon a reaction of the vehicle operator to the first response based upon the received second sensor data, and wherein recording in the user profile the indication of the effect of the first response on the vehicle operator includes generating the user profile based upon the received first and second sensor data; and

(ii) during a monitoring period following the training period;

receive third sensor data from the one or more sensors regarding at least one of the following;

alertness of the vehicle operator, distractedness of the vehicle operator, emotional state of the vehicle operator, traffic conditions, environmental conditions, or the movement of the vehicle;

determine the vehicle operator is impaired, based upon the received third sensor data, with respect to at least one of the following vehicle operator states;

alertness of the vehicle operator, distractedness of the vehicle operator, anxiety of the vehicle operator, agitation of the vehicle operator, aggression of the vehicle operator, or health of the vehicle operator;

select a second response from the plurality of potential responses based upon the third sensor data and the user profile by;

determining two or more predicted reactions of the vehicle operator to two or more of the plurality of potential responses based upon the third sensor data and the user profile;

comparing the two or more predicted reactions of the vehicle operator to determine the most effective response from the two or more of the plurality of potential responses; and

selecting the most effective response from the two or more of the plurality of potential responses; and

causing the second response to be implemented.

14. The tangible, non-transitory computer-readable medium of claim 13, wherein the executable instructions that cause the computer system to record in the user profile the indication of the effect of the first response on the vehicle operator further include executable instructions that cause the computer system to record an operator reaction to the first response.

15. The tangible, non-transitory computer-readable medium of claim 13, wherein:

the first response includes one or more stimuli to be presented to the vehicle operator, and

the indication of the effect of the first response on the vehicle operator includes information regarding one or more reactions of the vehicle operator to the one or more stimuli.

16. The tangible, non-transitory computer-readable medium of claim 13, wherein the executable instructions that cause the computer system to record in the user profile the indication of the effect of the first response on the vehicle operator further cause the computer system to:

calculate a plurality of impairment scores based upon the received first sensor data,

calculate a corresponding plurality of impairment scores based upon the received second sensor data, and

record an indication of the change between the plurality of impairment scores based upon the received first sensor data and the corresponding plurality of impairment scores based upon the received second sensor data.

17. The tangible, non-transitory computer-readable medium of claim 16, further storing executable instructions that, when executed by the one or more processors of the computer system, cause the computer system to:

calculate one or more total impairment scores from the plurality of impairment scores based upon the received first sensor data or the corresponding plurality of impairment scores based upon the received second sensor data, and compare the one or more total impairment scores with total impairment scores of other vehicle operators to adjust a risk level used in rating and underwriting an insurance policy of the vehicle operator.

18. The tangible, non-transitory computer-readable medium of claim 13, wherein:

the received second sensor data includes one or more of the following types of physiological sensor data regarding the vehicle operator;

a heart rate, heart rate variability data, a grip pressure, electrodermal activity data, a telematics driving score, a body temperature, an arm movement, a head movement, a vocal amplitude, a vocal frequency, a vocal pattern, a gaze direction, a gaze duration, a head direction, an eyelid opening, a blink rate, pupillometry data, a blood pressure, electroencephalographic data, a respiration rate, a respiration pattern, a galvanic skin response, functional near infrared optical brain imaging data, functional magnetic resonance imaging data, or electromyographic data.