Systems and methodologies for real-time driver gaze location determination and analysis utilizing computer vision technology

• US 10,343,520 B1
• Filed: 10/06/2017
• Issued: 07/09/2019
• Est. Priority Date: 12/17/2012
• Status: Active Grant

• Current Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Original Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Inventors: Plummer, Bryan Allen, Brown, Parker Andrew, He, Jibo, Fields, Brian M., Roberson, Steve, Cielocha, Steve, Peng, Jufeng
• Primary Examiner(s): Tun, Nay
• Application Number:
  US15/726,670
• Time in Patent Office: 641 Days
• CPC Class Codes: B60K 28/066 actuating a signalling device B60K28/063 takes precedence
• CPC Class Codes: B60Q 9/00 Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 , e.g. haptic signalling
• CPC Class Codes: B60W 2040/0818 Inactivity or incapacity of driver
• CPC Class Codes: B60W 2040/0827 due to sleepiness
• CPC Class Codes: B60W 2040/0836 due to alcohol
• CPC Class Codes: B60W 2040/0845 due to drugs
• CPC Class Codes: B60W 2050/143 Alarm means
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2540/221 Physiology, e.g. weight, heartbeat, health or special needs
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: B60W 50/16 Tactile feedback to the driver, e.g. vibration or force feedback to the driver on the steering wheel or the accelerator pedal
• CPC Class Codes: G06Q 40/08 Insurance
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G07C 5/008 communicating information to a remotely located station transmission systems for measured values G08C
• CPC Class Codes: G08B 21/02 Alarms for ensuring the safety of persons

Abstract

The method, system, and computer-readable medium facilitates monitoring one or more eyes of a vehicle operator during a driving session to generate a plurality of gaze location logs with gaze location values. The gaze location value may be generated by determining a location of the vehicle operator'"'"'s gaze, determining which area of the vehicle is associated with the gaze location, and assigning the gaze location value based on the area of the vehicle associated with the gaze location. The gaze location logs may be analyzed to determine the duration of the vehicle operator'"'"'s gaze at each area of the vehicle. Based on the duration of the vehicle operators gaze, one or more recommendations to improve vehicle operator performance may be identified and communicated to the vehicle operator.

Claims

1. A method comprising:

monitoring, with one or more image capture devices, one or more eyes of a vehicle operator while operating a vehicle to generate a plurality of gaze location logs, each of the plurality of gaze location logs including a gaze location value,

wherein the gaze location value is generated by;

determining a gaze location corresponding to the one or more eyes of the vehicle operator,

determining a specific area of the vehicle associated with the gaze location, and

assigning the gaze location value based on the specific area of the vehicle associated with the gaze location;

analyzing, by one or more processors, the plurality of gaze location logs to determine times that a gaze of the vehicle operator is directed at specific areas of the vehicle;

analyzing, by the one or more processors, the times that the gaze of the vehicle operator is directed at specific areas of the vehicle to determine whether the operator is impaired, at least in part by determining whether the gaze of the vehicle operator shifts to a first specific area of the vehicle with at least a threshold frequency, wherein

the first specific area is a mirror of the vehicle or a dashboard of the vehicle, and

failing to shift the gaze of the vehicle operator to the first specific area with at least the threshold frequency causes an impairment score to move closer to indicating, or to more strongly indicate, impairment of the vehicle operator;

identifying, by the one or more processors and based at least in part on the impairment score, one or more recommendations to improve performance of the vehicle operator; and

electronically communicating the one or more recommendations to the vehicle operator.

2. The method of claim 1, wherein the recommendations to improve performance of the vehicle operator include a recommendation to adjust a scanning frequency.

3. The method of claim 1, wherein identifying the one or more recommendations includes determining whether the vehicle operator is frequently impaired.

4. The method of claim 1, wherein each of the plurality of gaze location logs further includes a time stamp.

5. The method of claim 1, wherein the first specific area is a front windshield.

6. A computer system comprising:

one or more processors;

one or more image capture devices; and

a program memory storing instructions that when executed by the one or more processors cause the computer system to;

monitor, with the one or more image capture devices, one or more eyes of a vehicle operator while operating a vehicle to generate a plurality of gaze location logs, each of the plurality of gaze location logs including a gaze location value,

wherein the gaze location value is generated by;

determining a gaze location corresponding to the one or more eyes of the vehicle operator,

determining a specific area of the vehicle associated with the gaze location, and

assigning the gaze location value based on the specific area of the vehicle associated with the gaze location;

analyze the plurality of gaze location logs to determine times that a gaze of the vehicle operator is directed at specific areas of the vehicle;

analyze the times that the gaze of the vehicle operator is directed at specific areas of the vehicle to determine whether the operator is impaired, at least in part by determining whether the gaze of the vehicle operator shifts to a first specific area of the vehicle with at least a threshold frequency, wherein

the first specific area is a mirror of the vehicle or a dashboard of the vehicle, and

failing to shift the gaze of the vehicle operator to the first specific area with at least the threshold frequency causes an impairment score to move closer to indicating, or to more strongly indicate, impairment of the vehicle operator;

identify, based at least in part on the impairment score, one or more recommendations to improve performance of the vehicle operator; and

communicate the one or more recommendations to the vehicle operator.

7. The computer system of claim 6, wherein the recommendations to improve performance of the vehicle operator include a recommendation to adjust a scanning frequency.

8. The computer system of claim 6, wherein the instructions cause the computer system to identify the one or more recommendations at least by:

determining whether the vehicle operator is frequently impaired.

9. The computer system of claim 6, wherein each of the plurality of gaze location logs further includes a time stamp.

10. The system of claim 6, wherein the first specific area is a front windshield.

11. A non-transitory, computer-readable medium storing instructions that when executed by one or more processors cause a computer system to:

monitor, with one or more image capture devices, one or more eyes of a vehicle operator while operating a vehicle to generate a plurality of gaze location logs, each of the plurality of gaze location logs including a gaze location value,

wherein the gaze location value is generated by;

determining a gaze location corresponding to the one or more eyes of the vehicle operator,

determining a specific area of the vehicle associated with the gaze location, and

assigning the gaze location value based on the specific area of the vehicle associated with the gaze location;

analyze the plurality of gaze location logs to determine times that a gaze of the vehicle operator is directed at specific areas of the vehicle;

analyze the times that the gaze of the vehicle operator is directed at specific areas of the vehicle to determine whether the operator is impaired, at least in part by determining whether the gaze of the vehicle operator shifts to a first specific area of the vehicle with at least a threshold frequency, wherein

the first specific area is a mirror of the vehicle or a dashboard of the vehicle, and

failing to shift the gaze of the vehicle operator to the first specific area with at least the threshold frequency causes an impairment score to move closer to indicating, or to more strongly indicate, impairment of the vehicle operator;

identify, based at least in part on the impairment score, one or more recommendations to improve performance of the vehicle operator; and

communicate the one or more recommendations to the vehicle operator.

12. The non-transitory, computer-readable medium of claim 11, wherein the one or more recommendations include a recommendation to adjust a scanning frequency.

13. The non-transitory, computer-readable medium of claim 11, wherein the instructions cause the computer system to identify the one or more recommendations at least by determining whether the vehicle operator is frequently impaired.

14. The non-transitory, computer-readable medium of claim 11, wherein the first specific area is a front windshield.