Systems and methodologies for real-time driver gaze location determination and analysis utilizing computer vision technology

• US 9,868,352 B1
• Filed: 12/22/2015
• Issued: 01/16/2018
• Est. Priority Date: 12/17/2012
• Status: Active Grant

• Current Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Original Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Inventors: Plummer, Bryan Allen, Brown, Parker Andrew, He, Jibo, Fields, Brian M., Roberson, Steve, Cielocha, Steve, Peng, Jufeng
• Primary Examiner(s): Tun, Nay
• Application Number:
  US14/978,416
• Time in Patent Office: 756 Days
• CPC Class Codes: B60K 28/066 actuating a signalling device B60K28/063 takes precedence
• CPC Class Codes: B60Q 9/00 Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 , e.g. haptic signalling
• CPC Class Codes: B60W 2040/0818 Inactivity or incapacity of driver
• CPC Class Codes: B60W 2040/0827 due to sleepiness
• CPC Class Codes: B60W 2040/0836 due to alcohol
• CPC Class Codes: B60W 2040/0845 due to drugs
• CPC Class Codes: B60W 2050/143 Alarm means
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2540/221 Physiology, e.g. weight, heartbeat, health or special needs
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: B60W 50/16 Tactile feedback to the driver, e.g. vibration or force feedback to the driver on the steering wheel or the accelerator pedal
• CPC Class Codes: G06Q 40/08 Insurance
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G07C 5/008 communicating information to a remotely located station transmission systems for measured values G08C
• CPC Class Codes: G08B 21/02 Alarms for ensuring the safety of persons

Abstract

The method, system, and computer-readable medium facilitates monitoring one or more eyes of a vehicle operator during a driving session to generate a plurality of gaze location logs with gaze location values. The gaze location value may be generated by determining a location of the vehicle operator'"'"'s gaze, determining which area of the vehicle is associated with the gaze location, and assigning the gaze location value based on the area of the vehicle associated with the gaze location. The gaze location logs may be analyzed to determine the duration of the vehicle operator'"'"'s gaze at each area of the vehicle. Based on the duration of the vehicle operators gaze, one or more recommendations to improve vehicle operator performance may be identified and communicated to the vehicle operator.

Claims

1. A method comprising:

monitoring, with one or more image capture devices, one or more eyes of a vehicle operator while operating a vehicle to generate a plurality of gaze location logs, each of the plurality of gaze location logs including a gaze location value,

wherein the gaze location value is generated by;

determining a gaze location corresponding to the one or more eyes of the vehicle operator,

determining an area of the vehicle associated with the gaze location, and

assigning the gaze location value based on the area of the vehicle associated with the gaze location;

analyzing, by one or more processors, the plurality of gaze location logs to determine a duration of a gaze of the vehicle operator for each of a plurality of areas of the vehicle;

for each of the plurality of areas of the vehicle, collectively analyzing, by the one or more processors, the durations of the gaze of the vehicle operator to identify one or more recommendations to improve performance of the vehicle operator, wherein

collectively analyzing the durations of the gaze of the vehicle operator includes determining whether the gaze of the vehicle operator shifts from a front windshield of the vehicle to one or more different areas of the plurality of areas with a sufficiently high frequency, and

the one or more different areas include at least one or both of (i) a mirror of the vehicle, and (ii) a dashboard of the vehicle; and

electronically communicating the one or more recommendations to the vehicle operator.

2. The method of claim 1, wherein collectively analyzing the durations of the gaze of the vehicle operator to identify one or more recommendations to improve performance of the vehicle operator includes:

determining a scanning frequency of the vehicle operator.

3. The method of claim 2, wherein the recommendations to improve performance of the vehicle operator include a recommendation to one or both of remove distractions or adjust the scanning frequency.

4. The method of claim 1, wherein the recommendations to improve performance of the vehicle operator include a recommendation to adjust the duration of the gaze of the vehicle operator for at least one of the plurality of areas of the vehicle.

5. The method of claim 1, wherein collectively analyzing the durations of the gaze of the vehicle operator to identify one or more recommendations to improve performance of the vehicle operator includes determining whether the vehicle operator is frequently impaired.

6. The method of claim 1, wherein the plurality of areas of the vehicle further includes one or more other objects inside the vehicle.

7. The method of claim 1, wherein each of the plurality of gaze location logs further includes a time stamp.

8. The method of claim 7, wherein analyzing the plurality of gaze location logs to determine a duration of a gaze of the vehicle operator for each of the plurality of areas of the vehicle includes calculating a difference between two timestamps included in the plurality of gaze location logs.

9. A computer system comprising:

one or more processors;

one or more image capture devices; and

a program memory storing instructions that when executed by the one or more processors cause the computer system to;

monitor, with the one or more image capture devices, one or more eyes of a vehicle operator while operating a vehicle to generate a plurality of gaze location logs, each of the plurality of gaze location logs including a gaze location value,

wherein the gaze location value is generated by;

determining a gaze location corresponding to the one or more eyes of the vehicle operator,

determining an area of the vehicle associated with the gaze location, and

assigning the gaze location value based on the area of the vehicle associated with the gaze location;

analyze the plurality of gaze location logs to determine a duration of a gaze of the vehicle operator for each of a plurality of areas of the vehicle;

for each of the plurality of areas of the vehicle, collectively analyze the durations of the gaze of the vehicle operator to identify one or more recommendations to improve performance of the vehicle operator, wherein

collectively analyzing the durations of the gaze of the vehicle operator includes determining whether the gaze of the vehicle operator shifts from a front windshield of the vehicle to one or more different areas of the plurality of areas with a sufficiently high frequency, and

the one or more different areas include at least one or both of (i) a mirror of the vehicle, and (ii) a dashboard of the vehicle; and

communicate the one or more recommendations to the vehicle operator.

10. The computer system of claim 9, wherein the instructions cause the computer system to collectively analyze the durations of the gaze of the vehicle operator to identify one or more recommendations to improve performance of the vehicle operator at least by:

determining a scanning frequency of the vehicle operator.

11. The computer system of claim 10, wherein the recommendations to improve performance of the vehicle operator include one or more of a recommendation to remove distractions or a recommendation to adjust the scanning frequency.

12. The computer system of claim 9, wherein the recommendations to improve performance of the vehicle operator include a recommendation to adjust the duration of the gaze of the vehicle operator for at least one of the plurality of areas of the vehicle.

13. The computer system of claim 9, wherein the instructions cause the computer system to collectively analyze the durations of the gaze of the vehicle operator to identify one or more recommendations to improve performance of the vehicle operator at least by:

determining whether the vehicle operator is frequently impaired.

14. The computer system of claim 9, wherein each of the plurality of gaze location logs further includes a time stamp.

15. The computer system of claim 14, wherein the instructions cause the computer system to analyze the plurality of gaze location logs to determine a duration of a gaze of the vehicle operator for each of the plurality of areas of the vehicle at least by:

calculating a difference between two timestamps included in the plurality of gaze location logs.

16. A non-transitory, computer-readable medium storing instructions that when executed by one or more processors cause a computer system to:

monitor, with one or more image capture devices, one or more eyes of a vehicle operator while operating a vehicle to generate a plurality of gaze location logs, each of the plurality of gaze location logs including a gaze location value,

wherein the gaze location value is generated by;

determining a gaze location corresponding to the one or more eyes of the vehicle operator,

determining an area of the vehicle associated with the gaze location, and

assigning the gaze location value based on the area of the vehicle associated with the gaze location;

analyze the plurality of gaze location logs to determine a duration of a gaze of the vehicle operator for each of a plurality of areas of the vehicle;

for each of the plurality of areas of the vehicle, collectively analyze the durations of the gaze of the vehicle operator to identify one or more recommendations to improve performance of the vehicle operator, wherein

collectively analyzing the durations of the gaze of the vehicle operator includes determining whether the gaze of the vehicle operator shifts from a front windshield of the vehicle to one or more different areas of the plurality of areas with a sufficiently high frequency, and

the one or more different areas include at least one or both of (i) a mirror of the vehicle, and (ii) a dashboard of the vehicle; and

communicate the one or more recommendations to the vehicle operator.

17. The non-transitory, computer-readable medium of claim 16, wherein the instructions cause the computer system to collectively analyze the durations of the gaze of the vehicle operator to identify one or more recommendations to improve performance of the vehicle operator at least by:

determining a scanning frequency of the vehicle operator.

18. The non-transitory, computer-readable medium of claim 17, wherein the recommendations to improve performance of the vehicle operator also include one or more of a recommendation to remove distractions or a recommendation to adjust the scanning frequency.

19. The non-transitory, computer-readable medium of claim 16, wherein the instructions cause the computer system to collectively analyze the durations of the gaze of the vehicle operator to identify one or more recommendations to improve performance of the vehicle operator at least by determining whether the vehicle operator is frequently impaired.

20. The non-transitory, computer-readable medium of claim 16, wherein:

each of the plurality of gaze location logs further includes a time stamp; and

the instructions cause the computer system to determine the duration of the gaze of the vehicle operator for each of the plurality of areas of the vehicle at least by calculating a difference between two timestamps included in the plurality of gaze location logs.