Eye gaze tracking utilizing surface normal identification

• US 10,088,899 B2
• Filed: 11/30/2016
• Issued: 10/02/2018
• Est. Priority Date: 05/20/2014
• Status: Active Grant

• Current Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Original Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Inventors: Plummer, Bryan Allen, Brown, Parker Andrew, Fields, Brian Mark, Roberson, Steve
• Primary Examiner(s): ANDERSON II, JAMES M
• Application Number:
  US15/365,421
• Publication Number:
  US 20170083087A1
• Time in Patent Office: 671 Days
• Field of Search: None
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris

Abstract

A gaze tracking system captures images of a vehicle operator. The gaze tracking system may detect facial features in the images and track the position of the facial features over time. The gaze tracking system may detect a triangle in an image, wherein the vertices of the triangle correspond to the facial features. The gaze tracking system may analyze the detected triangle to identify a surface normal for the triangle, and may track the surface normal (e.g., across multiple images) to track the eye gaze direction of the driver over time. The images may be captured and analyzed in near-real time. By tracking movement of the driver'"'"'s head and eyes over time, the gaze analysis system may predict or estimate head position and/or gaze direction when one or more facial features are not detectable.

Claims

1. A method comprising:

capturing, via an image sensor, a sequence of images of a driver of a vehicle over a time period;

in each of the sequence of images, detecting a triangle in three-dimensional (3D) space representing a relationship between three facial features including a first pupil, a second pupil, and a third feature of the driver depicted in the sequence of images, the triangle having vertices corresponding to the first pupil, the second pupil, and the third feature, wherein the third feature is selected from a group consisting of a nose, a mouth, and a chin;

for each of the sequence of images, analyzing the detected triangle in 3D space to identify a plane of the triangle in 3D space;

for each of the sequence of images, analyzing the identified plane of the triangle in 3D space to identify a surface normal for the identified plane in 3D space; and

tracking a change in an eye gaze direction of the driver over the time period by tracking a change in the identified surface normal for the plane in 3D space throughout the sequence of images, wherein the change in the eye gaze direction can be tracked regardless of whether the driver'"'"'s head pose changes during the time period.

2. The method of claim 1, further comprising:

calibrating the image sensor.

3. The method of claim 2, wherein calibrating the image sensor comprises:

adjusting the image sensor to account for lighting conditions in the vehicle.

4. The method of claim 2, wherein calibrating the image sensor comprises:

capturing a reference image of the driver gazing in a known direction with a known head pose; and

analyzing the reference image to detect the three facial features in the reference image;

wherein detecting the triangle in 3D space in each of the sequence of images includes;

identifying, in each of the sequence of images, the three facial features detected in the reference image.

5. The method of claim 4, wherein calibrating the image sensor further comprises:

displaying audio or visual instructions to instruct the driver to gaze in the known direction and to assume the known head pose.

6. The method of claim 4, wherein calibrating the image sensor further comprises:

estimating that the driver is gazing in the known direction with the known head pose based on an analysis of a distribution of the facial features in one or more previously captured images of the driver;

wherein analyzing the reference image to detect the three facial features in the reference image occurs in response to estimating that the driver is gazing in the known direction with the known head pose.

7. The method of claim 1, further comprising:

generating a report identifying one or more areas to which the tracked eye gaze direction is directed.

8. The method of claim 1, further comprising:

generating an alarm when the tracked eye gaze direction is outside of a predefined range of safe gaze directions.

9. A system comprising:

an image sensor that captures a sequence of images of a driver of a vehicle over a time period;

one or more processors that are communicatively connected to the image sensor and that;

(i) detect, in each of the sequence of images captured by the image sensor, a triangle in three-dimensional (3D) space representing a relationship between three facial features including a first pupil, a second pupil, and a third feature of the driver of the vehicle depicted in the sequence of images, the triangle having vertices corresponding to the first pupil, the second pupil, and the third feature, wherein the third feature is selected from a group consisting of a nose, a mouth, and a chin;

(ii) for each of the sequence of images, analyze the detected triangle in 3D space to identify a plane of the triangle in 3D space;

(iii) for each of the sequence of images, analyze the identified plane of the triangle in 3D space to identify a surface normal for the identified plane in 3D space; and

(iv) track a change in an eye gaze direction of the driver over the time period by tracking a change in the identified surface normal for the plane in 3D space throughout the sequence of images, wherein the change in the eye gaze direction can be tracked regardless of whether the driver'"'"'s head pose changes during the time period.

10. The system of claim 9, wherein the image sensor is embedded in a dash or a console of the vehicle.

11. The system of claim 9, wherein the one or more memory devices further comprise instructions to perform a calibration.

12. The system of claim 11, wherein the instructions to perform the calibration comprise instructions to adjust the image sensor to account for lighting conditions in the vehicle.

13. The system of claim 11, wherein the instructions to perform the calibration comprise:

instructions to cause the image sensor to capture a reference image of the driver gazing in a known direction with a known head pose; and

instructions to analyze the reference image to detect the three facial features in the reference image;

wherein the instructions to detect the triangle in 3D space includes;

instructions to identify, in each of the sequence of images, the three facial features detected in the reference image.

14. The system of claim 13, wherein the instructions to perform the calibration further comprise:

instructions display audio or visual instructions to instruct the driver to gaze in the known direction and to assume the known head pose.

15. The system of claim 13, wherein the instructions to perform the calibration further comprise:

instructions to estimate that the driver is gazing in the known direction with the known head pose based on an analysis of a distribution of the facial features in one or more previously captured images of the driver;

wherein the instructions to analyze the reference image to detect the three facial features in the reference image occurs in response to estimating that the driver is gazing in the known direction with the known head pose.

16. The system of claim 9, further comprising a speaker or display communicatively coupled to the one or more processors;

wherein the one or more processors generate an alarm via the speaker or display when the tracked eye gaze direction is outside of a predefined range of safe gaze directions.

17. A system comprising:

a means for capturing a sequence of images of a driver of a vehicle over a time period;

a means for detecting, in each of the sequence of images, a triangle in three-dimensional (3D) space representing a relationship between three facial features including a first pupil, a second pupil, and a third feature of the driver depicted in the sequence of images, the triangle having vertices corresponding to the first pupil, the second pupil, and the third feature, wherein the third feature is selected from a group consisting a nose, a mouth, and a chin;

a means for analyzing, for each of the sequence of images, the detected triangle in 3D space to identify a plane of the triangle in 3D space;

a means for analyzing, for each of the sequence of images, the identified plane of the triangle in 3D space to identify a surface normal for the identified plane in 3D space; and

a means for tracking a change in an eye gaze direction of the driver over the time period by tracking a change in the identified surface normal for the plane in 3D space throughout the sequence of images, wherein the change in the eye gaze direction can be tracked regardless of whether the driver'"'"'s head pose changes during the time period.

18. The system of claim 17, further comprising:

a means for performing a calibration including;

(i) a means for displaying audio or visual instructions to instruct the driver to gaze in a particular direction and to assume a particular head pose;

(ii) a means for capturing a reference image of the driver gazing in the particular direction with the particular head pose; and

wherein the means for detecting the triangle in 3D space includes;

a means for identifying, in each of the sequence of images, the three facial features detected in the reference image.

19. The system of claim 17, wherein the means for capturing the sequence of images of the driver of the vehicle over the time period comprises:

a mobile device.

20. The system of claim 17, wherein the means for capturing the sequence of images of the driver of the vehicle over the time period comprises:

an image sensor embedded in a dash or a console of the vehicle.