Eye tracking method and system

• US 10,713,813 B2
• Filed: 02/22/2019
• Issued: 07/14/2020
• Est. Priority Date: 02/22/2018
• Status: Active Grant

• Current Assignee: Innodem Neurosciences
• Original Assignee: Innodem Neurosciences
• Inventors: De Villers-Sidani, Etienne, Drouin-Picaro, Paul Alexandre
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US16/283,360
• Publication Number:
  US 20190259174A1
• Time in Patent Office: 508 Days
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris

Abstract

A computer-implemented method for determining a gaze position of a user, comprising: receiving an initial image of at least one eye of the user; extracting at least one color component of the initial image to obtain a corresponding at least one component image; for each component image, determining a respective internal representation; determining an estimated gaze position in the initial image by applying a respective primary stream to obtain a respective internal representation for each of the at least one component image; and outputting the estimated gaze position. The processing of the component images is performed using a neural network configured to, at run time and after the neural network has been trained, process the component images using one or more neural network layers to generate the estimated gaze position. A system for determining a gaze position of a user is also provided.

Claims

1. A computer-implemented method for determining a gaze position of a user, comprising:

from an initial image of at least one eye of the user, extracting at least one color component of the initial image to obtain a corresponding at least one component image;

applying a respective primary stream, comprising at least one convolutional layer, to each one of the at least one component image to obtain a respective internal representation for each one of the at least one component image; and

determining an estimated gaze position for the initial image using the respective internal representation for each of the at least one component image.

2. The computer-implemented method of claim 1, wherein the received initial image contains additional features other than the at least one eye, the method further comprising:

identifying the at least one eye within the received initial image using a facial feature or landmark recognition method; and

extracting a portion of the initial image containing only the at least one eye, thereby obtaining a cropped image,

wherein said extracting at least one color component is performed in the cropped image to obtain the corresponding at least one component image.

3. The computer-implemented method of claim 1, wherein each respective primary stream comprises at least one fully-connected layer downstream the at least one convolutional layer.

4. The computer-implemented method of claim 1, wherein said extracting at least one color component comprises extracting at least two distinct color components of the initial image to obtain at least two corresponding component images, and further wherein said determining an estimated gaze position comprises combining each of the respective internal representations together using weight factors.

5. The computer-implemented method of claim 1, wherein said determining an estimated gaze position comprises independently determining each of a first coordinate and a second coordinate of the estimated gaze position.

6. The computer-implemented method of claim 1, further comprising:

receiving at least one calibration image of the user associated with a calibration position; and

determining a calibrated estimated gaze position based on said at least one calibration image.

7. The computer-implemented method of claim 6, wherein said determining the calibrated estimated gaze position comprises determining each of a first coordinate and a second coordinate independently.

8. The computer-implemented method of claim 6, wherein said determining the calibrated estimated gaze position is performed using one of:

a calibration neural network comprising one or more fully-connected neural network layers, a ridge regression, decision trees, a support vector machine, and a linear regression.

9. The computer-implemented method of claim 1, further comprising:

determining an orientation of the initial image relative to a reference;

wherein said determining an estimated gaze position is performed for a predetermined one orientation of the initial image.

10. A system for determining a gaze position of a user, comprising:

an extracting unit configured for extracting at least one color component of an initial image of at least one eye of the user to obtain a corresponding at least one component image;

an internal representation determining unit configured for applying a respective primary stream to each one of the at least one component image to obtain a respective internal representation for each one of the at least one component image; and

a gaze position estimating unit configured for determining an estimated gaze position in the initial image according to the respective internal representation of each of the at least one component image.

11. The system for determining a gaze position of claim 10, wherein the internal representation determining unit and the gaze position estimating unit are part of a neural network implemented by one or more computers and comprising one or more neural network layers, and wherein the neural network is configured to, at run time and after the neural network has been trained, process the at least one component image using the one or more neural network layers to determine the estimated gaze position.

12. The system of claim 10, wherein each of the respective primary stream comprises at least one convolutional layer, and at least one fully-connected layer downstream the corresponding at least one convolutional layer.

13. The system of claim 12, wherein the gaze position estimating unit comprises at least one fusion layer, each of the at least one fusion layer having at least one fully-connected layer and an output layer downstream the at least one fusion layer and comprising at least one fully-connected layer.

14. The system of claim 10, wherein the extracting unit is configured for extracting at least two distinct color components of the initial image to obtain at least two corresponding component images, and wherein said gaze position estimating unit is configured for combining each of the respective gaze positions together using weight factors.

15. The system of claim 10, wherein the extracting unit is configured to process each of a first coordinate and a second coordinate of the estimated gaze position independently.

16. The system of claim 15, further comprising a calibration model comprising one of:

a calibration neural network, a ridge regression, decision trees, a support vector machine, or a linear regression, configured to determine and output a calibrated estimated gaze position, and wherein the calibration model is configured to process each of a first coordinate and a second coordinate of the calibrated estimated gaze position independently.

17. The system of claim 10, further comprising an image orientation determination module for determining an orientation of the initial image relative to a reference, and wherein the gaze position estimating unit each comprises four orientation modules, each being configured for processing the initial image for a predetermined one orientation of the initial image.

18. A computer-implemented method for determining a gaze position of a user, comprising:

from an initial image of at least one eye of the user, extracting at least one color component of the initial image to obtain a corresponding at least one component image;

for each one of the at least one component image, determining a respective gaze position; and

determining an estimated gaze position in the initial image according to the respective gaze position of each of the at least one component image.

19. The computer-implemented method of claim 18, wherein said determining a respective gaze position comprises performing a regression method.

20. The computer-implemented method of claim 19, wherein said determining an estimated gaze position comprises performing a regression method.