Eye tracking method and system

• US 10,713,814 B2
• Filed: 06/07/2019
• Issued: 07/14/2020
• Est. Priority Date: 02/22/2018
• Status: Active Grant

• Current Assignee: Innodem Neurosciences
• Original Assignee: Innodem Neurosciences
• Inventors: De Villers-Sidani, Etienne, Drouin-Picaro, Paul Alexandre
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US16/435,062
• Publication Number:
  US 20190295287A1
• Time in Patent Office: 403 Days
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris

Abstract

A computer-implemented method for determining a gaze position of a user, comprising: receiving an initial image of at least one eye of the user; extracting at least one color component of the initial image to obtain a corresponding at least one component image; for each component image, determining a respective internal representation; determining an estimated gaze position in the initial image by applying a respective primary stream to obtain a respective internal representation for each of the at least one component image; and outputting the estimated gaze position. The processing of the component images is performed using a neural network configured to, at run time and after the neural network has been trained, process the component images using one or more neural network layers to generate the estimated gaze position. A system for determining a gaze position of a user is also provided.

Claims

1. A system for determining a gaze position of a user, comprising:

an extracting unit configured for receiving an initial image of at least one eye of the user and extracting at least one color component of the initial image to obtain a corresponding at least one component image;

an internal representation determining unit configured for applying a respective primary stream to each one of the at least one component image to obtain a respective internal representation for each one of the at least one component image; and

a gaze position estimating unit configured for determining an estimated gaze position in the initial image according to the respective internal representation of each of the at least one component image and outputting the estimated gaze position.

2. The system for determining a gaze position of claim 1, wherein the internal representation determining unit and the gaze position estimating unit are part of a neural network implemented by one or more computers and comprising one or more neural network layers, and wherein the neural network is configured to, at run time and after the neural network has been trained, process the at least one component image using the one or more neural network layers to generate the estimated gaze position.

3. The system of claim 2, wherein the neural network comprises:

at least one first neural network stream, each being configured to generate the respective internal representation; and

a second neural network stream configured to generate the estimated gaze position.

4. The system of claim 3, wherein the extracting unit is further configured for:

identifying the at least one eye within the received initial image using a facial feature or landmark recognition method;

extracting a portion of the initial image containing only the at least one eye to obtain a cropped image; and

extracting the at least one color component of the cropped image to obtain the corresponding at least one component image.

5. The system of claim 1, further comprising a camera for acquiring the initial image of at least one eye of the user.

6. The system of claim 5, further comprising an electronic device having a screen to which estimated gaze position relates.

7. The system of claim 6, wherein the electronic device comprises the camera.

8. A computer-implemented method for determining a gaze position of a user, comprising:

receiving an initial image of at least one eye of the user;

extracting at least one color component of the initial image to obtain a corresponding at least one component image;

applying a respective primary stream to each one of the at least one component image to obtain a respective internal representation for each one of the at least one component image;

determining an estimated gaze position for the initial image using the respective internal representation for each of the at least one component image; and

outputting the estimated gaze position.

9. The computer-implemented method of claim 8, wherein said applying the primary stream to obtain the respective internal representation and said determining an estimated gaze position are performed in combination and comprise:

processing the at least one component image using a neural network, at least one portion of the neural network being the respective primary stream respective to each of the at least one component image, wherein the neural network is implemented by one or more computers and comprises one or more neural network layers, and wherein the neural network is configured to, at run time and after the neural network has been trained, process the at least one component image using the one or more neural network layers to generate the estimated gaze position.

10. The computer-implemented method of claim 9, wherein said applying the respective primary stream to obtain the respective internal representation is performed separately for each of the at least one component image, and comprises at least one convolutional layer and at least one fully-connected layer downstream of the at least one convolutional layer for each one of the at least one component image.

11. The computer-implemented method of claims 10, wherein the neural network comprises another portion, namely an internal stream, downstream the respective primary stream, wherein said determining an estimated gaze position is performed using the internal stream comprising at least one fusion layer, the internal stream having at least one fully-connected layer.

12. The computer-implemented method of claim 11, wherein the internal stream starts at a fusion layer receiving at least the respective internal representation and further comprises an output layer downstream the fusion layer and comprising at least one fully-connected layer.

13. The computer-implemented method of claim 8, wherein said extracting at least one color component comprises extracting each of three RGB components of the initial image.

14. The computer-implemented method of claim 8, further comprising:

identifying the at least one eye within the received initial image using a facial feature or landmark recognition method; and

extracting a portion of the initial image containing only the at least one eye, thereby obtaining a cropped image,

wherein said extracting at least one color component is performed in the cropped image to obtain the corresponding at least one component image.

15. The computer-implemented method of claim 14, wherein said identifying the at least one eye comprises identifying at least one of:

an outline of the at least one eye, a position of at least one of a limbus, an iris and a pupil of the at least one eye.

16. The computer-implemented method of claim 15, wherein said extracting at least one color component comprises extracting at least two distinct color components of the initial image to obtain at least two corresponding component images, the method further comprising:

for each of the at least two corresponding component images, determining an illuminant value representative of the relative contribution of the each of the at least two corresponding component images to the initial image,

wherein said determining an estimated gaze position further comprises combining the illuminant values with the respective internal representation of the at least one component image.

17. The computer-implemented method of claim 16, wherein the illuminant values are processed using an illuminant neural network comprising one or more fully-connected neural network layers.

18. The computer-implemented method of claim 17, wherein the received initial image further contains at least one facial landmark, the method further comprising:

extracting the at least one facial landmark to obtain a corresponding at least one landmark position;

wherein said determining an estimated gaze position further comprises combining the at least one landmark position with the respective internal representation of the at least one component image, wherein said combining is performed using a landmark neural network comprising one or more fully-connected neural network layers.

19. The computer-implemented method of claim 18, wherein said combining is performed using a landmark neural network comprising one or more fully-connected neural network layers.

20. The computer-implemented method of claim 8, further comprising determining the estimated gaze position relative to a screen of an electronic device using a referential transformation, comprising querying screen properties of the electronic device for performing the referential transformation, and performing user interactions with the electronic device based on the determining the estimated gaze position relative to the screen of the electronic device.