Eye detection unit using sequential data flow

• US 8,610,768 B2
• Filed: 07/09/2010
• Issued: 12/17/2013
• Est. Priority Date: 07/16/2009
• Status: Active Grant

• Current Assignee: Tobii Technology AB
• Original Assignee: Tobii Technology AB
• Inventors: Holmberg, Johnny, Blixt, Peter, Elvesj, John
• Primary Examiner(s): Czekaj, David
• Assistant Examiner(s): PHAM, NAM D
• Application Number:
  US12/833,575
• Publication Number:
  US 20110013007A1
• Time in Patent Office: 1,257 Days
• Field of Search: 348/78
• US Class Current: 348/78
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

The invention generally relates to systems and methods for eye detection for use in, for example, installations and systems for tracking eyes and gaze angle/directions. The system includes a plurality of eye detection units connected to an image sensor. The eye detection units are adapted to receive regions of picture elements of a digital image frame substantially in parallel and wherein the eye detection units are adapted to operate substantially in parallel, each eye detection unit being adapted to receive a specified region of the picture elements of at least one digital image frame and including an eye feature extraction module adapted to perform a eye feature extraction process on the received picture elements to identify predetermined eye features. An eye data generating module is adapted to receive information of potential identified eye features including picture element coordinates for the identified eye features, and the received region of picture elements, and generate eye data based on the information of potential eye features, wherein the eye data is generated if identified eye features fulfill predetermined conditions, and transfer generated eye data and the received regions of picture elements to a processing module, and wherein a data flow of the eye data and the region of picture elements of a digital image frame is substantially parallel.

Claims

1. An eye detection system comprising:

at least one illuminator element adapted to emit light in a direction towards the head of a user;

at least one image sensor being adapted to receive light from the illuminated head of a user and repeatedly capture pictures, each being represented by a digital image frame comprising a matrix of picture elements;

a plurality of eye detection units connected to said image sensor, said eye detection units being adapted to receive a selected data flow including at least one row or column of picture elements of a digital image frame, said digital frame being divided in multiple data flows, substantially in parallel from said image sensor and wherein said eye detection units are adapted to operate substantially in parallel, each eye detection unit being adapted to receive a selected data flow of the picture elements of at least one digital image frame and including an eye feature extraction module adapted to perform a eye feature extraction process on the received picture elements to identify predetermined eye features; and

an eye data generating module connected to said eye detection units and being adapted to;

receive information of potential identified eye features from said eye detection units, said information including picture element coordinates for said identified eye features, and the received region of picture elements,

generate eye data based on said information of potential eye features, wherein said eye data is generated if identified eye features fulfil predetermined conditions, and

transfer generated eye data and the received regions of picture elements to a processing module, and wherein a data flow of said eye data and the region of picture elements of a digital image frame is substantially parallel; and

said eye feature extraction module comprising a glint extraction module adapted to perform a glint extraction process on the received picture elements to identify picture elements including at least a part of a potential glint of an eye of the user, with said glint extraction module being configured to

determine edge values of the respective picture elements of a received subset of picture elements, and

determine a potential glint by connecting picture elements of the subset having correlated edge values.

2. The eye detection system according to claim 1, wherein said eye feature extraction module is adapted to:

process each intensity value of the picture elements of a received data flow of picture elements;

sequentially determine whether adjacent picture elements of a row or column of the data flow of picture elements are correlated by comparing the processed intensity value of said picture elements, wherein picture elements are determined to be correlated if the respective processed intensity value fulfils predetermined conditions;

determine whether correlated elements in a row or a column are correlated with elements of a former row or column, respectively;

connect such correlated elements into sub-regions of elements; and

determine such sub-regions to include potential eye features.

3. The eye detection system according to claim 1, wherein said glint extraction module is adapted, in said glint extraction process, to:

perform an edge calculation to determine edge values of the respective picture elements of a received data flow of picture elements;

sequentially determine whether adjacent picture elements of a row or a column of the data flow of picture elements are correlated by comparing the respective edge value, wherein picture elements are determined to be correlated if the edge value is the same;

determine whether correlated elements in a row or a column is correlated with elements of a former row or column, respectively;

connect such correlated elements into sub-regions of elements; and

determine such sub-regions to be a potential glint.

4. The eye detection system according to claim 1, wherein said eye feature extraction module includes a pupil extraction module adapted to perform a pupil region extraction process on the received picture elements to identify picture elements including at least a part of a potential pupil region of an eye of the user, wherein said pupil extraction module is adapted, in said pupil extraction process, to:

sequentially determine whether adjacent picture elements of a row or a column of the received at a flow of picture elements are correlated by comparing the respective intensity value, wherein two elements are determined to be correlated if the intensity values are within a predetermined interval;

determine whether correlated elements in a row or a column is correlated with elements of a former row or column, respectively;

connect such correlated elements into sub-regions of elements; and

determine such sub-regions to be at least a potential pupil.

5. The eye detection unit according to claim 2, wherein said eye data generating module is adapted to generate eye data if the picture elements including at least a part of a potential pupil and the picture elements including at least a part of a potential glint is within a predetermined distance from each other in a sub-region.

6. The eye detection unit according to claim 5, wherein said eye data generating module is adapted to:

map sub-regions determined to be at least a part of a potential pupil and sub-regions determined to be a potential glint to establish whether there is an overlap or close relation in distance; and

if such an overlap or close relation in distance is established between at least a part of a potential pupil and at least one glint, then determine that the sub-regions including said at least a part of a potential pupil and said at least one glint constitute an eye.

7. The eye detection unit according to claim 1, wherein said eye data generating module is adapted use information about in which picture elements eye features were identified in a data flow at least one preceding digital image frame when identifying eye features in a data flow of a present digital image.

8. The eye detection unit according to claim 1, wherein said eye data generating module is adapted to send information to the image sensor instructing the sensor to forward picture elements of a data flow of at least one subsequent digital image frame that corresponds to picture elements in which eye features were identified in a data flow of a preceding digital image frame to at least one eye detection unit.

9. A method for detecting an eye using an eye detection system comprising at least one illuminator element adapted to emit light in a direction towards the head of a user;

and at least one image sensor being adapted to receive light from the illuminated head of a user and repeatedly capture pictures, each picture being represented by a digital image frame comprising a matrix of picture elements, said method comprising the steps of;

providing selected data flows including at least one row or column of picture elements of a digital image frame divided in multiple data flows in parallel to a plurality of eye detection units;

performing parallel eye feature extraction processes on the received picture elements to identify predetermined eye features in said selected data flows in eye feature extracting modules, comprising

determining edge values of the respective picture elements of a received subset of picture elements, and

determining a potential glint by connecting picture elements of the subset having correlated edge values;

sending information of potential identified eye features from said eye detection units, said information including picture element coordinates for said identified eye features, and the received region of picture elements to an eye data generating module;

generating eye data based on said information of potential eye features, wherein said eye data is generated if identified eye features fulfil predetermined conditions; and

transferring generated eye data and the received regions of picture elements to a processing module, and wherein a data flow of said eye data and the region of picture elements of a digital image frame are substantially parallel.

10. The method according to claim 9, wherein the step of performing parallel eye feature extracting processes includes the steps of:

processing each intensity value of the picture elements of a received data flow of picture elements;

sequentially determining whether adjacent picture elements of a row or column of the data flow of picture elements are correlated by comparing the processed intensity value of said picture elements, wherein picture elements are determined to be correlated if the respective processed intensity value fulfils predetermined conditions;

determining whether correlated elements in a row or column is correlated with elements of a former row or column, respectively;

connecting such correlated elements into sub-regions of elements; and

determining such sub-regions to include potential eye features.

11. The method according to claim 9, wherein each eye feature extraction process includes a glint extraction process including the steps of:

performing an edge calculation to determine edge values of the respective picture elements of a received data flow of picture elements;

sequentially determining whether adjacent picture elements of a row or a column of the data flow of picture elements are correlated by comparing the respective edge value, wherein picture elements are determined to be correlated if the edge value is the same;

determining whether correlated elements in a row or a column is correlated with elements of a former row or column, respectively;

connecting such correlated elements into sub-regions of elements; and

determining such sub-regions to be a potential glint.

12. The method according to claim 9, wherein said eye feature extraction process includes a pupil extraction process including the steps of:

sequentially determining whether adjacent picture elements of a row or column of the received data flow of picture elements are correlated by comparing the respective intensity value, wherein two elements are determined to be correlated if the intensity values are within a predetermined interval;

determining whether correlated elements in a row or column is correlated with elements of a former row or column, respectively;

connecting such correlated elements into sub-regions of elements; and

determining such sub-regions to be at least a potential pupil.

13. The method according to claim 10, wherein said eye data generating step includes generating eye data if the picture elements including at least a part of a potential pupil and the picture elements including at least a part of a potential glint is within a predetermined distance from each other in a sub-region.

14. The method according to claim 13, wherein said eye data generating step includes the steps of:

mapping sub-regions determined to be at least a part of a potential pupil and sub-regions determined to be a potential glint to establish whether there is an overlap or close relation in distance; and

if such an overlap or close relation in distance is established between at least a part of a potential pupil and at least one glint, then determine that the sub-regions including said at least a part of a potential pupil and said at least one glint constitute an eye.

15. The method according to claim 9, wherein said eye data generating step includes the step of using information about in which picture elements eye features were identified in a data flow at least one preceding digital image frame when identifying eye features in a data flow of a present digital image.

16. The method according to claim 9, further including the steps of controlling said the image sensor to forward picture elements of a data flow of at least one subsequent digital image frame that corresponds to picture elements in which eye features were identified in a data flow of a preceding digital image frame to at least one eye detection unit.

17. An eye detection system comprising:

a plurality of eye detection units connectable to at least one image sensor being adapted to receive light from the illuminated head of a user and repeatedly capture pictures, each being represented by a digital image frame comprising a matrix of picture elements, said eye detection units being adapted to receive data flows including at least one row or column of picture elements of a digital image frame divided in multiple data flows substantially in parallel and wherein said eye detection units are adapted to operate substantially in parallel, each eye detection unit being adapted to receive a selected data flow of the picture elements of at least one digital image frame and including an eye feature extraction module adapted to perform a eye feature extraction process on the received picture elements to identify predetermined eye features; and

an eye data generating module connected to said eye detection units and being adapted to;

receive information of potential identified eye features from said eye detection units, said information including picture element coordinates for said identified eye features, and the received region of picture elements,

generate eye data based on said information of potential eye features, wherein said eye data is generated if identified eye features fulfil predetermined conditions, and

transfer generated eye data and the received regions of picture elements to a processing module, and wherein a data flow of said eye data and the region of picture elements of a digital image frame is substantially parallel; and

said eye feature extraction module comprising a glint extraction module adapted to perform a glint extraction process on the received picture elements to identify picture elements including at least a part of a potential glint of an eye of the user, with said glint extraction module being configured to

determine edge values of the respective picture elements of a received subset of picture elements, and

determine a potential glint by connecting picture elements of the subset having correlated edge values.

18. The eye detection system according to claim 17, wherein said eye feature extraction module is adapted to:

process each intensity value of the picture elements of a received data flow of picture elements;

sequentially determine whether adjacent picture elements of a row or column of the data flow of picture elements are correlated by comparing the processed intensity value of said picture elements, wherein picture elements are determined to be correlated if the respective processed intensity value fulfils predetermined conditions;

determine whether correlated elements in a row or a column is correlated with elements of a former row or column, respectively;

connect such correlated elements into sub-regions of elements; and

determine such sub-regions to include potential eye features.

19. The eye detection system according to claim 17, wherein said eye feature extraction module includes a glint extraction module adapted to perform a glint extraction process on the received picture elements to identify picture elements including at least a part of a potential glint of an eye of the user, wherein said glint extraction module is adapted, in said glint extraction process, to:

perform an edge calculation to determine edge values of the respective picture elements of a received data flow of picture elements;

sequentially determine whether adjacent picture elements of a row or a column of the data flow of picture elements are correlated by comparing the respective edge value, wherein picture elements are determined to be correlated if the edge value is the same;

determine whether correlated elements in a row or a column is correlated with elements of a former row or column, respectively;

connect such correlated elements into sub-regions of elements; and

determine such sub-regions to be a potential glint.

20. The eye detection system according to claim 17, wherein said eye feature extraction module includes a pupil extraction module adapted to perform a pupil region extraction process on the received picture elements to identify picture elements including at least a part of a potential pupil region of an eye of the user, wherein said pupil extraction module is adapted, in said pupil extraction process, to:

sequentially determine whether adjacent picture elements of a row or a column of the received data flow of picture elements are correlated by comparing the respective intensity value, wherein two elements are determined to be correlated if the intensity values are within a predetermined interval;

determine whether correlated elements in a row or a column is correlated with elements of a former row or column, respectively;

connect such correlated elements into sub-regions of elements; and

determine such sub-regions to be at least a potential pupil.

21. The eye detection unit according to claim 18, wherein said eye data generating module is adapted to generate eye data if the picture elements including at least a part of a potential pupil and the picture elements including at least a part of a potential glint is within a predetermined distance from each other in a sub-region.

22. The eye detection unit according to claim 21, wherein said eye data generating module is adapted to:

map sub-regions determined to be at least a part of a potential pupil and sub-regions determined to be a potential glint to establish whether there is an overlap or close relation in distance; and

if such an overlap or close relation in distance is established between at least a part of a potential pupil and at least one glint, then determine that the sub-regions including said at least a part of a potential pupil and said at least one glint constitute an eye.

23. The eye detection unit according to claim 17, wherein said eye data generating module is adapted use information about in which picture elements eye features were identified in a data flow at least one preceding digital image frame when identifying eye features in a data flow of a present digital image.

24. The eye detection unit according to claim 17, wherein said eye data generating module is adapted to send information to the image sensor instructing the sensor to forward picture elements of a data flow of at least one subsequent digital image frame that corresponds to picture elements in which eye features were identified in a data flow of a preceding digital image frame to at least one eye detection unit.