Real time eye tracking for human computer interaction

• US 8,885,882 B1
• Filed: 07/16/2012
• Issued: 11/11/2014
• Est. Priority Date: 07/14/2011
• Status: Active Grant

• Current Assignee: The Research Foundation for The State University of New York (State University of New York)
• Original Assignee: The Research Foundation for The State University of New York (State University of New York)
• Inventors: Yin, Lijun, Reale, Michael
• Primary Examiner(s): Drennan, Barry
• Assistant Examiner(s): BEG, SAMAH A
• Application Number:
  US13/549,730
• Time in Patent Office: 848 Days
• Field of Search: None
• US Class Current: 382/103
• CPC Class Codes: G06F 3/00 Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A gaze direction determining system and method is provided. A two-camera system may detect the face from a fixed, wide-angle camera, estimates a rough location for the eye region using an eye detector based on topographic features, and directs another active pan-tilt-zoom camera to focus in on this eye region. A eye gaze estimation approach employs point-of-regard (PoG) tracking on a large viewing screen. To allow for greater head pose freedom, a calibration approach is provided to find the 3D eyeball location, eyeball radius, and fovea position. Both the iris center and iris contour points are mapped to the eyeball sphere (creating a 3D iris disk) to get the optical axis; then the fovea rotated accordingly and the final, visual axis gaze direction computed.

Claims

1. A method for gaze direction estimation, comprising:

detecting at least one iris of an eye in an image;

estimating a 3D eyeball center and radius, a fovea position, and an iris center;

determining an estimated eye gaze directional vector between the fovea position and the iris center, based on at least the estimated 3D eyeball center and radius, fovea position, and iris center;

calibrating the estimated eye gaze directional vector with respect to a known condition, by directing the eye to look into at least one camera and toward at least one calibration point;

solving for t_a using the following quartic equations;

St_a⁴+Tt_a³+Ut_a²+Vt_a+W=0,

S=Y²,

T=−

2TZ−

2(A*B)XY,

U=Z²+2(A*B)XY−

r²y²+x²,

V2r²YZ,

W=−

r²Z²,

X=(A×

B×

A)*P,

Y=(A×

B×

A)*B,

Z=(A×

B×

P)*B;

wherein;

r is a 3D eyeball radius,

E is an estimate for the 3D eyeball center,

E′

is an estimate for the 3D iris center,

P is a calibration point having a known 3D location,

m1 and m2 are iris centers represented in the image,

A and B are normalized vectors between m1 and m2 and the camera,

t_a and t_b represent lengths such that At_a=E and Bt_b=E′

,

wherein ∥

At_a−

Bt_b∥

=r and point E′

(or Bt_b) is a point of intersection between C=At_a−

P and B, equivalent to a point of intersection between B and a plane determined by point P and normal N=(A×

B×

C), such that

2. The method according to claim 1, wherein a pair of eyes are detected in the image, and an eye gaze directional vector determined for each eye, further comprising determining at least one of a statistical composite of the eye gaze directional vector for each eye, and a selected eye gaze directional vector for one eye, to represent the gaze direction.

3. The method according to claim 1, further comprising determining an iris center V, a set of iris contour vectors C_i, and determining an optical axis G such that it is parallel to V and the dot product of G and each C_i is aveDot, using the following linear equations:

4. The method according to claim 1, further comprising detecting a face, and based on the detected face, detecting an eye of the face.

5. The method according to claim 4, further comprising employing a lighting invariant linear-time process and selecting a set of Haar-like features.

6. The method according to claim 5, wherein the Haar-like features are selected using an adaptive boosting process.

7. The method according to claim 4, further comprising terrain mapping the detected face, extracting pit locations as pupil candidates, and selecting an eye pair using a Gaussian Mixture Model (GMM) and facial model-based heuristics.

8. The method according to claim 1, wherein a color filtered channel of an image is selectively processed to detect the iris.

9. The method according to claim 1, wherein the image is searched for specular reflections off the eye.

10. The method according to claim 1, further comprising iteratively seeking the best estimate of eyeball center E and the fovea location, using a binary search process within a range about the initial estimate of E.

11. The method according to claim 1, further comprising:

defining a search region around the estimated iris position, and blurring a grayscale image;

extracting gradient magnitudes and directions from the blurred image;

given a normalized 2D vector H that goes from the iris center estimate to the 2D eyeball center, shooting rays R, outwards around H;

for all pixels (within search region) along each ray, computing score_i,j=m_i,j*(dot(H, D_i,j)>

0.5), where i is a ray index, j is a pixel index, m_i,j is a gradient magnitude, and D_i,j is a normalized gradient direction vector from dark to light pixels;

choosing a highest scoring pixel along each ray as a potential iris contour point;

removing duplicate points caused by overlapping rays;

mapping 2D iris contour points to an eyeball sphere by converting them into 3D perspective vectors and intersecting them with the estimated eyeball sphere, wherein a normalized vector going from the eyeball center to one of those intersection points is an iris contour vector C_i;

computing an average dot product between an eye optical axis and each C_i to estimate a 3D iris radius;

approximating an optical gaze vector by intersecting the 2D iris center with the eyeball sphere, subtracting the eyeball center, and normalizing the result, to provide vector V;

obtaining an average of the dot products (aveDot) between V and each C_i, to define the iris disk;

solving the following linear equations;

12. The method according to claim 1, further comprising mapping the eye gaze directional vector to a display coordinate of a display device.

13. An apparatus for determining an eye gaze direction, comprising:

a camera, a display screen; and

an automated processor, configured to;

detect at least one iris of an eye in an image from the camera, to estimate a 3D eyeball center and radius, a fovea position, and an iris center;

estimate an eye gaze directional vector between the fovea position and the iris center;

calibrate the eye gaze directional vector estimate of the eye by acquiring at least one image each of the eye directed looking into at least one camera and of the eye directed looking toward at least one calibration point;

solve for t_a using the following quartic equations;

St_a⁴+Tt_a³+Ut_a²+Vt_a+W=0,

S=Y²,

T=−

2TZ−

2(A*B)XY,

U=Z²+2(A*B)XY−

r²y²+x²,

V=2r²YZ,

W=−

r²Z²,

X=(A×

B×

A)*P,

Y=(A×

B×

A)*B,

Z=(A×

B×

P)*B;

wherein;

r is a 3D eyeball radius,

E is an estimate for the 3D eyeball center,

E′

is an estimate for the 3D iris center,

P is a calibration point having a known 3D location,

m1 and m2 are iris centers represented in the image,

A and B are normalized vectors between m1 and m2 and the camera,

t_a and t_b represent lengths such that At_a=E and Bt_b=E′

,

wherein ∥

At_a−

Bt_b∥

=r and point E′

(or Bt_b) is a point of intersection between C=At_a−

P and B, equivalent to a point of intersection between B and a plane determined by point P and normal N=(A×

B×

C), such that

14. The apparatus according to claim 13, wherein the automated processor is configured to detect a pair of eyes in the image, determine an eye gaze directional vector for each eye, and determine at least one of a statistical composite of the eye gaze directional vector for each eye and a selected eye gaze directional vector for one eye to represent the eye gaze direction.

15. The apparatus according to claim 13, wherein the automated processor is configured to determine an iris center V, a set of iris contour vectors C_i, and to determine an optical axis G such that it is parallel to V and the dot product of G and each C_i is aveDot, using the following linear equations:

16. The apparatus according to claim 15, wherein the automated processor is further configured to detect a face, and based on the detected face, to detect an eye of the face.

17. The apparatus according to claim 16, wherein the automated processor is further configured to employ a lighting invariant linear-time process and to select a set of Haar-like features.

18. The apparatus according to claim 17, wherein the automated processor is further configured to select the Haar-like features using an adaptive boosting process.

19. The apparatus according to claim 16, wherein the automated processor is further configured to terrain map the detected face, extract pit locations as pupil candidates, and select an eye pair using a Gaussian Mixture Model (GMM) and facial model-based heuristics.

20. The apparatus according to claim 13, wherein the automated processor is further configured to selectively process a color filtered channel of an image to detect the iris.

21. The apparatus according to claim 13, wherein the automated processor is further configured to search the image for specular reflections off the eye.

22. The apparatus according to claim 13, wherein the automated processor is further configured to iteratively seek the best estimate of eyeball center E and the fovea location, using a binary search process within a range about the initial estimate of E.

23. The apparatus according to claim 13, wherein the automated processor is further configured to:

define a search region around the estimated iris position, and blurring a grayscale image;

extract gradient magnitudes and directions from the blurred image;

given a normalized 2D vector H that goes from the iris center estimate to the 2D eyeball center, shoot rays R_i outwards around H;

for all pixels (within search region) along each ray, compute score_i,j=m_i,j*(dot(H,D_i,j)>

0.5), where i is a ray index, j is a pixel index, m_i,j is a gradient magnitude, and D_i,j is a normalized gradient direction vector from dark to light pixels;

choose a highest scoring pixel along each ray as a potential iris contour point;

remove duplicate points caused by overlapping rays;

map 2D iris contour points to an eyeball sphere by converting them into 3D perspective vectors and intersecting them with the estimated eyeball sphere, wherein a normalized vector going from the eyeball center to one of those intersection points is an iris contour vector C_i;

compute an average dot product between an eye optical axis and each C_i to estimate a 3D iris radius;

approximate an optical gaze vector by intersecting the 2D iris center with the eyeball sphere, subtracting the eyeball center, and normalizing the result, to provide vector V;

obtain an average of the dot products (aveDot) between V and each C_i, to define the iris disk;

solve the following linear equations;

24. The apparatus according to claim 13, wherein the automated processor is further configured to map the eye gaze directional vector to a display coordinate of a display device.

25. The apparatus according to claim 24, wherein the automated processor is further configured to define a cursor location associated with the display coordinate.

26. A method for gaze direction estimation, comprising:

detecting at least one iris of an eye in an image;

estimating a 3D eyeball center and radius, a fovea position, and an iris center;

determining an estimated eye gaze directional vector between the fovea position and the iris center, based on at least the estimated 3D eyeball center and radius, the estimated fovea position, and the estimated iris center,

wherein the 3D eyeball center and radius, the fovea position, and the iris center upon which the estimated eye gaze directional vector is based are calibrated by imaging the eye while directing the eye to look into at least one camera and toward at least one calibration point;

solving for t_a using the following quartic equations;

St_a⁴+Tt_a³+Ut_a²+Vt_a+W=0,

S=Y²,

T=−

2TZ−

2(A*B)XY,

U=Z²+2(A*B)XY−

r²y+x²,

V=2r²YZ,

W=−

r²Z²,

X=(A×

B×

A)*P,

Y=(A×

B×

A)*B,

Z=(A×

B×

P)*B;

wherein;

r is a 3D eyeball radius,

E is an estimate for the 3D eyeball center,

E′

is an estimate for the 3D iris center,

P is a calibration point having a known 3D location,

m1 and m2 are iris centers represented in the image,

A and B are normalized vectors between m1 and m2 and the camera,

t_a and t_b represent lengths such that At_a=E and Bt_b=E′

,

wherein ∥

At_a−

Bt_b∥

=r and point E′

(or Bt_b) is a point of intersection between C=At_a−

P and B, equivalent to a point of intersection between B and a plane determined by point P and normal N=(A×

B×

C), such that

27. The method according to claim 26, wherein a pair of eyes are detected in the image, and an eye gaze directional vector determined for each eye, further comprising determining at least one of a statistical composite of the eye gaze directional vector for each eye, and a selected eye gaze directional vector for one eye, to represent the gaze direction.

28. The method according to claim 26, further comprising determining an iris center V, a set of iris contour vectors C_i,and determining an optical axis G such that it is parallel to V and the dot product of G and each C_i is aveDot, using the following linear equations:

29. The method according to claim 26, further comprising detecting a face, and based on the detected face, detecting an eye of the face by employing a lighting invariant linear-time process and selecting a set of Haar-like features using an adaptive boosting process.