Method of detecting eye fixation using image processing

• US 5,818,954 A
• Filed: 02/27/1991
• Issued: 10/06/1998
• Est. Priority Date: 07/14/1988
• Status: Expired due to Fees

• Current Assignee: ATR Communication Systems Research Laboratories
• Original Assignee: ATR Communication Systems Research Laboratories
• Inventors: Tomono, Akira, Iida, Muneo, Ohmura, Kazunori
• Primary Examiner(s): Couso, Jose L.
• Application Number:
  US07/661,367
• Time in Patent Office: 2,778 Days
• Field of Search: 382/6, 382/61, 382/2, 382/68, 382/5
• US Class Current: 382/115
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 10/88 Image or video recognition using optical means, e.g. reference filters, holographic masks, frequency domain filters or spatial domain filters
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

The inventive method calculates a position of the center of the eyeball as a fixed displacement from an origin of a facial coordinate system established by detection of three points on the face, and computes a vector therefrom to the center of the pupil. The vector and the detected position of the pupil are used to determine the visual axis.

Claims

1. In an eye tracking method for detecting eye fixation from an image of a face picked up by an image pickup apparatus, including the steps of detecting a position and a direction of a head from the image of the face picked up by said image pickup apparatus, detecting a pupil of an eye from the image of the face picked up by said image pickup apparatus and calculating eye fixation based on the detected position of the pupil,

the improvement wherein said step of detecting the position and direction of a head comprises the steps of establishing a plane defining a facial coordinate system describing a position of the head and a position and a direction of the face;

determining a position of a center of an eyeball on the facial coordinate system as a function of the position and direction of the face;

identifying a vector connecting the center of the eye ball and a center of the pupil; and

establishing a visual axis identifying the eye fixation based on a position of the pupil of the eye ball and the vector connecting the center of the eyeball thereto.

2. An improved eye tracking method as recited in claim 1, wherein said step of determining the position of the center of the eye ball comprises detecting three dimensional positions of three points on the face to establish said plane defining said facial coordinate system and adding a constant vector thereto.

3. A method for detecting a point of eye fixation, in a predetermined coordinate system using as inputs an image of a face picked up by an image pickup device, comprising the steps of:

using an image processing method for generating a facial coordinate system by finding from said image of the face a spacial position and direction in said predetermined coordinate system of a head including the face imaged by the image pickup device;

estimating a position in the facial coordinate system of a center of rotation of the eyeball in the facial coordinate system;

determining a spacial position in the predetermined coordinate system of the center of the eyeball by converting the position of the center of rotation of the eyeball represented in said facial coordinate system to said predetermined coordinate system;

using said image processing method for extracting an image of a characteristic feature of the eye from said image of the face for finding an angle of rotation of the eyeball in the predetermined coordinate system; and

finding the point of eye fixation from the spacial position of said center of the eyeball and from the angle of rotation of the eyeball.

4. A method according to claim 3, wherein said step of using image processing for extracting the image of the characteristic feature of the eye comprises the step of detecting a position of one of the group consisting of a pupil of the eye, a dark portion of the eye and an image reflected from a cornea of the eye.

5. A method of detecting eye fixation or a point of gazing in a desired coordinate system, comprising the steps of:

describing an object of gazing in the desired coordinate system;

utilizing an image pick up device to pick up an image of a subject'"'"'s face and to provide input image signals,

performing image processing on the input image signals for detecting spatial positions of feature points of said face and a spatial position of a pupil or an iris;

finding position and direction angle of a head portion of the subject from the spatial positions of said feature points of the face for generating a facial coordinate system;

processing the spatial positions of said feature points for setting a constant vector in said facial coordinate system to describe a position of the center of the eyeball;

converting said constant vector to said desired coordinate system; and

detecting, in the desired coordinate system, a connecting line connecting an end point of said constant vector and the position of the pupil or the iris;

generating electrical signals representing the connecting line; and

outputting electrical signals identifying the line of sight as signals representing a line having a predetermined inclination to the connecting line.

6. The method of detecting eye fixation including a step of image processing according to claim 5, wherein said step of detecting spatial positions of said feature points of the face comprises the step of

providing marks facilitating extraction of said spatial positions of said feature points on at least three portions of a frame;

attaching the marked frame to the subject'"'"'s face prior to picking up said image of the subject'"'"'s face by the image pickup device.

7. The method of detecting eye fixation including a step of image processing according to claim 5, wherein said step of outputting electrical signals identifying the line of sight comprises outputting electrical signals representing said line of sight as a line having a predetermined inclination relative to said connecting line.

8. The method of detecting eye fixation including a step of image processing according to claim 5, comprising the further step of inputting the outputted electrical signals identifying the line of sight to a man-machine interface for application to an input device thereof responsive to eye fixation.

9. The method of detecting eye fixation according to claim 5, wherein said predetermined inclination is zero, the outputting step comprising outputting electrical signals representing the connecting line as said line of sight.

10. The method of detecting eye fixation according to claim 5, further comprising the steps of:

preparing at least two points having known positions in the desired coordinate system;

measuring positions of the feature points of the face, of the pupil or of the iris when the subject gazes at said point of gazing; and

wherein said step of setting a constant vector in said facial coordinate system to describe a position of the center of the eyeball comprises;

a) assuming that the center of the eyeball is on a line coupling said point and the center of said pupil or on a line having a prescribed inclination therefrom; and

b) estimating said constant vector based on said assumption; and

wherein said outputting step comprises outputting electrical signals representing said line of sight as a line having a predetermined inclination to the connecting line connecting an end point of said estimated constant vector and the position of the pupil or the iris.

11. A non-contact method of determining a point of eye fixation for input to a man-machine interface between a subject and a utilizing device, comprising the steps of:

using an imaging device to form an image of a face and an eye of the subject and to provide image signals representative thereof;

using a programmed computer to perform image processing of the image signals, the computer programmed for performing the steps of;

a) extracting three fixed image points on the subject'"'"'s face from the image signals;

b) identifying a plane defined by the three fixed points;

c) defining a facial coordinate system including the plane;

d) establishing a position of a center of an eyeball of the user as a fixed displacement from an origin of the facial coordinate system;

e) extracting a position of a center of a pupil of the eyeball from the image signals;

f) defining a vector from the center of the eyeball to the center of the pupil of the eyeball,

g) determining an orientation of a visual axis corresponding to a line of sight in accordance with the vector, and

h) generating output signals identifying the visual axis for input to the man-machine interface.

12. The method recited in claim 11, wherein said step of using an imaging device comprises using at least two imaging devices to form two images of a face and an eye of the subject and to provide image signals representative thereof;

processing the image signals to generate differential signals representing a differential image obtained from said two images to emphasize signals representing a pupil of the eye relative to background noise in the image signals and to permit image extraction to be performed on the image signals; and

using the programmed computer to process information from the differential signals for extracting the three fixed image points and for extracting the position of the center of the pupil of the eyeball from the image signals.