Eye movement detector

• US 4,973,149 A
• Filed: 03/21/1989
• Issued: 11/27/1990
• Est. Priority Date: 08/19/1987
• Status: Expired due to Term

• Current Assignee: University of Virginia
• Original Assignee: The Center For Innovative Technology
• Inventors: Hutchinson, Thomas E.
• Primary Examiner(s): Dzierzynski, Paul M.
• Application Number:
  US07/326,787
• Time in Patent Office: 616 Days
• Field of Search: 351/209, 351/210, 351/246, 354/62, 354/126
• US Class Current: 351/210
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: A61F 4/00 Methods or devices enabling patients or disabled persons to operate an apparatus or a device not forming part of the body operating or control means for prostheses A61F2/48, A61F2/68
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G09B 21/00 Teaching, or communicating with, the blind, deaf or mute audible presentation of material to be studied G09B5/04; devices or methods for replacing direct visual or auditory perception by another kind of perception A61F9/08, A61F11/04; audible indication of meter readings or of colour G01D7/12; watches for blind persons G04B25/02; speech analysis, speech recognition G10L; sound-recording or reproducing, per se G11B

Abstract

A system for eye movement detection is disclosed that utilizes an infrared light emitting diode mounted coaxially in front of the lens of an infrared sensitive video camera for remotely making images of the eye of a computer operator. The reflected light causes bright eye effect which outlines the pupil as brighter than the rest of the eye and also causes an even brighter small glint from the surface of the cornea. The computer includes graphic processing which takes a video image, digitizes it into a matrix of pixels and analyzes the matrix. Using special algorithms the analysis determines the location of the pupil'"'"'s center and the location of the glint relative to each other and with this information determines where the eye is gazing. If the eye-gaze is for a predetermined time at images in selected areas on the computer screen, the area is selected and results in actuation of other devices or the presentation of additional images on the screen. This is especially useable for handicapped persons to control their environment. Other uses include operator interfacing with workstations, cockpit controls and in industrial environments.

Claims

1. An eye movement detector comprising:

a display to be viewed by an operator;

a camera with a lens located near said display and directed in a manner to obtain an image of an operator'"'"'s eye when the display is being viewed;

an illuminator directing an infrared source of illumination coaxial with said camera lens towards the position of an operator'"'"'s eye so that the reflected illumination causes the operator'"'"'s pupil to be brighter than the surrounding part of the eye due to the bright eye effect and also causes a glint reflection from the cornea of the eye;

a computer with memory for analyzing said image of an illuminated operator'"'"'s eye to determine the center of the pupil and location of the glint and utilizing said information in determining the eye-gaze point and correlating said eye-gaze point with where on the display the operator is gazing;

said computer analysis including using said image of the operator'"'"'s eye in a digitized matrix of pixels that is first analyzed by scanning a selected sampling of pixels, which may include every pixel or some lesser number of pixels to determine the boundary as represented by different pixel light intensities between the pupil and the part of eye and face surrounding the pupil; and

said computer analysis further including creating a region of interest in said matrix that surrounds said pupil that is fractionally larger than said pupil and includes said glint with an approximate center of said region of interest being the midpoint of said pupil'"'"'s diameter, said computer then re-scanning the entire region of interest, using said re-scan information in calculating from said re-scan the location of said pupil'"'"'s center and the location of said glint, then determining the pupil-glint displacement between said pupil center location and said glint location and from said pupil-glint displacement calculating said eye-gaze point.

2. The eye movement detector of claim 1, wherein said display contains multiple individual areas and said computer contains in its memory information representing each of said individual areas on the display and said computer determines when said eye-gaze point has lingered on an individual area for greater than a predetermined interval of time and causes an action to take place when this occurs.

3. The eye movement detector of claim 1, wherein said computer includes means for determining a series of eye-gaze points and determining when said eye-gaze points lingers on an individual area longer then a predetermined first sub-interval of time and causing an icon to appear to said screen at said eye-gaze point.

4. The eye movement detector of claim 3, wherein said computer includes means for determining if said eye-gaze point remains unchanged for a second predetermined sub-interval of time and causing a first sound to be emitted when this occurs.

5. The eye movement detector of claim 4, wherein said computer includes means for determining if said eye-gaze point remains unchanged for a third predetermined sub-interval of time which in summary with said first and second sub-intervals equals a full interval of time and causing a second sound to be emitted and said action to take place when this occurs.

6. The eye movement detector of claim 1, wherein said computer includes means for determining subsequent eye-gaze points after a prior eye-gaze point has been determined by determining if the prior pupil center of the prior gaze point is within the new pupil and, if so, utilizing the prior pupil center to erect vertical and horizontal chords through the said new pupil, utilizing said chords in determining an estimate of said new pupil center and if said estimated new pupil center is within said prior pupil, utilizing said estimated new pupil center as the actual new pupil center.

7. The eye movement detector of claim 1, wherein said computer further includes means for calibrating said eye movement detector by displaying at least two calibration points in two separate areas of said display and determining the uncalibrated eye-gaze location when the operator is looking at a calibration point followed by a calculation based on the known location of the calibration look point and the uncalibrated look point of the eye to calculate the calibration coefficients for both horizontal and vertical directions.

8. A method for determining eye-gaze comprising the following steps:

placing on a display multiple individual areas representing different choices that can be made by an operator when viewed by an operator for a predetermined interval of time;

obtain an image of an operator'"'"'s eye by a camera with a lens located near the display and when the display is being viewed;

illuminate the operator'"'"'s eye by directing an infrared source of illumination coaxial with said camera lens towards the position of an operator'"'"'s eye so that the reflected illumination causes the operator'"'"'s pupil to be brighter than the surrounding part of the eye due to the bright eye effect and also causes a glint reflection from the cornea of the eye;

analyze said image of the operator'"'"'s eye by using a computer to determine the center of the pupil and location of the glint and utilize said information to determine the eye-gaze point and correlate said eye-gaze point with whether or not the operator is gazing at an individual area of the display;

using said image of the operator'"'"'s eye to produce a digitized matrix of pixels that is first analyzed by scanning a selected sampling of pixels, which may include every pixel or some lesser number of pixels to determine the boundary as represented by different pixel light intensities between the pupil and the part of eye and face surrounding the pupil; and

said computer analysis further including creating a region of interest in said matrix that surrounds said pupil that is fractionally larger than said pupil and includes said glint with an approximate center of said region of interest being the midpoint of said pupil'"'"'s diameter, said computer then re-scanning the entire region of interest, using said re-scan information in calculating from said re-scan the location of said pupil'"'"'s center and the location of said glint, then determining the pupil-glint displacement between said pupil center location and said glint location and from said pupil-glint displacement calculating said eye-gaze point.

9. The method of determining eye-gaze of claim 8, including determining a series of eye-gaze points and determining when the eye-gaze lingers on an individual area longer than a predetermined first sub-interval of time and causing an icon to appear on said screen at said eye-gaze point.

10. The method of determining eye-gaze of claim 9, including determining if said eye-gaze point remains unchanged for a second predetermined sub-interval of time, causing a first sound to be emitted when this occurs and determining if said eye-gaze point remains unchanging for a third predetermined sub-interval of time which in summary with said first and second sub-intervals equals a full interval of time and causing a second sound to be emitted and said action to take place when this occurs.

11. The method of determining eye-gaze of claim 8 including the steps of determine subsequent eye-gaze points rapidly by first making a determination of whether the prior pupil center of the prior gaze point is within the new pupil and, if so, utilizing the prior pupil center to determine the new pupil center.