Eye movement detector

• US 4,836,670 A
• Filed: 08/19/1987
• Issued: 06/06/1989
• Est. Priority Date: 08/19/1987
• Status: Expired due to Term

• Current Assignee: University of Virginia
• Original Assignee: The Center For Innovative Technology
• Inventors: Hutchinson, Thomas E.
• Primary Examiner(s): Bovernick, Rodney B.
• Assistant Examiner(s): Dzierzynski, P. M.
• Application Number:
  US07/086,809
• Time in Patent Office: 657 Days
• Field of Search: 351/209, 351/210, 351/426, 354/62, 354/126
• US Class Current: 351/210
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: A61F 4/00 Methods or devices enabling patients or disabled persons to operate an apparatus or a device not forming part of the body operating or control means for prostheses A61F2/48, A61F2/68
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G09B 21/00 Teaching, or communicating with, the blind, deaf or mute audible presentation of material to be studied G09B5/04; devices or methods for replacing direct visual or auditory perception by another kind of perception A61F9/08, A61F11/04; audible indication of meter readings or of colour G01D7/12; watches for blind persons G04B25/02; speech analysis, speech recognition G10L; sound-recording or reproducing, per se G11B

Abstract

A system for eye movement detection is disclosed that utilizes an infrared light emitting diode mounted coaxially in front of the lens of an infrared sensitive video camera for remotely making images of the eye of a computer operator. The reflected light causes bright eye effect which outlines the pupil as brighter than the rest of the eye and also causes an even brighter small glint from the surface of the cornea. The computer includes graphic processing which takes a video image, digitizes it into a matrix of pixels and analyzes the matrix. Using special algorithms the analysis determines the location of the pupil'"'"'s center and the location of the glint relative to each other and with this information determines where the eye is gazing. If the eye-gaze is for a predetermined time at images in selected areas on the computer screen, the area is selected and results in actuation of other devices or the presentation of additional images on the screen. This is especially useable for handicapped persons to control their environment. Other uses include operator interfacing with workstations, cockpit controls and in industrial environments.

Claims

1. An eye movement detector comprising:

a display to be viewed by an operator;

a camera with a lens located near said display and directed in a manner to obtain an image of an operator'"'"'s eye when the display is being viewed;

a small illuminator mounted in front of said camera lens and not further from the axis of said lens than one-half the distance from the center of said lens and the edge thereof and out of said lens'"'"' focus directed towards the position of an operator'"'"'s eye so that the reflected illumination causes the operator'"'"'s pupil to be brighter than the surrounding part of the eye due to the bright eye effect; and

a computer for analyzing the image of an operator'"'"'s eye to determine the center of the pupil and utilize said information to detect the eye position and correlate said position with whether or not the operator is gazing at the display.

2. The eye movement detector of claim 1, wherein there are multiple individual areas on the display representing different choices that can be made by an operator and said computer analysis determines at which individual area the operator is gazing.

3. The eye movement detector of claim 2, wherein said illuminator also causes a glint reflection from the cornea of the eye with said computer determining the gaze point by the displacement between the location of said glint and location of said center of the pupil relative to one another.

4. The eye movement detector of claim 3, wherein said illuminator is a near infrared light emitting semiconductor and said camera is sensitive to the near infrared wavelengths.

5. The eye movement detector of claim 4, wherein said infrared light emitting semiconductor is a gallium arsenide diode emitting light of a wavelength in the approximate range of 900 nanometers.

6. The eye movement detector of claim 4, wherein said computer analysis includes taking an image of the illuminated eye and placing it in a digitized matrix of pixels that is first analyzed by scanning a selected sampling of pixels, which may include every pixel or some lesser number of pixels, and determining with respect to each scanned pixel the light intensity value of the pixel and the frequency of occurrence of each pixel having the same intensity and making a determination of the pupil threshold by determining the minimum number of occurrences of pixel intensity of the same value between the pixel intensity represented by the pupil and the pixel intensity represented by the part of the eye and face surrounding the pupil.

7. The eye movement detector of claim 6, wherein the computer analysis further includes determining the glint threshold by finding the pixel intensity level between the maximum intensity of pixels representing the pupil and the minimum intensity of the pixels representing the glint.

8. The eye movement detector of claim 7, wherein said pupil-glint displacement is determined by first scanning the pixels in the matrix by a fine scan of the pixels in a given row and a rough scan of the number of rows to determine a first chord of predetermined length passing through the pupil image followed by determining the diameter of said pupil from a chord through the pupil at right angles to the midpoint of said first chord.

9. The eye movement detector of claim 8, wherein said fine scan is approximately every other pixel in a given row and said rough scan is approximately every 32 rows.

10. The eye movement detector of claim 9, wherein said computer creating a region of interest in said matrix that surrounds said pupil that is fractionally larger than said pupil with an approximate center of said region of interest being the midpoint of said pupil'"'"'s diameter, said computer then re-scanning the entire region of interest at every pixel, using said re-scan information in calculating from said re-scan the location of said pupil'"'"'s center and the location of said glint, then determining the pupil-glint displacement between said pupil center location and said glint location and from said pupil-glint displacement calculating the eye-gaze point.

11. The eye movement detector of claim 10, wherein said computer contains in its memory information representing each of said individual areas on the display, determining when said eye-gaze point has lingered on an individual area for greater than a predetermined interval of time and causing an action to take place when this occurs.

12. The eye movement detector of claim 11, wherein said computer determining a series of eye-gaze points and determining when said eye-gaze point lingers on an individual area longer than a predetermined first sub-interval of time and causing an icon to appear on said screen at said eye-gaze point.

13. The eye movement detector of claim 12, wherein said computer determining if said eye-gaze point remains unchanged for a second predetermined sub-interval of time and causing a first sound to be emitted when this occurs.

14. The eye movement detector of claim 13, wherein said computer determining if said eye-gaze point remains unchanging for a third predetermined sub-interval of time which in summary with said first and second sub-intervals equals a full interval of time and causing a second sound to be emitted and said action to take place when this occurs.

15. An eye movement detector comprising:

a display having multiple individual areas representing different choices that can be made by an operator when viewed by an operator for a predetermined interval of time;

a camera with a lens located near said display and directed in a manner to obtain an image of an operator'"'"'s eye when the display is being viewed;

an illuminator directing an infrared source of illumination coaxial with said camera lens towards the position of an operator'"'"'s eye so that the reflected illumination causes the operator'"'"'s pupil to be brighter than the surrounding part of the eye due to the bright eye effect and also causes a glint reflection from the cornea of the eye;

a computer for analyzing said image of an illuminated operator'"'"'s eye to determine the center of the pupil and location of the glint and utilizing said information in determining the eye-gaze point and correlating said eye-gaze point with whether or not the operator is gazing at an individual area of the display; and

said computer analysis including using said image of the operator'"'"'s eye in a digitized matrix of pixels that is first analyzed by scanning a selected sampling of pixels, which may include every pixel or some lesser number of pixels, and determining with respect to each scanned pixel the light intensity value of the pixel and the frequency of occurrence of each pixel having the same intensity and making a determination of the pupil intensity threshold by determining the minimum number of occurrences of pixel intensity of the same value between the pixel intensity represented by the pupil and the pixel intensity represented by the part of the eye and face surrounding the pupil and also determining the glint intensity threshold by finding the pixel intensity level between the maximum intensity of pixels representing the pupil and the minimum intensity of the pixels representing the glint.

16. The eye movement detector of claim 15, wherein said eye-gaze point is determined by utilizing said pupil threshold and said glint threshold by the computer first scanning the pixels in said matrix by a fine scan of the pixels in a given row and a rough scan of the number of rows to determine a first chord of predetermined length passing through the pupil image followed by determining the diameter of said pupil from a chord through the pupil at right angles to the midpoint of said first chord.

17. The eye movement detector of claim 16, wherein said fine scan is approximately every other pixel in a given row and said rough scan is approximately every 32 rows.

18. The eye movement detector of claim 17, wherein said computer creating a region of interest in said matrix that surrounds said pupil that is fractionally larger than said pupil with an approximate center of said region of interest being the midpoint of said pupil'"'"'s diameter, said computer then re-scanning the entire region of interest at every pixel, using said re-scan information in calculating from said re-scan the location of said pupil'"'"'s center and the location of said glint, then determining the pupil-glint displacement between said pupil center location and said glint location and from said pupil-glint displacement calculating said eye-gaze point.

19. The eye movement detector of claim 18, wherein said computer contains in its memory information representing each of said individual areas on the display, determining when said eye-gaze point has lingered on an individual area for greater than a predetermined interval of time and causing an action to take place when this occurs.

20. The eye movement detector of claim 19, wherein said computer determining a series of eye-gaze points and determining when said eye-gaze point lingers on an individual area longer than a predetermined first sub-interval of time and causing an icon to appear on said screen at said eye-gaze point.

21. The eye movement detector of claim 20, wherein said computer determining if said eye-gaze point remains unchanged for a second predetermined sub-interval of time and causing a first sound to be emitted when this occurs.

22. The eye movement detector of claim 21, wherein said computer determining if said eye-gaze point remains unchanging for a third predetermined sub-interval of time which in summary with said first and second sub-intervals equals a full interval of time and causing a second sound to be emitted and said action to take place when this occurs.

23. The eye movement detector of claim 19 wherein after determining an eye-gaze point, rapidly determining subsequent eye-gaze points by said computer determining if the prior pupil center of the prior gaze point is within the new pupil and, if so, utilizing the prior pupil center to erect vertical and horizontal chords through the said new pupil, utilizing said chords in determining an estimate of said new pupil center and if said estimated new pupil center is within said prior pupil, utilizing said estimated new pupil center as the actual new pupil center.

24. A method for determining eye-gaze comprising the follows steps:

placing on a display multiple individual areas representing different choices that can be made by an operator when viewed by an operator for a predetermined interval of time;

obtain an image of an operator'"'"'s eye by a camera with a lens located near the display and when the display is being viewed;

illuminate the operator'"'"'s eye by directing an infrared source of illumination coaxial with said camera lens towards the position of an operator'"'"'s eye so that the reflected illumination causes the operator'"'"'s pupil to be brighter than the surrounding part of the eye due to the bright eye effect and also causes a glint reflection from the cornea of the eye;

analyze said image of the operator'"'"'s eye to determine the center of the pupil and location of the glint and utilize said information to determine the eye-gaze point and correlate said eye-gaze point with whether or not the operator is gazing at an individual area of the display; and

using said image of the operator'"'"'s eye to produce a digitized matrix of pixels that is first analyzed by scanning a selected sampling of pixels, which may include every pixel or some lesser number of pixels, and determine with respect to each scanned pixel the light intensity value of the pixel and the frequency of occurrence of each pixel having the same intensity and make a determination of the pupil intensity threshold by determining the minimum number of occurrences of pixel intensity of the same value between the pixel intensity represented by the pupil and the pixel intensity represented by the part of the eye and face surrounding the pupil and also determine the glint intensity threshold by finding the pixel intensity level between the maximum intensity of pixels representing the pupil and the minimum intensity of the pixels representing the glint.

25. The method of determining eye-gaze of claim 24, including first scanning the pixels in the matrix by a fine scan of the pixels in a given row and a rough scan of the number of rows to determine a first chord of predetermined length passing through the pupil image followed by determining the diameter of said pupil from a chord through the pupil at right angles to the midpoint of said first chord.

26. The method of determining eye-gaze of claim 25, including creating a region of interest in the matrix that surrounds the pupil that is fractionally larger than the pupil with an approximate center of said region of interest being the midpoint of said pupil'"'"'s diameter, then re-scanning the entire region of interest at every pixel, calculate from said re-scan information the location of said pupil'"'"'s center and the location of said glint, then determine the pupil-glint displacement between said pupil center location and said glint location and from said pupil-glint displacement calculate the eye-gaze point.

27. The method of using as a computer interface the eye gaze determination of claim 26, including placing in the computer memory information representing each of the individual areas on the display, determine when said eye-gaze point has lingered on an individual area for greater than a predetermined interval of time and cause an action to take place when this occurs.

28. The method of using eye-gaze as a computer interface of claim 27, including determining a series of eye-gaze points and determine when the eye-gaze lingers on an individual area longer than a predetermined first sub-interval of time and cause an icon to appear on said screen at said eye-gaze point.

29. The method of using eye-gaze as a computer interface of claim 28, including determining if said eye-gaze point remains unchanged for a second predetermined sub-interval of time, cause a first sound to be emitted when this occurs and determine if said eye-gaze point remains unchanging for a third predetermined sub-interval of time which in summary with said first and second sub-intervals equals a full interval of time and cause a second sound to be emitted and said action to take place when this occurs.

30. The method of determining eye-gaze of claim 24 including the steps of determine subsequent eye-gaze points rapidly by first making a determination of whether the prior pupil center of the prior gaze point is within the new pupil and, if so, utilize the prior pupil center to erect vertical and horizontal chords through the new pupil then utilize said chords to determine an estimate of the new pupil center and, if said estimated new pupil center is within said prior pupil, utilize said estimated new pupil center as the actual new pupil center.

31. For use in an eye movement detector, a camera lens and illuminator subassembly comprising:

a solid state infrared sensitive camera with a lens for use in obtaining an image of an operator'"'"'s eye;

a small infrared illuminator mounted coaxially with and in the center and front of said camera lens; and

said illuminator having a lens to form and direct the infrared radiation outward from the front of said camera lens.