Apparatus for measuring eye gaze and fixation duration, and method therefor

• US 5,583,795 A
• Filed: 03/17/1995
• Issued: 12/10/1996
• Est. Priority Date: 03/17/1995
• Status: Expired due to Term

• Current Assignee: The United States of America As Represented By The Secretary of Agriculture
• Original Assignee: The United States Of America As Represented By The Secretary Of The Army
• Inventors: Smyth, Christopher C.
• Primary Examiner(s): Voeltz, Emanuel T.
• Assistant Examiner(s): MILLER, CRAIG S
• Application Number:
  US08/407,142
• Time in Patent Office: 634 Days
• Field of Search: 351/209, 351/211, 351/212, 351/246, 340/825.19, 348/15, 345/7, 356/139.03, 364/516, 364/571.01, 364/559
• US Class Current: 702/150
• CPC Class Codes: A61B 3/0025 characterised by electronic signal processing, e.g. eye models
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 5/30 Polarising elements light-modulating devices with active elements G02F1/00

Abstract

The invention is an apparatus for measuring eye gaze and fixation duration employing an electronic video display array as sequential sources of eye illumination. The video display of the apparatus may be task related or near infrared light sources used merely to track the user'"'"'s eye. The apparatus of the invention is based on an opto-transistor array, a comparator array and an encoder and latch clocked by the raster-scan pulses of the display driver, which are configured to construct a pairing table of sequential source corneal reflections to sensor activations over the display field refresh cycle. An accurate three dimensional ocular model is computed from opthalmometric techiniques from which the viewing direction and fixation duration are derived. The apparatus is easily extended to the simultaneous tracking of both eyes and is designed to be used with head-mounted video displays. The accuracy of the apparatus is independent of shifts of the helmet on the user'"'"'s head. Additionally, the apparatus is readily incorporated into panel mounted displays and head-free sensor arrays. The apparatus can be used as an eyetracker to control computerized machinery by ocular gaze point of regard and fixation duration.

Claims

1. An apparatus for measuring the ocular gaze points of regard and fixation durations and binocular convergence point of a viewer, comprising:

a) a headset;

b) a position and orientation sensor having an electrical output, wherein said sensor is constructed so as to connect to said headset, said position and orientation sensor comprising means to determine position and orientation of said headset on head of said viewer;

c) an optical unit constructed so as to provide a clocked raster-scan pulse electrical output and further constructed so as to provide a clocked display field refresh pulse electrical output having an optical output, said optical unit is constructed so as to connect to said headset so as to display a visual image onto eyes of said viewer, and further constructed so as to optically output cornea and pupil reflections of the viewer'"'"'s eyes;

d) an opto-electronic device constructed so as to receive an optical input, said optical input connected to said optical output of said optical unit, said opto-electronic device further constructed so as to receive a clocked raster scan pulse electrical input from said optical unit, said opto-electronic device further constructed so as to provide an analog electrical output, said opto-electronic device further constructed so as to provide a digital electrical output for transmitting an electrical signal said opto-electronic device further comprising means for electronically representing spatially optical corneal reflection transmitted from said optical unit;

e) a digital processor constructed so as to receive a clocked raster-scan pulse electrical input from said optical unit, said digital processor further constructed so as to receive an electrical digital input, said electrical digital input connected to the digital output from said opto-electronic device, said processor further constructed to read an output from said opto-electronic device on each clock-pulse and further constructed to write said output from said opto-electronic device to a double-buffered digital memory;

f) an image processor constructed so as to receive an electrical analog input, said electrical analog input connected to said analog output of said opto-electronic device, said image processor further constructed so as to receive a clocked display field refresh pulse input from said optical unit, said image processor further constructed so as to provide a memory mapped digital output of pupil image coordinates and principal axes following each of a clocked field refresh pulse;

g) a digital computer, constructed so as to receive an input connected to the output of said position and orientation sensor, said digital computer comprising means to access an output of said double-buffered digital memory of said digital processor, said digital computer constructed so as to access the memory mapped output of said image processor, said digital computer further constructed so as to receive a clocked display field refresh pulse input, said digital computer further constructed so as to provide a digital output for each of calibration, computed gaze points of regard and fixation durations, and binocular convergence point of the viewer, following said each clocked field refresh pulse, during application.

2. The apparatus as recited in claim 1, wherein said headset further comprises:

a) head and hat bands each having contact pads, said contact pads constructed to be positioned on load bearing portions of the viewer'"'"'s face and head, said load bearing portions comprising at least one of cheek bones, temporal regions, top of the head, back of the head, and inion budge at rear of skull;

b) said hat bands further comprising an additional contact pad constructed so as to force the hat bands to stand off of the viewer'"'"'s forehead, said pad further constructed so as to prevent eyebrow movement from disturbing said headset;

c) a nose bridge in front and pivot joints at the sides of the said hat bands, said nose bridge further constructed so as to support said optical unit.

3. The apparatus as recited in claim 1, wherein said optical unit further comprises:

a) a video display unit, constructed so as to provide a clocked raster scan pulse electrical output and a clocked display field refresh pulse electrical output;

b) a dielectric polarizer disposed in path of light emanating from said video display, said polarizer constructed so as to polarize light emanating from said video display;

c) an optical system said optical system constructed so as to direct and focus light emanating from said video display onto eyes of the viewer;

d) a quarter-waveplate, positioned in path of polarized light reaching the eyes of the viewer from said optical system and further positioned in path of light reflected back from the eyes, said quarter-waveplate further constructed so as to rotate polarized light one-half wavelength;

e) a semi-reflective mirror positioned in path of polarized light reflected back from the eyes of the viewer through said waveplate, said mirror constructed so as to reflect polarized light in a direction away from the eyes of the viewer;

f) an optical lens system, having an optical output, positioned in path of light reflecting from the eyes of the viewer after light reflecting from the eyes has passed through said waveplate and reflected from the said semi-reflective mirror, said lens system constructed so as to optically filter and focus corneal and pupil reflections of the eyes of the viewer in response to illumination caused by said video display.

4. The apparatus as recited in claim 2, wherein said video display further comprises:

a) an array of light emitting elements wherein said elements are selected from the group consisting of active emitter, passive emitter including active matrix, light emitting diode, or a combination thereof;

b) an electronic display driver wherein said driver is constructed so as to excite said display elements sequentially in accordance with a specific display refresh order, such that an initial flare of illumination common to all said emitter elements occurs as each display element is refreshed said display driver is further constructed such that display element refresh signal from said display driver is synchronized with said clocked raster-scan pulse electrical output from said video display, to said opto-electronic device and said digital processor;

said display driver is further constructed so as to synchronize with said clocked display field refresh pulse electrical output from said video display to said digital processor and said digital computer.

5. The apparatus as recited in claim 1, wherein said opto-electronic device further comprises:

a) a two-dimensional array of electronic opto-transistors said transistors further constructed so as to receive optical input from said optical lens system;

said transistors further constructed so as to provide electrical analog outputs;

b) an array of electronic comparators, said comparators constructed so as to match electrical analog input one-to-one to output of said array of opto-transistors and electrical digital output wherein output of each said comparators is constructed to be set when output of corresponding said opto-transistor exceeds a threshold level and further constructed to be reset when said opto-transistor remains below said threshold level, wherein the threshold level is less than electrical voltage generated by initial refresh illumination from a display element of said video display, but greater than the voltage generated by extraneous visual reflections and electronic noise;

c) an electronic encoder constructed so as to receive electrical digital inputs from said comparators and electrical digital output giving the array address of the said set comparator; and

d) an electronic latch which latches and holds the digital address output of the said encoder as output until reset by the clocked raster-scan pulse electrical output from said display driver of claim 4, where the reset is momentarily delayed to allow operation of the digital processor of claim 1 as described below.

6. The apparatus as recited in claim 1, wherein said digital processor further comprises:

a) means for reading latched address output of the opto-electronic device on said each clocked raster-scan pulse electrical output from said display driver, and means for writing address to a doubled-buffered digital memory, said means for writing further constructed so as to form a digital data table of comparator addresses as a function of clocked pulse count;

b) said digital data table further constructed so as to serve as a reference list of said sensor activated by excited light element, for said each clocked raster-scan pulse count, so as to serve as a source-to-sensor pairing table for the clocked display field refresh cycle; and

c) wherein, memory of said digital processor is buffered by the clocked display field refresh pulse output from said display driver and the clocked raster-scan pulse count is reset to zero.

7. The apparatus as recited in claim 1, wherein said image processor further comprises:

a) a charge coupled device array having electrical analog input from the analog output of the opto-transistor array such that electronic digitization of eye image is enabled over the clocked display field refresh cycle of said display driver;

b) means for isolating the viewer'"'"'s pupil image by binarization and thresholding of the accumulated digital eye image, said means further constructed so as to enable an action executed at receipt of the clocked display field refresh pulse output from said display driver; and

c) means for computing binary image centroid, principal axes and memory mapped electrical digital output of the image processor.

8. A method for computing, by digital computer using a digital computer program, the gaze point of regard and fixation duration for each eye, and the binocular convergence point for both eyes of a viewer, said method comprising the steps of:

(a) responding through an interrupt service routine operating in response to a clocked display field refresh pulse output from a display driver;

(b) reading the position and orientation of headset sensor;

(c) computing corneal center of said each eye from a three dimensional model of corneal reflections, and further computing cornea surface point locations of the reflections from a source-to-sensor pairing table and embedded memory tables of geometries for a light array and a sensor array;

(d) smoothing the computed corneal reflections for said each eye by clustering together surface points with similar orientation relative to the respective corneal center;

(e) computing pupil center for said each eye from the respective said corneal center of said each eye, further computing memory mapped pupil image center and source to sensor pairing table;

(f) computing optical origin and axis for said each eye from the corresponding corneal and pupil centers;

(g) computing median axes for cornea of said each eye from a corresponding said optical axis and from data obtained on said smoothed corneal reflection;

(h) computing viewing origin and axis for said each eye from the corresponding said optical origin and axis and median axes;

(i) converting viewing origin and axis for said each eye to the viewer'"'"'s workspace from a headset position and orientation sensor reading;

(j) classifying viewing status of said each eye as saccadic, fixation, or pursuit from data obtained in steps (a) through (i);

(k) classifying gaze status of said each eye by grouping local fixations, and computing the gaze point of regard in the viewer'"'"'s workspace and fixation duration, from the respective viewing origin and axis, and an embedded memory table of workspace geometry; and

(l) computing the binocular convergence point from the viewing origins and axes of both of said eyes for the viewer'"'"'s workspace.

9. A method as recited in claim 8, said method further comprising:

(a) determining certain optical constants for developing said routines, sequentially displaying a calibration marker at a set of predetermined locations within a visual field of a viewer, and directing the viewer'"'"'s gaze at said marker locations;

(b) obtaining a digital output for step a;

(c) displaying said calibration marker within said field of the viewer using a video display;

(d) sending a digital instruction to said display driver and displaying said calibration marker at a predetermined location within said visual field of the viewer, said sending step further comprises using a computer program with a display prompting routine;

(e) fixating said viewer'"'"'s gaze on a target;

(f) establishing an interrupt service routine in response to a clocked display field refresh pulse output from the display driver;

(g) establishing a routine for reading position and orientation of a headset sensor when said calibration display is not attached to the headset;

(h) establishing a routine using said three dimensional model of corneal reflections for computing the cornea center of said eye and surface locations of the corneal reflections, from a source-to-sensor pairing table and embedded memory tables of geometries for light array and a sensor array;

(i) establishing a routine for smoothing the computed corneal reflections for each eye by clustering together surface points with nearly the same orientation relative to the corresponding corneal center;

(j) establishing a routine for computing the optical origin and axis, and the median axes for cornea of each eye from said corneal center and smoothed corneal reflection data;

(k) computing a routine once three such calibration markers have been presented at separate locations, said routine computes using the marker location and headset position and orientation, a set of constant parameters for each eye relating the viewing origin and orientation to the optical origin and the optical and median axes;

(l) establishing a routine for computing, once three such calibration markers have been presented at separate locations, using pupil image centroid, interocular corneal center to pupil center distance, a constant parameter for each eye; and

(m) establishing a routine called by said display prompting routine following the display of the calibration marker at all locations, said routine computing best estimate to said optical constants and saving values of said constants in a memory file for reference.

10. A method for determining optical constants for developing routines for the gaze point of regard and fixation duration for each eye, and the binocular convergence point for both eyes of a viewer, said method further comprising:

(a) sequentially displaying a calibration marker at a set of predetermined locations within a visual field of a viewer, and directing the viewer'"'"'s gaze at said marker locations;

(b) obtaining a digital output for step a;

(c) displaying said calibration marker within said field of the viewer using a video display;

(d) sending a digital instruction to said display driver and displaying said calibration marker at a predetermined location within said visual field of the viewer, said sending step further comprises using a computer program with a display prompting routine;

(e) fixating said viewer'"'"'s gaze on a target and releasing said target from hibernation;

(f) establishing an interrupt service routine in response to a clocked display field refresh pulse output from the display driver;

(g) establishing a routine for reading position and orientation of a headset sensor when said calibration display is not attached to the headset;

(h) establishing a routine using said three dimensional model of corneal reflections, from a source-to-sensor pairing table and embedded memory tables of geometries for light array and a sensor array;

(i) establishing a routine for smoothing the computed corneal reflections for each eye by clustering together surface points with nearly the same orientation relative to the corresponding corneal center;

(j) establishing a routine for computing the optical origin and axis, and the median axes for cornea of each eye from said corneal center and smoothed corneal reflection data;

(k) computing a routine once three such calibration markers have been presented at separate locations, said routine computes using the marker location headset position and orientation, a set of constant parameters for each eye relating the viewing origin and orientation to the optical origin and the optical and median axes;

(l) establishing a routine for computing, once three such calibration markers have been presented at separate locations, using pupil image centroid, interocular corneal center to pupil center distance, a constant parameter for each eye; and

(m) establishing a routine called by said display prompting routine following the display of the calibration marker at all locations, said routine computing best estimate to said optical constants and saving values of said constants in a memory file for reference.