Device and method for estimating a mental decision

• US 5,649,061 A
• Filed: 05/11/1995
• Issued: 07/15/1997
• Est. Priority Date: 05/11/1995
• Status: Expired due to Fees

• Current Assignee: United States Of America As Represented By The Secretary Of The Army (Government of the United States of America)
• Original Assignee: The United States Of America As Represented By The Secretary Of The Army
• Inventors: Smyth, Christopher C.
• Primary Examiner(s): Moore, David K.
• Assistant Examiner(s): Smith, Jeffrey S.
• Application Number:
  US08/439,392
• Time in Patent Office: 796 Days
• Field of Search: 128/731, 128/745, 351/218, 351/209, 395/20
• US Class Current: 706/16
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: A61B 5/1114 Tracking parts of the body
• CPC Class Codes: A61B 5/378 Visual stimuli
• CPC Class Codes: A61B 5/6803 Head-worn items, e.g. helmets, masks, headphones or goggles
• CPC Class Codes: A61B 5/6821 Eye
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection

Abstract

A device and method for estimating a mental decision to select a visual cue from the viewer'"'"'s eye fixation and corresponding single event evoked cerebral potential. The device comprises an eyetracker, an electronic biosignal processor and a digital computer. The eyetracker determines the instantaneous viewing direction from oculometric measurements and a head position and orientation sensor. The electronic processor continually estimates the cerebral electroencephalogramic potential from scalp surface measurements following corrections for electrooculogramic, electromyogramic and electrocardiogramic artifacts. The digital computer analyzes the viewing direction data for a fixation and then extracts the corresponding single event evoked cerebral potential. The fixation properties, such as duration, start and end pupil sizes, end state (saccade or blink) and gaze fixation count, and the parametric representation of the evoked potential are all inputs to an artificial neural network for outputting an estimate of the selection interest in the gaze point of regard. The artificial neural network is trained off-line prior to application to represent the mental decisions of the viewer. The device can be used to control computerized machinery from a video display by ocular gaze point of regard alone, by determining which visual cue the viewer is looking at and then using the estimation of the task-related selection as a selector switch.

Claims

1. An apparatus for estimating a mental decision of a viewer for selecting a task, comprising ocular means for generating an ocular line of sight and pupil size of the viewer wherein;

said ocular means include;

measuring means for measuring an ocular viewing direction and pupil sizes of the viewer;

sensor means for measuring a head position and orientation of the viewer in a workspace; and

digital processor means coupled with said measuring means and said sensor means, for computing and generating the ocular line of sight and pupil size data of the viewer in the workspace;

biosignal processor means for determining electroencephalogramic potential of the viewer;

said biosignal means include;

first electrode means for sensing scalp surface electroencephalogramic potentials from the cerebral scalp of the viewer; and

first amplifying means for amplifying the sensed scalp surface electroencaphalogramic potentials;

second electrode means including a plurality of electrode sensors for generating extracerebral potential from body portions of the view; and

third amplifying means including a plurality of amplifiers for respectively amplifying the generated extracerebral potentials;

the extracerebral potentials including electrooculogramic potentials generated by ocular processes, electromygramic potentials generated from facial throat, back, shoulder, and arm muscles of the viewer, and electrocardiogramic potentials generated from the heart of the viewer

said biosignal processor means further include;

an analog-to-digital (A/D)converter, connected to said first and second amplifying means, for generating digital output signals of the scalp surface electroencephalogramic potentials and the extracerebral potentials; and

a digital signal processor receiving the digital output signal from said A/D converter,

said digital signal processor having first filtering means for filtering measurement noise from the digital output signals and second filtering means connected to said first filtering means for filtering attenuation from the digital output signals; and

including summing means for summing the filtered digital output signals, so as to remove the electrooculogramic, electromygramic and electrocardiogramic potential from the scalp surface electroencephalogramic potentials;

computer means coupled to said ocular means and said biosignal processor means, for computing an estimate of the mental decision from the ocular line of sight and pupil size data and the electroencephalogramic potentials

said computer means include;

a first computer routine which stores the ocular line of sight and pupil size data from said ocular means; and

a second computer routine which determines if a sight state of the viewer is saccadic or fixation, using the ocular line of sight and pupil size data;

wherein

when the fixation is determined, said computer routine isolates the start and end of the fixation, computes s fixation duration, extracts the start and end pupil sizes, and determines a fixation end state and a gaze fixation count;

wherein

when the fixation is determined, said second computer routine determines values of fixation parameters and stores the fixation parameter values in memory;

a third computer means which stores the electroencephalogramic potential determined by said biosignal processor means; and

a fourth computer routine which extracts a single event transient evoked potential for the fixation, from the starting and ending points of the fixation and using the stored electroencephalogramic potentials; and

and computes a parametric representation of the single transient evoked potential using filter modeling techniques;

a fifth computer routine which generates workspace coordinate data of gaze point of regard of the viewer using the stored fixation parameter values; and

a sixth computer routine which computes an estimate of the mental decision from an expert system employing an artifice neural network;

said artificial neural network representing a mental decision process and having eye gaze fixation properties and the parametric representation of the transit evoked potential as inputs.

2. The apparatus of claim 1, wherein said computer means outputs the estimate of the mental decision computed by said sixth computer routine and the workspace coordinate data of gaze point of regard generated by said fifth computer routine.

3. The apparatus of claim 1, wherein said second computer routine generates a service call for sequentially activating said fourth, fifth, and sixth computer routines.

4. A method for estimating a decision making process of selecting a workspace item for task implementation from an ocular gaze of a viewer alone, comprising:

computing a task-selection level for an ocular point of regard from a corresponding eye fixation duration and single event transient evoked potential including the steps of;

measuring cerebral and extracerebral signals from body portions of the viewer to establish record file for EHG, EMG, and EOG;

computing signals from the cerebral and extracerebral signals using a plurality of filters;

isolating the electroencephalogramic signals for a fixation duration of each interest and disinterest designation by the viewer;

estimating the average evoked potentials for the interest and disinterest designations using the electroencephalgramic signals;

refining the average evoked potentials using filtering techniques;

computing a transient evoked potential of each designation by the viewer using filter modeling techniques;

computing a parametric representation of the transient evoked potential;

training an artificial neural network using gaze fixation properties and parametric representation of the transient evoked potential to estimate the decision making process of the viewer; and

b) determining that a selection decision is made for task implementation, if the computed task-selection level exceeds a threshold established by an artificial neural network.

5. The method of claim 4, further including the step of performing regression analyses to determine parameters of said plurality of filters in order to compute the electroencephalogramic signals from the cerebral and extracerebral signals.