Automatic aiding of human cognitive functions with computerized displays

• US 6,092,058 A
• Filed: 01/08/1998
• Issued: 07/18/2000
• Est. Priority Date: 01/08/1998
• Status: Expired due to Fees

• Current Assignee: The United States of America As Represented By The Secretary of Agriculture
• Original Assignee: The United States Of America As Represented By The Secretary Of The Army
• Inventors: Smyth, Christopher C.
• Primary Examiner(s): Hafiz, Tariq R.
• Assistant Examiner(s): Brown, Edward G.
• Application Number:
  US09/004,702
• Time in Patent Office: 922 Days
• Field of Search: 706/10, 706/15, 706/16
• US Class Current: 706/10
• CPC Class Codes: A61B 5/318 Heart-related electrical modalities, e.g. electrocardiography [ECG]
• CPC Class Codes: A61B 5/378 Visual stimuli
• CPC Class Codes: A61B 5/389 Electromyography [EMG]
• CPC Class Codes: A61B 5/398 Electrooculography [EOG], e.g. detecting nystagmus; Electroretinography [ERG]
• CPC Class Codes: A61B 5/721 using a separate sensor to detect motion or using motion information derived from signals other than the physiological signal to be measured
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

An automatic aider of human cognitive functions for the operation of computerized displays. The invention estimates in real-time the mental decision made in response to a displayed stimulus from the single-even, transient, evoked cerebral potential and the corresponding visual response. The invention attains high accuracy levels of decision classification by combining a unique parametric model of the cerebral potential with advanced techniques drawn from numerical analysis, artificial intelligence, and nonlinear regression analysis. The invention uses an expert system to determine the decision aiding to be provided to the human operator from the disparities among the estimations and the decisions that are expected from task scripts for the displayed stimulus.

Claims

1. An apparatus for the automatic aiding of human cognitive functions during the operation of computerized displays from displayed stimuli, comprising:

a digital processor with inputs of recorded human biosignals from analog amplifiers appropriately placed to record the electroencephalogram, electrooculogram, and muscular potentials, that applies filtering to remove the artifacts from the electroencephalogram, that maintains a file of the processed signal, and that outputs a windowed cerebral potential commonly of one second in duration;

a digital processor that inputs the type of stimulus being displayed for the task being performed, that maintains a file of cerebral sources for the different stimuli-types, and which outputs the basic waveforms for the corresponding cerebral sources which are evocable by the stimuli-type;

a digital processor that inputs the said windowed cerebral potential and the said set of basic waveforms, that computes parameters for an autoregressive model of the cerebral potential which is driven by the summed outputs of parallel attenuators with the basic waveform sources as inputs, and which outputs the computed parametric values of the attenuations and autoregressive coefficients;

an expert system processor that consists of a front end spectrum analyzer cascaded in series with an inference engine and production rules, that inputs the said autoregressive coefficients to the said spectrum analyzer, that computes the power spectrum of the autoregressive coefficients as data input to the said inference engine, that estimates the cognitive state and from that a reliability measure by parsing the said production rules, and outputs the measure;

a digital processor that inputs the recorded visual responses of the human from a recorder of the head and eye movements, fixations, pupil sizes, and eye blinks;

which maintains a file of the visual responses, which computes visual indexes from the file, in particular, the location and duration of the visual fixations and the associated pupil sizes and the number of eye blinks, clutters the fixations into gaze points, determines the gaze durations, and the transitions between the same; and

which outputs these indexes;

a fuzzy logic processor that consists of a front-end statistical analyzer, a set of fuzzy membership functions, and a set of fuzzy classification rules;

that inputs the said visual indexes to the said statistical analyzer, computes gaze statistics from the indexes, and transforms the gaze statistics into normalized scores;

that applies the fuzzy membership functions to the scores to form fuzzy logic vector sets comprised of membership class probabilities;

that parses the fuzzy classification rules for the membership probability vectors into an estimate of the attention state; and

outputs the estimated state;

a digital classifier that inputs the said attenuation values, the said cognitive reliability, the said indexes of the visual response, and the said attention state, all to an artificial neural network which with interconnection weights, estimates the probabilities of the occurrences of all possible decisions that can be made in response to the displayed stimulus, and outputs the same;

a digital processor with input of the said decision probabilities, that selects the most likely decisions, and outputs the same; and

an expert system processor that consists of an inference engine and production rules with input of the said most likely decisions, that determines the decision aiding from rules based on the disparities among the most likely decision estimates and the decisions that are expected from task scripts for the displayed stimulus, and outputs the cognitive aiding to the computer controlling the display driver.

2. A method for estimating a cognitive decision made by a human operator in response to a displayed stimulus, from his single event, evoked potential response and the corresponding visual process, comprising the steps of:

mapping the elements of a set of parameters uniquely one-to-one onto the elements of a set of orthogonal basic waveforms, where the set of said basic waveforms are derived from a principal components analysis of the event average, evoked difference potential responses made by the human to the set of stimuli of which the displayed stimulus is an element, such that the set of said basic waveforms represent the cerebral sources of all possible decisions which can be made by the human in response to the displayed stimulus;

mapping the said set of parameters uniquely one-to-one onto a set of attenuators, where the input to each said attenuator is the correspondingly mapped said basic waveform source, the values of the said parameters are given by the attenuations for the corresponding sources, and the sum of the outputs of the attenuators is the deterministic input to an autoregressive filter used to model the single-event, evoked potential as a cerebral process;

estimating the values of the said attenuation parameters for the single event, evoked potential by;

(1) computing weights from the corresponding electroencephalogram and the said basic waveform sources for the stimuli set for the said parallel attenuator-autoregressive filter model,

(2) computing a least mean square solution to the autoregressive coefficients of the said model using the said weights,

(3) computing the attenuation values for the said basic waveforms from the said weights and autoregressive coefficients, and

(4) applying an iteration technique with the above said solutions as initial conditions;

computing the reliability of the estimation of the said attenuation parameters by computing the power spectrum of the electroencephalogram from the said autoregressive coefficients, and classifying the cognitive state of the human operator from the said power spectrum components and computing the reliability of the estimated decision from the said classified cognitive state with an inference engine and a set of production rules based on expert knowledge of the cognitive processes as a function of the cognitive state;

computing the visual response to the displayed stimulus consisting of the fixation and gaze indexes, by;

(1) maintaining a record of the operator'"'"'s eye-movements ordered by time, where the record contains the locations, start times, and durations of the visual fixation points on the display, as well as the pupil size and eye blinks,

(2) computing the indexes of the present fixation, such as the fixation duration, the changes in pupil size, and the number of eye blinks, and

(3) clustering adjacent eye fixations into a gaze point and computing;

the gaze indexes of the centroid displacement from the location of the displayed stimulus, the number of fixations constituting the gaze, the time duration, and the dispersion of the fixations about the centroid;

estimating the visual attention state with fuzzy-logic from the immediate gaze history commonly over the last 10 seconds, by;

(1) computing the statistics of the gaze clusters to include the time of occurrence of the first fixation in the cluster, the number of fixations within the gaze, the centroid of the locations of the fixations, dispersion of the locations, and time duration of the gaze clusters, and the times of transitions between gazes,

(2) grouping the gaze clusters by the times that display stimuli were presented and forming grand clusters from the gazes for the same stimulus by computing grand centroids and dispersions from the centroids and dispersion statistics for the clusters, and accumulates the transition counts, and the time durations,

(3) transforming the cluster statistics into quantitative scores by normalizing the values for fixation counts, the dispersions, the cluster durations, and the rate of transitions, from calibration data unique to the human user,

(4) applying fuzzy membership functions to the scored cluster parameters to determine fuzzy-logic vector sets comprised of membership class probabilities for the fuzzy terms for each of the cluster categories of fixation count, dispersion, duration, and transition rate, and

(5) parsing furzzy classification rules for the cluster membership probabilities to classify the attention state; and

applying the said attenuation parameters, said estimation reliability, said fixation and gaze response indexes, and said attention state as inputs to an artificial neural network used as a classifier, the outputs of which are the probabilities of the occurrences of all possible decisions which can he made in response to the displayed stimulus.

3. A method for the automatic aiding of human cognitive functions during the operation of computerized displays, by estimating the cognitive decisions made in response to the displayed stimuli, comprising the steps of:

establishing a record of normalization constants for the index numbers of the visual process, by;

(1) having the human operator performs a set of visual functions in response to a set of displayed stimuli in which he searches for a target stimulus among irrelevant clutter and then performs a set of visual gazing tasks,

(2) recording the visual responses of the human operator as he performs these tasks, where the visual responses are the eye-movements, the fixations, and the pupil sizes and number of eye blinks associated with the fixations, and

(3) clustering the eye fixations into gaze points, and for the gazes computing the fixation counts, the dispersions, the cluster durations, and the rate of transitions for the different viewing conditions, as well as the search and gaze task dispersion sizes, and setting the normalization constants equal to the corresponding computed values;

establishing a record set of orthogonal basic waveforms for the cerebral sources of the decision responses made to the set of display stimuli, by;

(1) recording a sequence of single-event evoked, transient cerebral potentials which are event-locked to the repeated presentations in turn of all elements of the stimuli set for multiple occurrences of all possible decisions as responses by the human operator,

(2) computing an event-average response potential for each of the possible decisions and each of the stimuli set elements from the corresponding set of all said single-event evoked, transient cerebral potentials,

(3) computing a grand average response potential from the set of said event-average responses, and in turn the set of event average difference potentials,

(4) computing the set of basic waveforms from the principal components analysis of the set of said event average difference potentials,

(5) computing the autoregressive coefficients for the electroencephalograms of the headers to the said single event recordings, and

(6) computing a set of waveforms for the cerebral sources by inverse filtering of the said set of basic waveforms with the said autoregressive coefficients;

supervised training of an artificial neural network used to represent the decision making process, by;

(1) computing a vector set of attenuations for each said single-event, evoked transient cerebral potential used in the computation of the record file of cerebral sources,

(2) augmenting the said vector set with the reliability of the cognitive state for the power spectrum computed from the said autoregressive coefficients,

(3) augmenting the said vector set with the indexes of the visual responses to the said displayed stimuli for each of the said single-event evoked responses,

(4) augmenting the said vector set with the visual attention state for the responses to the said displayed stimuli for each of the said single-event evoked responses,

(5) forming matched pairs between the said vector sets and the corresponding decision responses made for each of the said single-event evoked responses, and

(6) using the data matrix so formed from the said matched pairs, as an epoch of inputs consisting of the said attenuations, reliability, visual response indexes, and attention state, matched to the corresponding said decision response as output, for the training of the interconnection weights of the network;

continually recording the human electroencephalogram, electrooculogram, and muscular potentials from analog amplifiers appropriately placed, applying filtering to remove the artifacts from the said electroencephalogram, maintaining a file of the processed signal, and computing a windowed cerebral potential signal commonly of one second in duration;

continually recording the visual responses of the human from a recorder of the head and eye movements and maintaining a time ordered file of the said visual responses;

fixations, pupil sizes, and eye blinks, and determining visual indexes from the file by;

(1) computing the indexes of the present fixation, such as the fixation duration, the changes in pupil size, and the number of eye blinks, and

(2) clustering adjacent eye fixations into gaze points and computing the gaze centroid displacement from the location of the displayed stimulus, the number of fixations constituting the gaze, the time duration, and the dispersion of the fixations about the centroid;

presenting a display stimulus to the human operator from the set of known stimuli, determining the type of the stimulus being displayed for the task being performed, and determining the basic waveforms for the corresponding cerebral sources which are evocable by the stimuli-type;

mapping the elements of a set of parameters uniquely one-to-one onto the elements of the set of said orthogonal basic waveforms, mapping the said set of parameters uniquely one-to-one onto a set of attenuators, where the input to each said attenuator is the correspondingly mapped said basic waveform source, the values of the said parameters are given by the attenuations for the corresponding sources, and the sum of the outputs of the attenuators is the deterministic input to an autoregressive filter used to model the single event, evoked potential as a cerebral process;

estimating the values of the said attenuation parameters for the single event, evoked potential by;

(1) computing weights from the corresponding electroencephalogram and the said basic waveform sources for the stimuli set for the said parallel attenuator autoregressive filter model,

(2) computing a least mean square solution to the autoregressive coefficients of the said model using the said weights,

(3) computing the attenuation values for the said basic waveforms from the said weights and autoregressive coefficients, and

(4) applying an iteration technique with the above said solutions as initial conditions;

computing the reliability of the estimation of the said attenuation parameters by computing the power spectrum of the electroencephalogram from the said autoregressive coefficients, and classifying the cognitive state of the human operator from the said power spectrum components and computing the reliability of the estimated decision from the said classified cognitive state with an inference engine of an expert system and a set of production rules based on expert knowledge of the cognitive processes as a function of the cognitive state;

estimating the visual attention state with fuzzy-logic from the immediate gaze history commonly over the last 10 seconds, by;

(1) computing the statistics of the gaze clusters to include the time of occurrence of the first fixation in the cluster, the number of fixations within the gaze, the centroid of the locations of the fixations, dispersion of the locations, and time duration of the gaze clusters, and the times of transitions between gazes,

(2) grouping the gaze clusters by the times that display stimuli were presented and forming grand clusters from the gazes for the same stimulus by computing grand centroids and dispersions from the centroids and dispersion statistics for the clusters, and accumulates the transition counts, and the time durations,

(3) transforming the cluster statistics into quantitative scores by normalizing the values for fixation counts, the dispersions, the cluster durations, and the rate of transitions, from calibration data unique to the human user,

(4) applying fuzzy membership functions to the scored cluster parameters to determine fuzzy-logic vector sets comprised of membership class probabilities for the fuzzy terms for each of the cluster categories of fixation count, dispersion, duration, and transition rate, and

(5) parsing fuzzy classification rules for the cluster membership probabilities to classify the attention state;

applying the said attenuation parameters, said estimation reliability, said fixation and gaze response indexes, and said attention state as inputs to the said artificial neural network used as a classifier, the outputs of which are the probabilities of the occurrences of all possible decisions which can he made in response to the displayed stimulus; and

selecting the most likely decisions, and determining the cognitive aid to he provided to the human with an inference engine of an expert system and a set of production rules based on the discrepancy between the estimations of the most likely decisions made and the decisions expected from task scripts for the said display stimulus.