Neurocognitive adaptive computer interface method and system based on on-line measurement of the user's mental effort

• US 5,447,166 A
• Filed: 01/19/1994
• Issued: 09/05/1995
• Est. Priority Date: 09/26/1991
• Status: Expired due to Term

• Current Assignee: Alan S. Gevins
• Original Assignee: Alan S. Gevins
• Inventors: Gevins, Alan S.
• Primary Examiner(s): Kamm, William E.
• Assistant Examiner(s): Getzow, Scott M.
• Application Number:
  US08/183,621
• Time in Patent Office: 594 Days
• Field of Search: 128/731, 128/732
• US Class Current: 600/544
• CPC Class Codes: A61B 5/16 Devices for psychotechnics using teaching or educational appliances G09B1/00 - G09B7/00; Testing reaction times ; Devices for evaluating the psychological state
• CPC Class Codes: A61B 5/377 using evoked responses
• CPC Class Codes: A61B 5/7207 of noise induced by motion artifacts
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: Y10S 128/925 Neural network

Abstract

A human-computer interface uses neuroelectric signals recorded from the user'"'"'s scalp i.e. electroencephalograms (EEGs) to alter the program being run by the computer, for example to present less or more difficult material to the user, depending on the user'"'"'s neurocognitive on-line workload score. Each user is tested with a standard battery of tasks, while wearing an EEG hat, to calibrate a neurocognitive workload function. The calibrated function is user-specific and is obtained by modifying a neural network pattern analyzer which has been previously trained to index neurocognitive workload using data from a group of subjects performing the same battery of tasks.

Claims

1. A method in human-computer interface for the computer to automatically react to the user'"'"'s neurocognitive workload without the user exerting muscle command action, by the computer changing a portion of a program being run by a computer;

including the steps of;

(a) presenting the user with a battery of standard tasks and, while the user performs the tasks, detecting and analyzing the brain waves of the user with an EEG (electroencephalograph) device having a plurality of electrodes removably connected to the scalp of the user to determine a normative neurocognitive workload calibration function for the user while the user performs the tasks, and recording the normative neurocognitive calibration function in the computer system memory;

(b) the user operates the computer system using the computer system muscle operated input control means;

(c) simultaneously with (b) detecting and analyzing the brain waves of the user with an EEG (electroencephalograph) device having a plurality of electrodes removably connected to the scalp of the user, to determine on-line neurocognitive workload score from the user'"'"'s normative neurocognitive calibration function; and

(d) adjusting a portion of the program being run by the computer system if the user'"'"'s on-line neurocognitive workload score is a predetermined amount below or above a threshold value.

2. A method as in claim 1 and simultaneouly with (a) and (c) detecting and measuring other physiological signals of the user selected from the group of facial muscle, eye muscle, heart activity and respiration.

3. A method as in claim 1 wherein the workload calibration function of (a) includes the overall normative neurocognitive workload function and the workload score of (c) includes the overall on-line neurocognitive workload score.

4. A method as in claim 1 wherein the neurocognitive workload function of (a) includes neurocognitive workload functions for each of a plurality of specialized brain systems and the neurocognitive workload score of (c) includes a score for each of the specialized brain systems.

5. A method as in claim 1 wherein in (a) the workload calibration function is determined using a neural network pattern analyzer in which generic networks are created derived from a normal population performing the standard tasks.

6. A method as in claim 5 wherein a specific calibration function is obtained for each user in (a) by transforming each user'"'"'s brain wave data to Z-score variables and inputting the Z-score variables into the neural network pattern analyzer.

7. A method as in claim 1 wherein the user in steps (a) and (c) wears a hat having a plurality of electrodes to contact the scalp of the user.

8. A method as in claim 1 wherein the computer is a portion of a computer-aided instruction system.

9. A method as in claim 8 wherein the portion of the program is adjusted to a less difficult instruction level if the user'"'"'s neurocognitive workload score is above a threshold value.

10. A method as in claim 8 wherein the portion of the program is adjusted to a more difficult level if the user'"'"'s neurocognitive workload score is below a threshold value.

11. A method as in claim 8 wherein the portion of the program being run is changed depending on the user'"'"'s use of specialized brain systems involved with perception, action, and cognition which include:

the planum temporale, superior temporal gyrus, Heschl'"'"'s gyrus and associated structures involved with auditory processing and speech perception;

the occipital and inferotemporal cortices and the parieto-occipito-temporal junction and associated structures involved in visual processing and pattern recognition;

the precentral gyrus, lateral premotor cortex, supplementary motor cortex, and associated structures involved with the planning, initiation, and execution of motor movements;

the dominant hemisphere frontal operculum, dorsalateral frontal cortex, and planum temporale and supramarginal gyrus and associated structures involved with language functions; and

the networks of structures encompassing the prefrontal, parietal, and temporal association cortices and associated regions involved in processing spatial information, in preparation and sequential planning, in reasoning, in focusing and shifting attention, in learning, and in working memory.

12. A method as in claim 1 wherein the computer is part of a computer based controlled system.

13. A method as in claim 12 and including the step of the computer based controlled system reacting to the user'"'"'s neurocognitive workload score being above a threshold by a predetermined amount, the computer adjusting a portion of the program so that the user is presented with less-demanding tasks.

14. A method as in claim 12 and including the step of the computer based controlled system reacting to the user'"'"'s neurocognitive workload score being below the threshold by a predetermined amount by the computer adjusting a portion of the program so that the user is presented with more-demanding tasks.

15. A method as in claim 12 and including the step of the computer based controlled system reacting to the user'"'"'s use of regional brain systems involved with perception, action, and cognition which include:

the planum temporale, superior temporal gyrus, Heschl'"'"'s gyrus and associated structures involved with auditory processing and speech perception;

the occipital and inferotemporal cortices and the parieto-occipito-temporal junction and associated structures involved in visual processing and pattern recognition;

the precentral gyrus, lateral premotor cortex, supplementary motor cortex, and associated structures involved with the planning, initiation, and execution of motor movements;

the dominant hemisphere frontal operculum, dorsalateral frontal cortex, and planum temporale and supramarginal gyrus and associated structures involved with language functions; and

the networks of structures encompassing the prefrontal, parietal, and temporal association cortices and associated regions involved in processing spatial information, in preparation and sequential planning, in reasoning, in focusing and shifting attention, in learning, and in working memory.

16. A system in human-computer interface including a computer which runs a plurality of programs and which automatically reacts to the user'"'"'s neurocognitive workload without the user exerting muscle command action by the computer changing a portion of the program being run by a computer, including:

(a) means for presenting the user with a battery of standard tasks;

means for detecting and analyzing the brain waves of the user comprising an EEG (electroencephalograph) device having a plurality of electrodes adapted to be removably connected to the scalp of the user to determine a normative neurocognitive workload calibration function for the user while the user performs the tasks and means for recording the normative neurocognitive calibration function comprising a portion of computer memory;

(b) muscle operated input control means to operate the computer;

(c) means for simultaneously with the operation of (b) to detect and analyze the brain waves of the user comprising said EEG device to determine on-line neurocognitive workload scores from the user'"'"'s normative neurocognitive calibration function; and

(d) means to adjust a portion of the program being run by the computer if the user'"'"'s on-line neurocognitive workload score is a predetermined amount below or above a threshold.

17. A system as in claim 16 and means for simultaneouly with (a) and (c) detecting and measuring other physiological signals of the user selected from the group of facial muscle, eye muscle, heart activity and respiration.

18. A system as in claim 16 wherein the workload calibration function of (a) includes the overall normative neurocognitive workload function and the workload score of (c) includes the overall on-line neurocognitive workload score.

19. A system as in claim 16 wherein the neurocognitive workload function of (a) includes neurocognitive workload functions for each of a plurality of specialized brain systems and the neurocognitive workload score of (c) includes a score for each of the specialized brain systems.

20. A system as in claim 16 and including a neural network pattern analyzer in which generic networks are created derived from a normal population performing the standard tasks.

21. A system as in claim 20 and including means for obtaining a specific calibration for each user in (a) by transforming each user'"'"'s brain wave data to Z-score variables and inputting the Z-score variables into the neural network pattern analyzer.

22. A system as in claim 16 and including a hat having a plurality of electrodes to contact the scalp of the user.

23. A system as in claim 16 wherein the computer is a portion of a computer-aided instruction system.

24. A system as in claim 23 wherein the portion of the program is adjusted to a less difficult instruction level if the user'"'"'s neurocognitive workload score is above a threshold.

25. A system as in claim 23 wherein the portion of the program is adjusted to a more difficult level of the user'"'"'s neurocognitive workload score is below a threshold.

26. A system as in claim 23 wherein the portion of the program being run is changed depending on the user'"'"'s use of regional brain systems involved with perception, action, and cognition which include:

the planum temporale, superior temporal gyrus, Heschl'"'"'s gyrus and associated structures involved with auditory processing and speech perception;

the occipital and inferotemporal cortices and the parieto-occipito-temporal junction and associated structures involved in visual processing and pattern recognition;

the precentral gyrus, lateral premotor cortex, supplementary motor cortex, and associated structures involved with the planning, initiation, and execution of motor movements;

the dominant hemisphere frontal operculum, dorsalateral frontal cortex, and planum temporale and supramarginal gyrus and associated structures involved with language functions; and

the networks of structures encompassing the prefrontal, parietal, and temporal association cortices and associated regions involved in processing spatial information, in preparation and sequential planning, in reasoning, in focusing and shifting attention, in learning, and in working memory.

27. A system as in claim 16 wherein the computer is part of a computer based controlled system.

28. A system as in claim 27 and including means for the computer-based controlled system to react to the user'"'"'s neurocognitive workload score being above a threshold by the computer adjusting a portion of the program so that the user is presented with less-demanding tasks.

29. A system as in claim 27 and including means for the computer based controlled system to react to the user'"'"'s neurocognitive workload score being below a threshold by the computer adjusting a portion of the program so that the user is presented with more-demanding tasks.

30. A system as in claim 27 and including means for the computer based controlled system to react to the user'"'"'s use of regional brain systems involved with perception, action, and cognition which include:

the planum temporale, superior temporal gyrus, Heschl'"'"'s gyrus and associated structures involved with auditory processing and speech perception;

the occipital and inferotemporal cortices and the parieto-occipito-temporal junction and associated structures involved in visual processing and pattern recognition;

the precentral gyrus, lateral premotor cortex, supplementary motor cortex, and associated structures involved with the planning, initiation, and execution of motor movements;

the dominant hemisphere frontal operculum, dorsalateral frontal cortex, and planum temporale and supramarginal gyrus and associated structures involved with language functions; and

the networks of structures encompassing the prefrontal, parietal, and temporal association cortices and associated regions involved in processing spatial information, in preparation and sequential planning, in reasoning, in focusing and shifting attention, in learning, and in working memory.