Neurocognitive adaptive computer-aided training method and system

• US 5,724,987 A
• Filed: 07/20/1995
• Issued: 03/10/1998
• Est. Priority Date: 09/26/1991
• Status: Expired due to Term

• Current Assignee: SAM Technology, Inc.
• Original Assignee: SAM Technology, Inc.
• Inventors: Smith, Michael, Gevins, Alan, Leong, Harrison, Sam-Vargas, Isabel
• Primary Examiner(s): Getzow, Scott M.
• Application Number:
  US08/504,653
• Time in Patent Office: 964 Days
• Field of Search: 128/731, 128/732, 434/326, 434/262, 434/118, 434/238, 434/258
• US Class Current: 600/544
• CPC Class Codes: A61B 5/16 Devices for psychotechnics using teaching or educational appliances G09B1/00 - G09B7/00; Testing reaction times ; Devices for evaluating the psychological state
• CPC Class Codes: A61B 5/377 using evoked responses
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

A computer-aided training system uses electroencephalograms (EEGs) recorded from the trainee'"'"'s scalp to alter the training protocol being presented by the computer, for example to present a new task to the trainee when he or she has mastered and automatized the current task. The index of the trainee'"'"'s skill mastery and automatization is determined by analysis of the EEG using mathematical classification functions which distinguish different levels of skill acquisition. The functions are computed by computer neural networks and consist of a combination of EEG and other physiological variables which specifically characterize a trainee'"'"'s level of focused attention and neurocognitive workload and his or her neurocognitive strategy. The functions are derived either for a group of trainees, or each trainee individually, performing a battery of one or more standard training tasks while wearing an EEG hat.

Claims

1. A method in computer based training of a human trainee for a computer to automatically assess learning related changes in the trainee'"'"'s brain state, including the steps of:

(a) presenting the trainee with a battery of learning tasks in which the trainee operates the computer using a muscle operated computer input control means;

(b) while the trainee performs the learning tasks, detecting and analyzing the brain waves of the trainee with an EEG (electroencephalograph) device having at least one electrode removably connected to the trainee to determine neurocognitive scores of the trainee'"'"'s focused attention;

(c) comparing the trainee'"'"'s scores of focused attention with previously obtained normal score standards of focused attention; and

(d) automatically adjusting a portion of a learning program being run by the computer if the trainee'"'"'s on-line neurocognitive scores of focused attention are a predetermined amount below or above a threshold value based upon the previously obtained normal score standards of focused attention.

2. A method as in claim 1 wherein the previous obtained normal score standards of focused attention are obtained for the same tasks, or for a battery of standard learning calibration tasks, either from the same trainee or from a group of similar trainees.

3. A method as in claim 2 wherein the trainee'"'"'s on-line neurocognitive workload scores includes the trainee'"'"'s overall mental effort scores and the normal score standards includes an overall mental effort score.

4. A method as in claim 3 wherein the overall mental effort score is measured by weighted combinations of power, crosspower, coherence and phase angle of individual frequency components, of the whole band and at peak frequency, and by parameters of parametric models, in the 7-13 Hz (alpha) and 13-26 Hz (beta) EEG bands recorded from electrodes over the entire cerebral cortex, in the 3-7 Hz (theta) band recorded over parietal, temporal and occipital cortex, and in higher frequency bands (greater than 26 Hz) recorded over temporalis, occipitalis and frontalis muscles.

5. A method as in claim 1 and also detecting and analyzing the brain waves to determine overall mental effort (neurocognitive workload) scores of the trainee while the trainee performs the learning tasks;

comparing the trainee'"'"'s overall mental effort scores with the previously obtained normal score standards of overall mental effort from the same trainee or from a group of similar trainees performing the same learning tasks or performing a battery of standard learning calibration tasks; and

adjusting a portion of a learning program being run by the computer if the trainee'"'"'s on-line neurocognitive scores of overall mental effort are a predetermined amount below or above a threshold value based upon the previously obtained normal score standards of overall mental effort.

6. A method as in claims 1, 2 or 5 and automatically assessing a type of neurocognitive strategy (differential degrees of utilization of brain systems associated with perception, action and cognition) the trainee is using to perform the learning tasks by scoring and analysis of the trainee'"'"'s brain wave frequencies and amplitudes at each electrode compared to the previously obtained normal score standards of brain wave frequencies and amplitudes;

and adjusting a portion of a learning program being run by the computer if the trainee'"'"'s neurocognitive strategy differs by a predetermined amount from the previously obtained normal score standards of brain wave frequencies and amplitudes.

7. A method as in claims 1, 2 or 5 wherein the adjustment of the program is an adjustment in timing when a next learning task is presented.

8. A method as in claims 1, 2 or 5 wherein the adjustment of the program is an adjustment between visual and auditory modalities.

9. A method as in claims 1, 2 or 5 wherein the adjustment of the program is within the visual modality between linguistic and graphical types of displayed information.

10. A method as in claims 1, 2 or 5 wherein the adjustment of the program is within the auditory modality between verbal and non-verbal types of information presented.

11. A method as in claims 1, 2 or 5 wherein the adjustment of the program changes the content of information presented.

12. A method as in claims 1, 2 or 5 wherein the adjustment of the program is to not present information to the trainee which otherwise would have been presented.

13. A method as in claims 1, 2 or 5 wherein the adjustment of the program is to present information to the trainee which otherwise would not have been presented.

14. A method as in claims 1, 2 or 5 wherein the trainee views and listens to selected training tasks but is not required to respond to the selected training tasks using muscle operated computer input control means.

15. A method as in claims 2 or 5 in which the battery of standard learning calibration tasks is selected from the group of working memory, divided attention, visuomotor and tracking tasks.

16. A method as in claims 1, 2 or 5 wherein the computer executes a pattern recognition algorithm and the method includes the step of training the algorithm to extract the neurocognitive scores selected from the group of focused attention, mental effort and strategy corresponding to stages of skill acquisition, neurocognitive automatization and learning strategy.

17. A method as in claim 16 in which the pattern recognition algorithm is selected from the group of a neural network algorithm, a fuzzy logic algorithm, a genetic learning algorithm, a statistical pattern classification algorithm, a syntactic pattern classification algorithm, a Bayesian or entropy (information) based cluster analysis algorithm, and a combination of the aforesaid algorithms.

18. A method as in claim 16 using the pattern recognition algorithm to separately extract neurocognitive scores of focused attention, mental effort and strategy.

19. A method as in claim 16 using the pattern recognition algorithm to generate a single function containing scores of neurocognitive focused attention, overall mental effort and strategy.

20. A method as in claim 16 and the step of training the pattern recognition algorithm using data generated by the trainee.

21. A method as in claim 16 and the step of training the pattern recognition algorithm using data generated by the group.

22. A method as in claim 16 and the step of adjusting a group pattern classification function previously derived from a group using the trainee'"'"'s scores to adjust the group function to a trainee.

23. A method as in claim 1, 2 or 5 and wherein the trainee'"'"'s neurocognitive scores (focused attention, mental effort or strategy) are compared to threshold values based upon standard values in the neurocognitive literature.

24. A method as in claims 1, 2 or 5 and making the comparisons using a pattern recognition algorithm in which the trainee'"'"'s data are compared with generic learning patterns derived from a normal population performing the same learning task or a battery of standard learning calibration tasks, or with learning patterns derived from the trainee performing the same learning task or a battery of standard learning calibration tasks.

25. A method as in claim 24 wherein a specific score is obtained for each trainee by transforming each trainee'"'"'s brain wave data to Z-score variables using the statistical distribution of the trainee'"'"'s initialization data and subsequent data, and inputting the Z-score variables into the pattern recognition algorithm.

26. A method as in claims 1, 2 or 5 and placing a hat, head set or holder having a plurality of at least two electrodes on the trainee'"'"'s head with the electrodes in contact with the scalp of the trainee while the trainee performs the learning tasks.

27. A method as in claim 5 wherein the portion of the program is adjusted to a less difficult training level if the trainee'"'"'s overall mental effort score is above a threshold value.

28. A method as in claim 5 wherein the portion of the program is adjusted to a more difficult level if the trainee'"'"'s overall mental effort score is below a threshold value.

29. A method as in claims 1, 2 or 5 wherein the portion of the program being run is changed depending on the trainee'"'"'s use of specialized brain systems involved with perception, action, and cognition which includes:

planum temporale, superior temporal gyrus, Heschl'"'"'s gryus and associated structures involved with auditory processing and speech perception;

occipital and inferotemporal cortices and parieto-occipito-temporal junction and associated structures involved in visual processing and pattern recognition;

precentral gyrus, lateral premotor cortex, supplementary motor cortex, and associated structures involved with planning, initiation and execution of motor movements;

dominant hemisphere frontal operculum, dorsolateral, orbitofrontal and mediobasal frontal cortex, and planum temporale and supramarginal gyrus and associated structures involved with language functions and networks of structures encompassing the anterior and posterior cingulate, hippocampus, hippocampal gyrus, and prefrontal, parietal, and temporal association cortices and associated regions involved in processing spatial information, in preparation and sequential planning, in reasoning, in focusing and shifting attention in learning and in working memory.

30. A method as in claim 1 wherein the neurocognitive scores of (b) includes the trainee'"'"'s on-line overall neurocognitive scores of focused attention and the previously obtained normal score standards of focused attention of (c) includes overall on-line neurocognitive focused attention scores.

31. A method as in claim 30 wherein the overall neurocognitive score of focused attention is measured by weighted combinations of power, crosspower, coherence and phase angle of individual frequency components, of the whole band, and at peak frequency, and by parameters of parametric models, in the 3-7 Hz (theta) and 7-13 Hz (alpha) EEG bands recorded from electrodes over the prefrontal cortex.

32. A method as in claim 1 wherein the neurocognitive scores of focused attention of (b) includes on-line neurocognitive normal score standards of focused attention of the trainee for each of a plurality of specialized brain systems and the normal score standards of focused attention of (c) includes normal scores for standards of focused attention for each of the specialized brain systems.

33. A method as in claim 1 wherein the portion of the program is adjusted to a more difficult training level if the trainee'"'"'s neurocognitive focused attention score is above a threshold value.

34. A method as in claim 1 wherein the portion of the program is adjusted to a more difficult or less difficult level if the trainee'"'"'s neurocognitive focused attention score is below a threshold value.

35. A system in computer based training of a human trainee in which the computer runs a plurality of programs to automatically assess learning related changes in the trainee'"'"'s brain state without the trainee exerting muscle command action, including:

(a) a computer having computer memory and muscle operated input means to operate the computer;

(b) means for presenting the trainee with a battery of learning tasks involving the trainee operating the input means;

(c) means for detecting and analyzing the brain waves of the trainee comprising an EEG (electroencephalograph) device having a plurality of electrodes removably connected to the scalp of the trainee to determine neurocognitive scores of focused attention for the trainee while the trainee performs the learning tasks, and means for recording the trainee'"'"'s focused attention scores comprising a portion of the computer memory;

(d) means in the computer memory for storing previously obtained normal score standards of focused attention;

(e) means in the computer for comparing the trainee'"'"'s scores with the normal score standards; and

(f) means in the computer for automatically adjusting a portion of a program being run by the computer as determined by the comparison, if the trainee'"'"'s on-line score is a predetermined amount below or above a threshold.

36. A system as in claim 35 and further including means for detecting and analyzing the trainee'"'"'s brain waves to determine the trainee'"'"'s on-line overall mental effort scores (neurocognitive workload) while the trainee performs the learning tasks;

means for storing normal score standards of overall mental effort;

means in the computer for comparing the trainee'"'"'s on-line overall mental effort scores with the stored normal score standards of overall mental effort; and

means in the computer for automatically adjusting a portion of a learning program being run on the computer if the trainee'"'"'s on-line neurocognitive scores of overall mental effort based on said comparison are a predetermined amount below or above a threshold value.

37. A system as in claims 35 or 36 wherein the normal score standards are obtained from a battery of standard calibration tasks or the same learning tasks and from the same trainee or from a group of similar trainees.

38. A system as in claims 35 or 36 and means for simultaneously detecting and measuring other physiological signals of the trainee while the trainee is performing the learning tasks, said signals being selected from the group of facial muscle, eye muscle, heart activity and respiration.

39. A system as in claim 36 wherein the trainee'"'"'s on-line neurocognitive workload scores include the trainee'"'"'s overall neurocognitive mental effort scores and the normal score standards include an overall neurocognitive mental effort score.

40. A system as in claim 36 wherein the trainee'"'"'s on-line overall mental effort scores include neurocognitive mental effort scores for each of a plurality of specialized brain systems and the normal score standards of mental effort includes a score for each of the specialized brain systems.

41. A system as in claims 35 or 36 and including a pattern recognition means for making the comparisons in which the trainee'"'"'s data are compared with generic learning patterns derived from a normal population performing the same learning tasks or a battery of standard learning calibration tasks, or with learning patterns derived from the trainee performing the same learning tasks or a battery of standard learning calibration tasks.

42. A system as in claim 41 and including means for obtaining a specific calibration for each trainee by transforming each trainee'"'"'s brain wave data to Z-score variables using the statistical distribution of the trainee'"'"'s initialization and subsequent data, and inputting the Z-score variables into the pattern recognition means.

43. A system as in claims 35 or 36 and including a hat, headband or holder having a plurality of at least two electrodes to contact the scalp of the trainee.

44. A system as in claims 35 or 36 wherein the portion of the program is adjusted to a more difficult training level if the trainee'"'"'s neurocognitive focused attention score is above a threshold.

45. A system as in claims 35 or 36 wherein the portion of the program is adjusted to a less or more difficult level if the trainee'"'"'s neurocognitive focused attention score is below a threshold.

46. A system as in claim 35 or 36 wherein the portion of the program is adjusted to a less difficult training level if the trainee'"'"'s overall mental effort score is above a threshold.

47. A system as in claim 35 or 36 wherein the portion of the program is adjusted to a more difficult level if the trainee'"'"'s neurocognitive overall mental effort score is below a threshold.

48. A system as in claims 35 or 36 wherein the portion of the program being run is changed depending on the trainee'"'"'s use of regional brain systems involved with perception, action, and cognition which include:

planum temporale, superior temporal gyrus, Heschl'"'"'s gyrus and associated structures involved with auditory processing speech perception;

occipital and inferotemporal cortices and parieto-occipito-temporal junction and associated structures involved in visual processing and pattern recognition;

precentral gyrus, lateral premotor cortex, supplementary motor cortex, and associated structures involved with planning, initiation and execution of motor movements;

dominant hemisphere frontal operculum, dorsolateral, orbitofrontal and mediobasal frontal cortex, and planum temporale and supramarginal gyrus and associated structures involved with language functions; and

networks of structures encompassing the anterior and posterior cingulate, hippocampus, hippocampal gyrus, and prefrontal, parietal, and temporal association cortices and associated regions involved in processing spatial information, in preparation and sequential planning, in reasoning, in focusing and shifting attention in learning and in working memory.

49. A system as in claim 35 wherein the focused attention scores of (c) include on-line overall neurocognitive focused attention scores and the normal standards of focused attention of (d) include overall normal standard neurocognitive focused attention scores.

50. A system as in claim 35 wherein the focused attention scores of (c) includes on-line neurocognitive focused attention scores for each of a plurality of specialized brain systems and the normal standard neurocognitive focused attention scores of (d) include a score for each of the specialized brain systems.