Method of and apparatus for evaluation and mitigation of microsleep events

• US 6,070,098 A
• Filed: 04/10/1998
• Issued: 05/30/2000
• Est. Priority Date: 01/11/1997
• Status: Expired due to Term

• Current Assignee: Circadian Technolgies, Inc.
• Original Assignee: Circadian Technolgies, Inc.
• Inventors: Trutschel, Udo E., Heitmann, Anneke M., Moore-Ede, Martin C., Guttkuhn, Rainer
• Primary Examiner(s): Nasser, Robert L.
• Application Number:
  US09/058,649
• Time in Patent Office: 781 Days
• Field of Search: 600/300-301, 600/544
• US Class Current: 600/544
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: A61B 5/1103 Detecting eye twinkling
• CPC Class Codes: A61B 5/369 Electroencephalography [EEG] devices for psychotechnics A61B5/16
• CPC Class Codes: A61B 5/398 Electrooculography [EOG], e.g. detecting nystagmus; Electroretinography [ERG]
• CPC Class Codes: A61B 5/6821 Eye
• CPC Class Codes: A61B 5/726 using Wavelet transforms
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: G16H 50/70 for mining of medical data, e.g. analysing previous cases of other patients

Abstract

A method and apparatus for determining, monitoring and predicting levels of alertness by detecting microsleep episodes includes a plurality of channel processing units and a channel combining unit. Each of the channel processing units receives an information channel which conveys information associated with the mental and behavorial state of the subject, such as for example an EEG channel, and classifies the information into a distinct category. Such categories may include microsleep, non-microsleep, one or more of a plurality of stages of sleep, one or more of a plurality of stages of wakefulness, or a transition state characterized by a transition from one of the aforementioned states to another. Each of the channel processing units includes a neural network which has been trained with a set of example input/result vector pairs. The example input/result vector pairs are generated by correlating actual information channel outputs with observed fatigue related events such as nodding off, head snapping, multiple blinks, blank stares, wide eyes, yawning, partial and complete prolonged eyelid closures, and slow rolling eye movements.

Claims

1. A method of detecting and indicating at least one occurrence of a microsleep event experienced by a subject, comprising the steps of:

training a neural network to detect said occurrence, said neural network having an input for receiving data, correlated to the occurrence and non-occurrence of microsleep events, from at least one descriptive data source, and an output representing an alertness level result indicating whether a microsleep event has occurred; and

,

applying a plurality of input feature vectors derived from data received from said descriptive data source to said neural network input, each of said input feature vectors includes data elements corresponding to at least one set of physiological, behavioral or performance data relating to at least one aspect of said subject'"'"'s alertness state, so as to produce an alertness level result at said output.

2. A method according to claim 1, wherein said step of training a neural network includes the following substeps:

providing a plurality of example input feature vectors, selected to represent at elast one desired alertness level result; and

,

applying each of said plurality of input feature vectors to said input of said neural network, and adjusting a plurality of neural network weights and biases, so as to minimize a difference between said alertness level result and said desired alertness level result.

3. A method according to claim 2, wherein each of said example input feature vectors is identified by one or more observed fatigue related events.

4. A method according to claim 3, wherein said fatigue related event includes at least one of the following:

head snapping, multiple blinks, blank stares, wide eyes, yawning, and partial and complete prolonged eyelid closures, partial and complete eye movements and combinations thereof.

5. A method according to claim 1, wherein said step of applying a plurality of input feature vectors further includes the step of applying data from said plurality of descriptive data sources sequentially, one vector at a time.

6. A method according to claim 1, wherein said step of applying a plurality of input feature vectors further includes the step of applying data from at least two of said plurality of descriptive data sources in parallel.

7. A method according to claim 1, wherein said alertness level result is indicitave of the occurrence of both microsleep events and non-microsleep events.

8. A method according to claim 7, wherein said alertness level result is further representative of the alertness state of the subject.

9. A method according to claim 8, wherein said alertness level result can indicate at least one type of transitional event in said alertness state.

10. A method according to claim 9, wherein the type of transitional event includes any one of the following types:

microsleep-to-non-microsleep, microsleep-to-sleep-stage-1, and microsleep-to-sleepstage-2.

11. A method according to claim 1, wherein said physiological, behavioral or performance criteria include at least one of the following:

EEG data, EOG data, EMG data, ECG data, eye blinking characteristics, eye tracking data, EDA data, blood pressure data, respiration data, facial expression data, and specific performance data, video data and combinations thereof.

12. A method according to claim 11, wherein data elements corresponding to said physiological, behavioral or performance criteria are transformed by at least one operation including determing the mean, median, variance, total power, discrete frequency power, average frequency, coefficients of autoregressive models, correlation dimension, mutual dimension, Lyapunov exponent, entropy and combinations thereof, so as to provide additional feature vectors to said neural network input.

13. An apparatus for detecting and indicating at least one occurrence of a microsleep event experienced by a subject, comprising:

a neural network trained to detect said occurrence, said neural network having an input for receiving data correlated to the occurrence and non-occurrence of microsleep events, from at least one descriptive data source, and an output representing an alertness level result indicating whether a microsleep event has occurred; and

,

means for receiving a plurality of input feature vectors derived from data received from said descriptive data source to said neural network input, wherein each of said input feature vectors includes data elements corresponding to at least one set of physiological, behavioral or performance criteria relating to at least one aspect of said subject'"'"'s alertness state, so as to produce an alertness level result at said output.

14. An apparatus according to claim 13, wherein said alertness level result is indicative of the occurrence of both microsleep events and non-microsleep events.

15. An apparatus according to claim 14, wherein said alertness level result is further representative of the alertness state of said subject.

16. An apparatus according to claim 15, wherein said alertness level result can indicate at least one type of transitional event in said alertness states.

17. An apparatus according to claim 16, wherein the types of said transitional event include any one of the following:

microsleep-to-non-microsleep, microsleep-to-sleep-stage-1, and microsleep-to-sleepstage-2.

18. An apparatus according to claim 13, wherein said data elements corresponding to said at least one set of physiological, behavioral or performance criteria include at least one of the following EEG data, EOG data, EMG data, ECG data, eye blinking characteristics, eye tracking data, EDA data, blood pressure data, respiration data, facial expression data, and specific performance data, and combinations thereof.

19. An apparatus according to claim 18, wherein prior to being provided to said neural network, the data elements corresponding to said at least one set of physiological, behavioral or performance criterial are transformed by at least one including determining the mean, median, variance, total power, discrete frequency power, average frequency, coefficients of autoregressive models, correlation dimension, mutual dimension, Lyapunov exponent, entropy and combinations thereof, so as to provide additional feature vectors to said neural network input.

20. An apparatus for automatically detecting microsleep events, comprising:

A. a data recording system for receiving at least one source of descriptive data related to a subject and for storing said data;

B. a feature extraction system for receiving said descriptive data stored by said recording system and generating processed data by producing a low dimensional representation of at least one fatigue-related data characteristic of said descriptive data;

C. a detection system including a plurality of neural networks trained by said data base of fatigue events and said database of non-fatigue events, each of said plurality of neural networks receiving one of said sources of said descriptive data and classifying said descriptive data into one of a plurality of fatigue events and non-fatigue events so as to produce a classified result; and

D. a contextual system for receiving said first classified result from each of said neural networks, and for applying a context interpretation algorithm to said classified results so as to produce an alertness level result related to said subject.

21. An apparatus according to claim 20, wherein said feature extraction system further includes a normalization and scaling system for normalizing said descriptive data such that a distribution of said descriptive data is more gaussian, and scaling said descriptive data such that said descriptive data occupies a reasonable and comparable range.

22. An apparatus according to claim 20, further including:

A. an event collection system for receiving said processed data for selection and storing a start time and an end time for a plurality of fatigue related events and non-fatigue related events; and

B. an event classification and categorization system for receiving said processed data, said start times and said end times, and classifying said processed data and said start and end times so as to produce a categorized database of fatigue events and an unauthorized database of non-fatigue related events.

23. An apparatus according to claim 20, wherein said alertness level result includes microsleep events.

24. An apparatus according to claim 20, wherein said contextual system includes a fuzzy logic system.

25. An apparatus according to claim 20, wherein said descriptive data includes physiological data, eye tracking data, video data, performance data, other data and combinations thereof.

26. An apparatus according to claim 25, wherein said physiological data are selected from the group consisting of EEG data, EOG data, EMG data, ECG data and combinations thereof.

27. An apparatus according to claim 25, wherein said eye tracking data are selected from the group consisting of pupil diameter, point of gaze, degree of eyelid closure, eye closure time, eye open time, blinking rate and combinations thereof.

28. An apparatus according to claim 25, wherein said video data are selected from the group consisting facial expressions, facial dynamics, face temperature distribution, and combinations thereof.

29. An apparatus according to claim 25, wherein said performance data are selected from the group consisting of reaction time, lane deviation, steering wheel variations, curvature error, throttle activity, heading error and combinations thereof.

30. An apparatus according to claim 25, wherein said other data includes respiration, blood pressure, electrodemial activity, heart rate, blinking rate, blinking amplitude, speed of eye movement and combinations thereof.

31. An apparatus according to claim 20, wherein said fatigue-related data characteristic of said descriptive data is selected from the group consisting of mean, median, variance, total power, power spectrum, power of certain frequency bands, mean frequencies, coefficients of autoregressive models, correlation dimension, Lypanow exponent, entropy and combinations thereof.

32. An apparatus according to claim 20, wherein said fatigue related events are selected from the group consisting of nodding off, partial and complete prolonged eye closure, multiple blinks, head snapping, blank stares, yawning, rapid pupil fluctuations, slow rolling eye movements and combinations thereof.

33. An apparatus according to claim 20, wherein said fatigue related events include person specific events selected from the group consisting of EEG type, gender, age chronotype, and event specific events selected from the group consisting of microsleep alpha-type, microsleep theta-type, transition microsleep/non-microsleep, transition microsleep/sleep stage 1, transition microsleep/sleep stage 2 and combinations thereof.

34. An apparatus according to claim 20, wherein said event classification and categorization system includes at least one unsupervised mode classification algorithm selected from the group consisting of Maxnet classifier, Adaptive Resonance Theory (ART1, ART2, fuzzy ART), Self Organizing Maps, Sammon'"'"'s algorithm and combinations thereof.

35. An apparatus according to claim 20, wherein said event classification and categorization system includes at least one supervised mode classification algorithm selected from the group consisting of Learning Vector Quantification (LQV), Hopfield Network, Boltzman Machine, K-means algorithm, fuzzy C-means algorithm, Multilayer Perceptron Networks.

36. An apparatus according to claim 20 wherein said neural networks include feedforward networks selected from the group consisting of multilayer perceptron networks, radial-basis function networks, higher order neural networks, probabilistic neural networks and combinations thereof.

37. An apparatus according to claim 20, wherein at least one neural network is trained for each type of said descriptive data.

38. An apparatus according to claim 20, wherein said alertness level result is one of a plurality of fatigue event categories, including microsleep alpha-type, microsleep theta-type, transition microsleep/non-microsleep, transition microsleep/sleep stage 1, transition microsleep/sleep stage 2 and related categories.

39. An apparatus according to claim 20, wherein said context interpretation algorithm is selected from the group consisting of voting schemes, confusion matrices, fuzzy logic algorithms, multilayer perceptron networks and combinations thereof.

40. An apparatus according to claim 20, wherein said contextual system includes an output unit for calculating alertness related parameters, including mean and variability as a function of time, alertness level as a function of time, alertness lapses per unit time period, and periodicity of alertness lapses.

41. An apparatus according to claim 20, wherein bio-compatible shiftwork schedule components are determined as a function of said alertness level result, said bio-compatible shiftwork schedule components being selected from the group consisting of shift length modification, shift sequence modification, nap scheduling, day off scheduling and combinations thereof.

42. An apparatus according to claim 20, wherein a sleep and nap schedule components are determined as a function of said alertness level result, said sleep and nap schedule components being selected from the group consisting sleep duration, sleep start time, surrounding circumstances, and combinations thereof.