Drowsiness onset detection

• US 9,955,925 B2
• Filed: 12/18/2015
• Issued: 05/01/2018
• Est. Priority Date: 12/18/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kannan, Aadharsh, Ramaswamy, Govind, Gujjar, Avinash, Bhaskar, Srinivas
• Primary Examiner(s): Alter, Alyssa M
• Application Number:
  US14/974,584
• Publication Number:
  US 20170172520A1
• Time in Patent Office: 865 Days
• CPC Class Codes: A61B 5/0024 for multiple sensor units attached to the patient, e.g. using a body or personal area network
• CPC Class Codes: A61B 5/02405 Determining heart rate variability
• CPC Class Codes: A61B 5/16 Devices for psychotechnics using teaching or educational appliances G09B1/00 - G09B7/00; Testing reaction times ; Devices for evaluating the psychological state
• CPC Class Codes: A61B 5/18 for vehicle drivers or machine operators
• CPC Class Codes: A61B 5/6801 specially adapted to be attached to or worn on the body surface
• CPC Class Codes: A61B 5/681 Wristwatch-type devices
• CPC Class Codes: A61B 5/6898 Portable consumer electronic devices, e.g. music players, telephones, tablet computers
• CPC Class Codes: A61B 5/7257 using Fourier transforms
• CPC Class Codes: A61B 5/726 using Wavelet transforms
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: A61B 5/7282 Event detection, e.g. detecting unique waveforms indicative of a medical condition A61B5/4094, A61B5/4818, A61B5/349, A61B5/0823 take precedence
• CPC Class Codes: A61B 5/7405 using sound
• CPC Class Codes: A61B 5/742 using visual displays A61B5/339 takes precedence
• CPC Class Codes: A61B 5/746 Alarms related to a physiological condition, e.g. details of setting alarm thresholds or avoiding false alarms alarms per se G08B
• CPC Class Codes: B60K 28/06 responsive to incapacity of driver
• CPC Class Codes: G08B 21/06 indicating a condition of sleep, e.g. anti-dozing alarms
• CPC Class Codes: G16H 50/70 for mining of medical data, e.g. analysing previous cases of other patients

Abstract

Drowsiness onset detection implementations are presented that predict when a person transitions from a state of wakefulness to a state of drowsiness based on heart rate information. Appropriate action is then taken to stimulate the person to a state of wakefulness or notify other people of their state (with respect to drowsiness/alertness). This generally involves capturing a person'"'"'s heart rate information over time using one or more heart rate (HR) sensors and then computing a heart-rate variability (HRV) signal from the captured heart rate information. The HRV signal is analyzed to extract features that are indicative of an individual'"'"'s transition from a wakeful state to a drowsy state. The extracted features are input into an artificial neural net (ANN) that has been trained using the same features to identify when an individual makes the aforementioned transition to drowsiness. Whenever an onset of drowsiness is detected, a warning is initiated.

Claims

1. A system for detecting the onset of drowsiness in an individual, comprising:

one or more heart rate (HR) sensors which capture HR information of the individual over time, the HR information comprising heart rate variability (HRV) information;

one or more computing devices, said computing devices being in communication with each other whenever there is a plurality of computing devices, and a drowsiness onset detector computer program having a plurality of sub-programs executable by the one or more computing devices, the one or more computing devices being directed by the sub-programs of the drowsiness onset detector computer program to,

receive the HR information from the heart rate sensor or sensors,

extract a plurality of different HRV-related features from the HR information, said extracted different HRV-related features, which are among many features that can be extracted from HR information, having been determined to be specifically indicative of a transition from a wakeful state of an individual to a drowsy state of an individual,

combine the extracted different HRV-related features to produce a drowsiness detection input,

input the drowsiness detection input into an artificial neural network (ANN) classifier that has been trained to distinguish between said wakeful state of an individual and said drowsy state of an individual based on the extracted different HRV-related features,

identify from an output of the ANN classifier if the drowsiness detection input indicates that the individual is exhibiting an onset of drowsiness, and

whenever the drowsiness detection input indicates that the individual is exhibiting an onset of drowsiness, initiate a drowsiness onset warning.

2. The system of claim 1, wherein the HR information comprises a raw HR signal captured by the one or more HR sensors, and wherein the sub-program for extracting a plurality of different HRV-related features from the HR information comprises first computing a HRV signal from the raw HR signal.

3. The system of claim 1, wherein the one or more HR sensors comprise a computing device which computes a HRV signal from a raw HR signal captured by the HR sensor or sensors, and wherein the HR information comprises the HRV signal.

4. The system of claim 1, wherein the one or more heart rate (HR) sensors and the one or more computing devices are resident in a unified drowsiness onset detection device.

5. The system of claim 4, wherein the drowsiness onset detection device is either a wearable device worn on the person of said individual, or a mobile device carried by said individual.

6. The system of claim 1, wherein the one or more heart rate (HR) sensors are resident in a wearable device worn on the person of said individual, and the one or more computing devices are resident in a mobile device carried by said individual, and wherein the wearable device is in wireless communication with the mobile device.

7. The system of claim 6, wherein the HR information comprises a raw HR signal captured by the one or more HR sensors which is transmitted to said mobile device carried by the individual.

8. The system of claim 6, wherein the HR information comprises a raw HR signal captured by the one or more HR sensors, and wherein the one or more HR sensors comprise a computing device which computes a HRV signal from the raw HR signal, said HRV signal being transmitted to said mobile device carried by the individual.

9. The system of claim 1, further comprising a segmenting sub-program that is executed before the execution of the sub-program for extracting a plurality of different HRV-related features from the HR information, said segmenting sub-program segmenting the HR information received from the heart rate sensor or sensors into a sequence of prescribed-length segments whenever the received HR information is in the form of a HRV signal, and said segmenting sub-program computing a HRV signal from the received HR information whenever the received HR information is not in the form of a HRV signal before segmenting the HRV signal into the sequence of prescribed-length segments, and wherein the sub-programs for extracting a plurality of different HRV-related features from the HR information, combining the extracted different HRV-related features, inputting the drowsiness detection input into the ANN classifier, identifying from an output of the ANN classifier if the drowsiness detection input indicates that the individual is exhibiting an onset of drowsiness, and whenever the drowsiness detection input indicates that the individual is exhibiting an onset of drowsiness, initiating a drowsiness onset warning, are executed on each of the HRV signal segments as they are created.

10. The system of claim 9, wherein the prescribed-length segments of the HRV signal are each two minutes in length and represent a rolling window such that there is a prescribed offset time between each segment.

11. The system of claim 10, wherein each two minute long HRV signal segment represents approximately 120 heart rate variation values, and wherein the segmenting sub-program further comprises up-sampling each HRV signal segment to approximately 840 samples heart rate variation values using curve fitting techniques.

12. The system of claim 9, wherein the sub-program for extracting a plurality of different HRV-related features from each HRV signal segment comprises extracting the plurality of different HRV-related features using a Discrete Fast Fourier Transform (DFFT) and a Discrete Wavelet Transform (DWT).

13. The system of claim 12, wherein the DFFT is used to compute a ratio of the power of the low frequency components of the HRV signal segment to the power of the high frequency (HF) components of the HRV signal segment as one of the extracted different HRV-related features, and wherein the DWT is used to compute from an 8-level decomposition of the HRV segment the D₁ through D₈ entropy coefficients, A₈ entropy coefficient and the mean of the A₁ through A₈ entropy coefficients as others of the extracted different HRV-related features.

14. The system of claim 1, wherein the sub-program for initiating a drowsiness onset warning, comprises initiating a wake-up call to the individual in the form of a message that is displayed on a display screen that is viewable by the individual.

15. The system of claim 14, wherein the message comprises at least one of text, or images, or flashing images that appear as a flashing light on the display screen.

16. The system of claim 1, wherein the sub-program for initiating a drowsiness onset warning, comprises initiating a wake-up call to the individual in the form of an audible alert that is played over speakers which are within a hearing range of the individual.

17. The system of claim 1, wherein the sub-program for initiating a drowsiness onset warning, comprises initiating a wake-up call to the individual in the form of a haptic alert that can be felt by the individual.

18. The system of claim 1, wherein the sub-program for initiating a drowsiness onset warning, comprises transmitting a message to a third party requesting the third party to warn the individual.

19. A system for detecting the onset of drowsiness in an individual, comprising:

one or more computing devices, said computing devices being in communication with each other whenever there is a plurality of computing devices; and

a drowsiness onset detector computer program having a plurality of sub-programs executable by the one or more computing devices, the one or more computing devices being directed by the sub-programs of the drowsiness onset detector computer program to,

receive heart rate (HR) information of the individual over time, the HR information comprising heart rate variability (HRV) information, wherein the one or more computing devices are in communication over a data communication network with a remote computing device associated with one or more HR sensors which capture the HR information, and wherein the HR information is received from the remote computing device via the data communication network,

extract a plurality of different HRV-related features from the HR information, said extracted different HRV-related features, which are among many features that can be extracted from HR information, having been determined to be specifically indicative of a transition from a wakeful state of an individual to a drowsy state of an individual,

combine the extracted different HRV-related features to produce a drowsiness detection input,

input the drowsiness detection input into an artificial neural network (ANN) classifier that has been trained to distinguish between said wakeful state of an individual and said drowsy state of an individual based on the extracted different HRV-related features,

identify from an output of the ANN classifier if the drowsiness detection input indicates that the individual is exhibiting an onset of drowsiness, and

whenever the drowsiness detection input indicates that the individual is exhibiting an onset of drowsiness, transmit a drowsiness onset notification.

20. A drowsiness onset detection classifier training system, comprising:

one or more computing devices, said computing devices being in communication with each other via a computer network whenever there is a plurality of computing devices; and

a drowsiness onset detector training computer program having a plurality of sub-programs executable by the one or more computing devices, the one or more computing devices being directed by the sub-programs of the drowsiness onset detector training computer program to,

receive the HR information output by one or more heart rate sensors for a plurality of individuals over time, as well as a drowsiness indicator for each of the individuals specifying whether that individual is in a wakeful state or in a drowsy state at the time the HR information was captured, the HR information comprising heart rate variability (HRV) information,

for the HR information associated with each of the plurality of individuals,

extract a plurality of different HRV-related features from the HR information, said extracted different HRV-related features, which are among many features that can be extracted from HR information, having been determined to be specifically indicative of a transition from a wakeful state of an individual to a drowsy state of an individual, and

combine the extracted different HRV-related features to produce a drowsiness detection input, and

train an artificial neural network (ANN) classifier to distinguish between said wakeful state of an individual and said drowsy state of an individual using said the drowsiness detection inputs and drowsiness indicator associated with each of the plurality of individuals.