Systems and methods for recognizing and analyzing emotional states of a vehicle operator

• US 10,293,830 B2
• Filed: 11/07/2016
• Issued: 05/21/2019
• Est. Priority Date: 11/07/2016
• Status: Active Grant

• Current Assignee: Hyundai Motor Company (Hyundai Motor Group), Kia Corp.
• Original Assignee: Honeywell International Inc.
• Inventors: Selvaraj, JeevananthaRaja, Janakiraman, Kirupakar, Vikraman, Rajesh, Jagannathan, Muralidharan
• Primary Examiner(s): Mortell, John F
• Application Number:
  US15/345,416
• Publication Number:
  US 20180126999A1
• Time in Patent Office: 925 Days
• Field of Search: None
• CPC Class Codes: B60K 28/06 responsive to incapacity of driver
• CPC Class Codes: B60W 2040/0872 Driver physiology
• CPC Class Codes: B60W 2050/0075 Automatic parameter input, automatic initialising or calibrating means
• CPC Class Codes: B60W 2050/143 Alarm means
• CPC Class Codes: B60W 2540/22 Psychological state; Stress level or workload
• CPC Class Codes: B60W 2540/26 Incapacity
• CPC Class Codes: B60W 2556/10 Historical data
• CPC Class Codes: B60W 2556/45 External transmission of data to or from the vehicle
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: G08B 21/06 indicating a condition of sleep, e.g. anti-dozing alarms

Abstract

A system is disclosed for reducing the likelihood of vehicle accidents, particularly accidents involving large commercial vehicles, such as aircraft. The system includes one or more wearable electronic devices worn by a vehicle operator, such as a pilot. Physiological emotional data is measured by the wearable device(s) and transmitted to a computing cloud or other computing device for analysis and comparison with a predictive model for indicating the emotional state of the operator. When an unsafe emotional state is detected, an alarm notification is generated and transmitted to an appropriate group of users for resolution.

Claims

1. A system for recognizing and analyzing emotional states of a plurality of vehicle operators, comprising:

one or more sensors configured to measure physiological emotional data indicating an emotional state of a plurality of operators of one or more vehicles;

a transceiver in communication with the one or more sensors, the transceiver configured to transmit the measured physiological emotional data to a computing cloud or other computing device;

an analysis module in communication with the transceiver and the computing cloud or other computing device, the analysis module being configured to;

receive the measured physiological emotional data;

compare the measured physiological emotional data against one or more preexisting patterns of stored physiological emotional data to determine the emotional state of each of the plurality of operators;

generate an alarm if a selected condition is satisfied; and

transmit the determined emotional states and generated alarms to a control station;

a control station located off-board the one or more vehicles and in communication with each of the one or more vehicles via the computing cloud or other computing device, the control station configured to;

receive the determined emotional states and generated alarms of each of the plurality of operators; and

display, simultaneously on a dashboard, the emotional states and generated alarms of each of the plurality of operators using the system at a given time.

2. The system of claim 1, wherein the one or more vehicles comprise an aircraft and the plurality of operators include a pilot and a co-pilot.

3. The system of claim 1, wherein the one or more sensors comprise one or more wearable electronic devices.

4. The system of claim 1, wherein the physiological emotional data comprises skin conductance, electromyography data, finger temperature, cardiovascular activity, blood volume pulse, systolic blood pressure, diastolic blood pressure, pulse transit time, pulse volume amplitude, or other measurable indicator.

5. The system of claim 1, wherein the emotional state of the plurality of operators comprises one or more of the following states:

fearful, frustrated, relaxed, joyful, excited, drunkenness, or sleepiness.

6. The system of claim 1, wherein the preexisting patterns of stored physiological emotional data are stored in the computing cloud or other computing device and comprise physiological emotional data regarding the plurality of operators using the system over time.

7. The system of claim 1, wherein the preexisting patterns of stored physiological emotional data are updated periodically as physiological emotional data is collected from the plurality of operators.

8. The system of claim 1, wherein the analysis module is configured to compute a relaxation score using a classification method, support vector machine, or Fisher'"'"'s linear discriminant.

9. The system of claim 1, wherein the alarm may be selected from a group of alarms of differing severities, including a countdown to crash indicator and a man down indicator,

wherein the countdown to crash indicator is situationally aware of a position and environment of each of the one or more vehicles and the emotional state of each of the plurality of operators.

10. The system of claim 1, wherein the alarm comprises one or more visual elements, audio elements, or haptic elements.

11. The system of claim 1, wherein the alarm is broadcast to a plurality of users, together with information about the emotional state of the plurality of operators.

12. A method for recognizing and analyzing emotional states of a plurality of vehicle operators, comprising:

receiving, from one or more vehicles, measured physiological emotional data indicating an emotional state of a plurality of operators of the one or more vehicles;

comparing the measured physiological emotional data against one or more preexisting patterns of stored physiological emotional data to determine the emotional state of each of the plurality of operators;

generating an alarm if comparing the measured physiological emotional data against the one or more preexisting patterns results in a selected condition being satisfied;

transmitting the determined emotional states and generated alarms of each of the plurality of operators to a control station located off-board the one or more vehicles, the control station comprising a dashboard for displaying simultaneously the emotional states and generated alarms of each of the plurality of operators simultaneously operating one of the one or more vehicles at a given time.

13. The method of claim 12, wherein the each of the one or more vehicles comprises an aircraft and the plurality of operators includes a pilot and a co-pilot.

14. The method of claim 12, wherein the physiological emotional data is measured by one or more wearable electronic devices worn by the each of the plurality of operators.

15. The method of claim 12, wherein the physiological emotional data comprises skin conductance, electromyography data, finger temperature, cardiovascular activity, blood volume pulse, systolic blood pressure, diastolic blood pressure, pulse transit time, pulse volume amplitude, or other measurable indicator.

16. The method of claim 12, wherein the emotional state of the plurality of operators comprises one or more of the following states:

fearful, frustrated, relaxed, joyful, or excited.

17. The method of claim 12, wherein the preexisting patterns of stored physiological emotional data are updated periodically as physiological emotional data is collected from the plurality of operators.

18. A system for recognizing and analyzing emotional states of a plurality of vehicle operators, comprising:

a control station in communication with one or more vehicles via a communication module, the control station located off-board the one or more vehicles and comprising a dashboard configured to display simultaneously an emotional state of a plurality of operators, the plurality of operators each operating one of the one or more vehicles, wherein each of the plurality of operators having a wearable electronic device with sensors configured for monitoring physiological data indicating the emotional state of each of the plurality of operators during operation of the one or more vehicles; and

a computing cloud or other computing device in communication with the control station and having a cloud node located on-board each of the one or more vehicles,

wherein the computing cloud is configured to collect emotional score data from each of the plurality of operators and compile an emotional score database over time, the database being configured to generate predictive models for unsafe emotional states to determine the emotional state of the plurality of operators,

wherein the control station is configured to generate and display a suitable alarm signal when an unsafe emotional state is detected for each of the plurality of operators.

19. The system of claim 18, wherein the suitable alarm signal comprises one or more alarm priorities with a corresponding response procedure for each of the one or more alarm priorities, the one or more alarm priorities being based on a severity of the emotional state of each of the plurality of operators.

20. The system of claim 19, wherein the one or more vehicles comprise an aircraft and the plurality of operators include a pilot and a co-pilot, and

wherein the one or more alarm priorities comprises at least one of a priority one, a priority two, a priority three, a priority four, and a priority five,

wherein if the one or more alarm priorities is priority one, the control station transmits an alert to the pilot for resolution,

wherein if the one or more alarm priorities is priority two, the control station transmits a notification to one or more flight attendants,

wherein if the one or more alarm priorities is priority three, the control station transmits a notification requesting support from a co-pilot for resolution,

wherein if the one or more alarm priorities is priority four, the control station broadcasts a severe alert signal to the computing cloud or other computing device, together with information about the compromised emotional state of the pilot,

wherein if the one or more alarm priorities is priority five, the control station broadcasts an “

SOS”

signal to a plurality of control stations in communication with the computing cloud or other computing device.