Safety enhanced computer assisted driving method and apparatus

• US 10,578,456 B2
• Filed: 03/28/2018
• Issued: 03/03/2020
• Est. Priority Date: 03/28/2018
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Huang, Jinshi
• Primary Examiner(s): Wilson, Brian
• Application Number:
  US15/938,492
• Publication Number:
  US 20190049267A1
• Time in Patent Office: 706 Days
• Field of Search: 3405731, 340575, 340576
• CPC Class Codes: A61B 5/02055 Simultaneously evaluating both cardiovascular condition and temperature
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/18 for vehicle drivers or machine operators
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: B60W 2040/0827 due to sleepiness
• CPC Class Codes: B60W 2510/0604 Throttle position
• CPC Class Codes: B60W 2510/0638 Engine speed
• CPC Class Codes: B60W 2520/10 Longitudinal speed
• CPC Class Codes: B60W 2540/18 Steering angle
• CPC Class Codes: B60W 2540/221 Physiology, e.g. weight, heartbeat, health or special needs
• CPC Class Codes: B60W 40/09 Driving style or behaviour
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: G01C 21/3415 Dynamic re-routing, e.g. recalculating the route when the user deviates from calculated route or after detecting real-time traffic data or accidents
• CPC Class Codes: G01C 21/3697 Output of additional, non-guidance related information, e.g. low fuel level G01C21/3679 takes precedence
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G16H 50/70 for mining of medical data, e.g. analysing previous cases of other patients

Abstract

Methods and apparatuses for safety enhanced computer-assisted driving. In embodiments, an apparatus for computer-assisted driving may include a neural network to determine a classification for behavior of a driver of a vehicle having the apparatus, based at least in part on data about the vehicle collected in real time, and a current level of stress or drowsiness of the driver determined in real time; and a safety action engine coupled to the neural network to determine a safety related action, based at least in part on the determined driver behavior classification and data related to current traffic or road condition of a route the vehicles is currently traveling on. The safety related action may be performed by an infotainment system or a navigation system of the vehicle to assist the driver in driving the vehicle in a safer manner.

Claims

1. An apparatus for computer-assisted driving, comprising:

a communication interface to receive, from a wearable device worn by a driver of a vehicle, a current level of stress or drowsiness of the driver, wherein the wearable device includes a first multilayer neural network to locally determine the current level of stress or drowsiness of the driver, based at least in part on physiological data of the driver collected in real time by the wearable device worn by the driver;

a second multilayer neural network implemented with a hardware accelerator in the vehicle and coupled with the communication interface to locally determine a classification for behavior of the driver among a spectrum of behavior classifications, based at least in part on operational data about the vehicle collected in real time by sensors of the vehicle, and the current level of stress or drowsiness of the driver determined by the first multi-layer neural network in real time;

a safety action engine operated by a processor in the vehicle and coupled to the second multilayer neural network to determine a safety related action, based at least in part on the determined driver behavior classification and data related to current traffic or road condition of a route the vehicle is currently traveling on; and

an infotainment system or a navigation system disposed in the vehicle to perform the safety related action to assist the driver in driving the vehicle in a safer manner.

2. The apparatus of claim 1, wherein the safety related action comprises a selected one of the infotainment system playing soothing music, the infotainment system providing a reminder to the driver to maintain a safe distance from another vehicle ahead of the vehicle, the navigation system suggesting to the driver to take a break at an upcoming location, or the navigation system suggesting to the driver to take an alternative route.

3. The apparatus of claim 1, wherein the second multilayer neural network is arranged to determine whether the classification for behavior of the driver belong to a timid class of behavior, a cautious class of behavior, a conservative class of behavior, a neutral class of behavior, an assertive class of behavior, or an aggressive class of behavior.

4. The apparatus of claim 3, wherein the timid class of behavior corresponds to behavior of drivers beyond a negative second standard deviation of a driver profile, the cautious class of behavior corresponds to behavior of drivers between a negative first standard deviation and the negative second standard deviation of the driver profile, the conservative class of behavior corresponds to behavior of drivers between an average and the negative first standard deviation of the driver profile, the neutral class of behavior corresponds to behavior of drivers between the average and a positive first standard deviation of the driver profile, the assertive class of behavior corresponds to behavior of drivers between the positive first standard deviation and a positive second standard deviation of the driver profile, and the aggressive class of behavior corresponds to behavior of drivers beyond the positive second standard deviation of the driver profile.

5. The apparatus of claim 1, wherein at least one of the first multilayer neural network or the second multilayer neural network is a selected one of a feedforward neural network, a convolution neural network or a recurrent neural network.

6. The apparatus of claim 1, further comprising a programmable processor to implement the infotainment system or the navigation system.

7. The apparatus of claim 1, further comprising another communication interface to receive the data related to current traffic or road condition of the route the vehicle is currently traveling on from one or more cloud servers.

8. The apparatus of claim 1, wherein the real-time physiological data of the driver comprises galvanic skin response data, components of galvanic skin response data, heart rate or electrocardiogram data of the driver.

9. The apparatus of claim 1, wherein the sensors of the vehicle include at least a first sensor to sense vehicle speed, a second sensor to sense engine speed, a third sensor to sense throttle position, a fourth sensor to sense engine load, a fifth sensor to sense brake pressure, or a sixth sensor to sense steering wheel angle.

10. The apparatus of claim 1 further comprising the infotainment system or the navigation system.

11. A method for computer assisted driving, comprising:

receiving, from a first multilayer neural network disposed in a wearable device worn by a driver of a vehicle, by a second multilayer neural network disposed in the vehicle, a current level of stress or drowsiness of a driver of the vehicle, based at least in part on physiological data of the driver collected in real time by sensors of the wearable device;

determining locally, with the second multilayer neural network disposed in the vehicle, a classification for behavior of the driver of the vehicle among a spectrum of behavior classifications, based at least in part on operational data about the vehicle collected in real time, and the current level of stress or drowsiness of the driver determined by the first multilayer neural network; and

determining locally, with a safety action engine, a safety related action, based at least in part on the determined driver behavior classification and data related to current traffic or road condition of a route the vehicle is currently traveling on;

wherein the determined safety related action assists the driver in driving the vehicle in a safer manner.

12. The method of claim 11, wherein determining a safety related action comprises determining a selected one of having an infotainment system playing soothing music, having the infotainment system provide a reminder to the driver to maintain a safe distance from another vehicle ahead of the vehicle, having a navigation system suggest to the driver to take a break at an upcoming location, or having the navigation system suggesting to the driver to take an alternative route.

13. The method of claim 11, wherein determining the classification for behavior of the driver of the vehicle comprises determining whether the behavior of the driver belongs to a timid class of behavior, a cautious class of behavior, a conservative class of behavior, a neutral class of behavior, an assertive class of behavior, or an aggressive class of behavior.

14. The method of claim 13, wherein the timid class of behavior corresponds to behavior of drivers beyond a negative second standard deviation of a driver profile, the cautious class of behavior corresponds to behavior of drivers between a negative first standard deviation and the negative second standard deviation of the driver profile, the conservative class of behavior corresponds to behavior of drivers between an average and the negative first standard deviation of the driver profile, the neutral class of behavior corresponds to behavior of drivers between the average and a positive first standard deviation of the driver profile, the assertive class of behavior corresponds to behavior of drivers between the positive first standard deviation and a positive second standard deviation of the driver profile, and the aggressive class of behavior corresponds to behavior of drivers beyond the positive second standard deviation of the driver profile.

15. At least one non-transitory computer readable media (CRM) comprising a plurality of instructions arranged to cause an infotainment system or a navigation system of an in-vehicle system disposed in a vehicle, in response to execution of the instructions by a processor of the in-vehicle system, to:

receive from a first multilayer neural network disposed in a wearable device worn by a driver of the vehicle, for a second multilayer neural network of the vehicle, a current level of stress or drowsiness of the driver of the vehicle, determined in real time based at least in part on physiological data of the driver collected in real time by sensors of the wearable device;

receive from the second multilayer neural network of the vehicle a classification for behavior of the driver of the vehicle among a spectrum of behavior classifications, locally determined in real time based at least in part on operational data about the vehicle collected in real time, and the current level of stress or drowsiness of the driver determined by the first multilayer neural network; and

determine locally, with a safety action engine, a safety related action to be performed by the infotainment system or the navigation system of the vehicle to assist the driver in driving the vehicle in a safer manner, based at least in part on the classification for behavior of the driver, and data related to current traffic or road condition of a route the vehicle is currently traveling on.

16. The non-transitory CRM of claim 15, wherein to determine a safety related action comprises to determine a selected one of the infotainment system playing soothing music, the infotainment system providing a reminder to the driver to maintain a safe distance from another vehicle ahead of the vehicle, the navigation system suggesting to the driver to take a break at an upcoming location, or the navigation system suggesting to the driver to take an alternative route.

17. The non-transitory CRM of claim 15, wherein the classification for behavior of the driver of the vehicle comprises to determine the behavior of the driver to belong to a timid class of behavior, a cautious class of behavior, a conservative class of behavior, a neutral class of behavior, an assertive class of behavior, or an aggressive class of behavior.

18. The non-transitory CRM of claim 17, wherein the timid class of behavior corresponds to behavior of drivers beyond a negative second standard deviation of a driver profile, the cautious class of behavior corresponds to behavior of drivers between a negative first standard deviation and the negative second standard deviation of the driver profile, the conservative class of behavior corresponds to behavior of drivers between an average and the negative first standard deviation of the driver profile, the neutral class of behavior corresponds to behavior of drivers between the average and a positive first standard deviation of the driver profile, the assertive class of behavior corresponds to behavior of drivers between the positive first standard deviation and a positive second standard deviation of the driver profile, and the aggressive class of behavior corresponds to behavior of drivers beyond the positive second standard deviation of the driver profile.

19. The non-transitory CRM of claim 15, wherein the in-vehicle system is further caused to receive the operational data about the vehicle collected in real time from a plurality of sensors of the vehicle, or to receive the data related to current traffic or road condition of the route the vehicle is currently traveling on from one or more cloud servers.

20. The non-transitory CRM of claim 15, wherein the physiological data of the driver comprises galvanic skin response data, components of galvanic skin response data, heart rate or electrocardiogram data of the driver.