METHOD TO ANALYZE ATTENTION MARGIN AND TO PREVENT INATTENTIVE AND UNSAFE DRIVING

• US 20190213429A1
• Filed: 11/21/2017
• Published: 07/11/2019
• Est. Priority Date: 11/21/2016
• Status: Active Grant

• Current Assignee: Telelingo, LLC
• Original Assignee: Telelingo, LLC
• Inventors: Sicconi, Roberto, Stys, Malgorzata Elzbieta
• Granted Patent: US 10,467,488 B2
• Time in Patent Office: Days
• CPC Class Codes: B60K 28/066 actuating a signalling device B60K28/063 takes precedence
• CPC Class Codes: B60W 2050/143 Alarm means
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 2540/229 Attention level, e.g. attentive to driving, reading or sleeping
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/045 Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence
• CPC Class Codes: G06Q 40/08 Insurance
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G08G 1/0962 : having an indicator mounted inside the vehicle, e.g. giving voice messages
• CPC Class Codes: G08G 1/09626 : where the origin of the information is within the own vehicle, e.g. a local storage device, digital map

Abstract

A method includes a computing device a computer device receiving extracted features from a driver-facing camera and from a road as viewed by a road-facing camera; the computer device further receiving extracted features reflecting the driver'"'"'s behavior including head and eyes movement, speech and gestures; the computer device further receiving extracted telemetry features from a vehicle; the computer device still further receiving extracted features reflecting the driver'"'"'s biometrics; and a decision engine receiving information from the computer device representing each of the extracted features of the driver, wherein a driver'"'"'s attention and emotional state is determined to evaluate risks associated to moving vehicles and the driver'"'"'s ability to deal with any projected risks.

Claims

1. A method comprising:

a computer device receiving extracted features from a driver-facing camera and from a road as viewed by a road-facing camera;

the computer device further receiving extracted features reflecting the driver'"'"'s behavior including head and eyes movement, speech and gestures;

the computer device further receiving extracted telemetry features from a vehicle;

the computer device still further receiving extracted features reflecting the driver'"'"'s biometrics; and

a decision engine receiving information from the computer device representing each of the extracted features of the driver, wherein a driver'"'"'s attention and emotional state is determined to evaluate risks associated to moving vehicles and the driver'"'"'s ability to deal with any projected risks.

2. The method according to claim 1, wherein if the decision engine dynamically determines that the driver'"'"'s attention is inadequate given a risk then the decision engine issues an informational alert to the driver.

3. The method according to claim 2, wherein the informational alert is a colored light and haptic signal.

4. The method according to claim 1, wherein if the decision engine dynamically determines which set of personalized models to use for the driver identified by biometrics.

5. The method according to claim 1, wherein if the decision engine dynamically determines that the driver'"'"'s attention is insufficient then the decision engine issues a colored light informational alert, a pattern acoustic warning, and a pattern spoken warning to the driver.

6. The method according to claim 1, wherein if the decision engine dynamically determines that the driver'"'"'s attention is insufficient then the decision engine starts an interaction dialog with the driver to restore the driver'"'"'s attention.

7. The method according to claim 1, wherein the decision engine further uses driving condition information and compares the driving condition information obtained by the extracted features of the road with the determined driver'"'"'s attention level before the decision engine issues any warnings.

8. A driver assistant system comprising:

one or more processors, one or more computer-readable memories and one or more computer-readable, tangible storage devices;

a driver'"'"'s attention state module operatively coupled to at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, configured to receive extracted features for a driver from a camera facing the driver and from a road as viewed by a camera facing the road;

the driver'"'"'s attention module operatively coupled to at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, is further configured to receive extracted features reflecting the driver'"'"'s facial and hand gestures, and speech;

the driver'"'"'s attention module operatively coupled to at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, is further configured to receive extracted features reflecting the driver'"'"'s biometrics; and

a decision engine module operatively coupled to at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, configured to receive information from the driver'"'"'s attention module representing each of the extracted features of the driver, wherein a driver'"'"'s attention, wellness and emotional state is determined.

9. The driver assistant system according to claim 8, wherein if the decision engine module dynamically determines that the driver'"'"'s attention is inadequate for the computed risk level then the decision engine module issues an informational alert to the driver, whereby escalation is applied to the alerts depending on level of severity and urgency.

10. The driver assistant system according to claim 8, wherein if the decision engine module dynamically determines that the driver'"'"'s attention is insufficient then the decision engine issues a colored light informational alert, a pattern acoustic warning, and a pattern spoken warning to the driver.

11. The driver assistant system according to claim 8, wherein if the decision engine module dynamically determines that the driver'"'"'s attention is insufficient then the decision engine starts an interaction dialog with the driver to restore the driver'"'"'s attention.

12. The driver assistant system according to claim 8, further comprises:

A driver reaction evaluation system which evaluates a driver'"'"'s reaction to the alerts and quality of the resulting intervention to create a driving profile that describes driver'"'"'s demonstrated skills, ability to react and contains historic data about responsiveness and performance trends over time.

13. The driver assistant system according to claim 12, wherein observed responsiveness of the driver to alerts is used to modify the alerts to improve effectiveness of the interaction and minimize time to action by the driver.

14. The driver assistant system according to claim 8, wherein the decision engine module further uses driving condition information and compares the driving condition information obtained by the extracted features of the road with the determined driver'"'"'s attention level before the decision engine issues any warnings.

15. A virtual co-pilot method comprising:

an image processor receiving images from a camera facing a driver;

the image processor receiving scans from an infrared scanner facing the driver;

road facing camera

a speech engine receiving speech from the driver using a microphone; and

biosensors providing biometric data from the driver to a processing unit, wherein the processing unit uses machine learning to dynamically evaluate risks associated by the received images and scans from the image processor, the received speech from the speech engine and the provided biometric data from the biosensors to determine a driver'"'"'s attention, emotional state and fatigue.

16. The virtual co-pilot method according to claim 15 further comprises allowing the driver to modify Electronic Driving Record (EDR) data access rights dynamically and identify who gets to see what EDR data to provide privacy protection and enable dynamic policy negotiation with an insurance company.

17. The virtual co-pilot method according to claim 15, wherein if the processing unit dynamically determines that the driver'"'"'s attention is marginal then the processing unit issues a colored light informational alert and a pattern acoustic warning to the driver.

18. The virtual co-pilot method according to claim 15, wherein if the processing unit dynamically determines that the driver'"'"'s attention is insufficient then the processing unit issues a colored light informational alert, and a pattern acoustic warning and a pattern spoken warning to the driver via a speaker.

19. The virtual co-pilot method according to claim 15, wherein if the processing unit dynamically determines that the driver'"'"'s attention is insufficient then the processing unit starts an interaction dialog with the driver using dialog management within the speech engine to restore the driver'"'"'s attention.

20. The virtual co-pilot method according to claim 15, wherein the processing unit further uses driving condition information and compares the driving condition information obtained by the extracted features of the road with the determined driver'"'"'s attention level before the processing unit issues any warnings.