Long Term Driving Danger Prediction System

• US 20170158129A1
• Filed: 01/06/2017
• Published: 06/08/2017
• Est. Priority Date: 04/13/2015
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Laboratories America Inc (NEC Corporation)
• Inventors: Cosatto, Eric
• Granted Patent: US 10,336,252 B2
• Time in Patent Office: Days
• CPC Class Codes: B60Q 5/006 indicating risk of collision between vehicles or with pedestrians
• CPC Class Codes: B60Q 9/00 Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 , e.g. haptic signalling
• CPC Class Codes: B60Q 9/008 for anti-collision purposes
• CPC Class Codes: B60W 10/04 including control of propulsion units
• CPC Class Codes: B60W 10/18 including control of braking systems
• CPC Class Codes: B60W 10/30 including control of auxiliary equipment, e.g. air-conditioning compressors or oil pumps
• CPC Class Codes: B60W 2040/0827 due to sleepiness
• CPC Class Codes: B60W 2050/143 Alarm means
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2540/22 Psychological state; Stress level or workload
• CPC Class Codes: B60W 2540/26 Incapacity
• CPC Class Codes: B60W 2540/30 Driving style
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2555/20 Ambient conditions, e.g. wind or rain
• CPC Class Codes: B60W 2556/50 of positioning data, e.g. GPS [Global Positioning System] data
• CPC Class Codes: B60W 2710/18 Braking system
• CPC Class Codes: B60W 2710/30 Auxiliary equipments
• CPC Class Codes: B60W 2720/106 Longitudinal acceleration
• CPC Class Codes: B60W 30/0953 : the prediction being responsive to vehicle dynamic parameters
• CPC Class Codes: B60W 30/0956 : the prediction being responsive to traffic or environmental parameters
• CPC Class Codes: B60W 40/04 : Traffic conditions
• CPC Class Codes: B60W 40/06 : Road conditions
• CPC Class Codes: B60W 40/09 : Driving style or behaviour
• CPC Class Codes: B60W 50/14 : Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: B60W 50/16 : Tactile feedback to the driver, e.g. vibration or force feedback to the driver on the steering wheel or the accelerator pedal

Abstract

Systems and methods are disclosed to assist a driver with a dangerous condition by creating a graph representation where traffic participants and static elements are the vertices and the edges are relations between pairs of vertices; adding attributes to the vertices and edges of the graph based on information obtained on the driving vehicle, the traffic participants and additional information; creating a codebook of dangerous driving situations, each represented as graphs; performing subgraph matching between the graphs in the codebook and the graph representing a current driving situation to select a set of matching graphs from the codebook; determining a distance metric between each selected codebook graphs and the matching subgraph of the current driving situation; from codebook graphs with a low distance, determining potential dangers; and generating an alert if one or more of the codebook dangers are imminent.

Claims

1. A guidance system for a vehicle, comprising:

a processor executing a code for;

receiving an input from sensors, the input indicating surrounding traffic participants and information regarding a driving situation;

receiving a local map locating the vehicle;

creating a graph representation where traffic participants and static elements are the vertices and the edges are relations between pairs of vertices;

adding attributes to the vertices and edges of the graph based on information obtained on the driving vehicle, the traffic participants and additional information;

creating or obtaining a codebook of dangerous driving situations, each represented as graphs;

performing subgraph matching between the graphs in the codebook and the graph representing a current driving situation to select a set of matching graphs from the codebook;

calculating a distance between each selected codebook graph and the matching subgraph of the current driving situation;

determining potential dangers from codebook graphs with a low distance; and

generating an alert if one or more of the codebook dangers are imminent.

2. The system of claim 1, comprising predicting a dangerous driving situation a few seconds before occurrence.

3. The system of claim 1, comprising an input modality obtained from radar.

4. The system of claim 1, comprising an input modality obtained from a video camera with a structure from motion 3D generator.

5. The system of claim 1, comprising recognizing object types using computer vision on a video image as an input modality.

6. The system of claim 1, comprising using GPS coordinates from a vehicle sensor combined with a GPS-enabled local map.

7. The system of claim 1, comprising receiving as an input modality one or more of:

a driver status including sleepiness, aggressiveness and impairment;

a weather status including road surface, temperature, daylight, rain, snow;

a traffic situation proximal to the vehicle including roadwork, delays, accident; and

vehicle physical characteristics including length, width, weight, braking capabilities.

8. The system of claim 1, wherein the codebook is populated by known dangerous driving situations.

9. The system of claim 1, wherein the codebook is refined by using a database of recorded driving and unsupervised learning techniques to discover attribute ranges of traffic participant relations.

10. The system of claim 1, wherein the distance function between subgraphs is learned by supervised machine learning using subgraph attributes as features.

11. The system of claim 10, wherein one classifier per each codebook graph is trained using labeled data to predict a distance between two matching subgraphs.

12. The system of claim 1, wherein the distance function between subgraphs is learned using supervised machine learning using statistics of the attributes and a single classifier is trained for the codebook.

13. The system of claim 1, wherein the distance function is crafted for each codebook graph based on a particular set of attributes.

14. The system of claim 1, where the distance function is a generic distance based on measured ranges of attributes.

15. The system of claim 1, comprising sounding an audible alert, a 3D-localized audible alert to attract a driver'"'"'s attention to a location of a potential danger, or a voice alert telling the driver about a danger.

16. The system of claim 1, comprising displaying one or more of:

a warning message on the dashboard, a warning message on the windshield, a warning hologram.

17. The system of claim 1, comprising illuminating or projecting a highlight on the windshield, side windows or car interior to locate traffic participants involved in a dangerous situation.

18. The system of claim 1, vibrating the steering wheel, brakes, accelerator or seat to warn of danger.

19. The system of claim 1, comprising acting on the brakes or an ability of a driver to accelerate.

20. A method executed on a vehicle guidance system with sensors to capture surrounding traffic participants and information regarding a driving situation to assist a driver with a dangerous condition, comprising:

creating a graph representation where traffic participants and static elements are the vertices and the edges are relations between pairs of vertices;

adding attributes to the vertices and edges of the graph based on information obtained on the driving vehicle, the traffic participants and additional information;

creating or obtaining a codebook of dangerous driving situations, each represented as graphs;

performing subgraph matching between the graphs in the codebook and the graph representing a current driving situation to select a set of matching graphs from the codebook;

calculating a distance between each selected codebook graph and the matching subgraph of the current driving situation;

determining potential dangers from codebook graphs with a low distance; and

generating an alert if one or more of the codebook dangers are imminent.