Advanced threat warning for autonomous vehicles

• US 10,580,296 B2
• Filed: 12/06/2017
• Issued: 03/03/2020
• Est. Priority Date: 12/06/2016
• Status: Active Grant

• Current Assignee: Nissan Motor Co., Ltd., United States of America as represented by the Administrator of NASA
• Original Assignee: Nissan North America Incorporated (Nissan Motor Co., Ltd.)
• Inventors: Pedersen, Liam, Sierhuis, Maarten, Utz, Hans, Penna, Mauro Della, Fong, Terrence, Allan, Mark, Bualat, Maria, Schafer, Eric
• Primary Examiner(s): Hilgendorf, Dale W
• Application Number:
  US16/461,200
• Publication Number:
  US 20190311621A1
• Time in Patent Office: 818 Days
• CPC Class Codes: B60W 2050/0075 Automatic parameter input, automatic initialising or calibrating means
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 2530/209 Fuel quantity remaining in tank
• CPC Class Codes: B60W 2552/00 Input parameters relating to infrastructure
• CPC Class Codes: B60W 2552/35 Road bumpiness, e.g. potholes
• CPC Class Codes: B60W 2552/40 Coefficient of friction
• CPC Class Codes: B60W 2552/50 Barriers
• CPC Class Codes: B60W 2552/53 Road markings, e.g. lane marker or crosswalk
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2554/4029 Pedestrians
• CPC Class Codes: B60W 2554/406 Traffic density
• CPC Class Codes: B60W 2554/80 Spatial relation or speed relative to objects
• CPC Class Codes: B60W 2556/10 Historical data
• CPC Class Codes: B60W 2556/50 of positioning data, e.g. GPS [Global Positioning System] data
• CPC Class Codes: B60W 2556/65 Data transmitted between vehicles
• CPC Class Codes: B60W 30/0956 the prediction being responsive to traffic or environmental parameters
• CPC Class Codes: B60W 40/04 Traffic conditions
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: G01C 21/3407 specially adapted for specific applications
• CPC Class Codes: G01C 21/3461 Preferred or disfavoured areas, e.g. dangerous zones, toll or emission zones, intersections, manoeuvre types, segments such as motorways, toll roads, ferries
• CPC Class Codes: G01C 21/3492 : employing speed data or traffic data, e.g. real-time or historical traffic control systems for road vehicles involving transmission of navigation instructions to the vehicle G08G1/0968
• CPC Class Codes: G01C 21/3691 : Retrieval, searching and output of information related to real-time traffic, weather, or environmental conditions arrangements for giving variable traffic instructions G08G1/09
• CPC Class Codes: G01C 21/3697 : Output of additional, non-guidance related information, e.g. low fuel level G01C21/3679 takes precedence
• CPC Class Codes: G05D 1/0276 : using signals provided by a source external to the vehicle involving a plurality of vehicles G05D1/0287; automatically controlling vehicle speed responsive to externally generated signals B60K31/0058
• CPC Class Codes: G08G 1/0133 : for classifying traffic situation
• CPC Class Codes: G08G 1/0962 : having an indicator mounted inside the vehicle, e.g. giving voice messages
• CPC Class Codes: G08G 1/0967 : Systems involving transmission of highway information, e.g. weather, speed limits transmission of navigation instructions to the vehicle G08G1/0968
• CPC Class Codes: G08G 1/096708 : where the received information might be used to generate an automatic action on the vehicle control
• CPC Class Codes: G08G 1/096775 : where the origin of the information is a central station
• CPC Class Codes: G08G 1/096791 : where the origin of the information is another vehicle
• CPC Class Codes: G08G 1/096811 : where the route is computed offboard
• CPC Class Codes: G08G 1/163 : involving continuous checking
• CPC Class Codes: G08G 1/164 : Centralised systems, e.g. external to vehicles
• CPC Class Codes: G08G 1/166 : for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

Methods, apparatuses, systems, and non-transitory computer readable storage media for generating risk indicators are described. The disclosed technology includes determining a vehicle route of a vehicle and external object routes of external objects. The vehicle route is determined using vehicle route data including a vehicle location and a vehicle destination. The external object routes are determined using external object route data including external object locations and external object destinations. Based on a comparison of the vehicle route data and the external object route data, external object routes that satisfy a proximity criterion are determined. Risk data for the vehicle is generated based on a vehicle state of the vehicle and external object states of the external objects corresponding to the external object routes that satisfy the proximity criterion. In response to determining that the risk data satisfies a risk criterion, at least one risk indicator is generated.

Claims

1. A method for generating risk indicators, the method comprising:

determining a vehicle route of a vehicle using vehicle route data including a current vehicle location, a current vehicle destination, and historical vehicle route data;

determining external object routes of external objects using external object route data including current external object locations, current external object destinations, and historical external object route data;

comparing the vehicle route data to the external object route data to determine external object routes that satisfy a proximity criterion, the external object routes that satisfy the proximity criterion being within a predefined proximity of the vehicle route during a predefined time period that is determined using past time periods associated with the vehicle;

generating risk data for the vehicle based on a vehicle state of the vehicle and external object states of the external objects corresponding to the external object routes that satisfy the proximity criterion, the risk data including any of proximity information, severity information, and probability information associated with an adverse event; and

in response to determining that the risk data satisfies a risk criterion, generating at least one risk indicator including a risk type.

2. The method of claim 1, further comprising:

generating traffic flow data based on the external object route data including velocities, densities, and relative locations of external objects corresponding to the external object routes that satisfy the proximity criterion; and

in response to determining that the traffic flow data satisfies a traffic flow criterion, generating external object indicators of the external objects corresponding to the external object routes that satisfy the proximity criterion.

3. The method of claim 1, further comprising:

receiving sensor data from sensors of any of the vehicle and the external objects, wherein any of the vehicle state and the external object states are based on the sensor data.

4. The method of claim 1, further comprising:

generating traffic control device data based on a state of one or more traffic control devices located along the vehicle route; and

in response to determining, based on the traffic control device data, that at least one of the one or more traffic control devices is malfunctioning, determining a malfunction type of the one or more traffic control devices, the risk data including the malfunction type.

5. The method of claim 1, further comprising:

determining a route condition using route condition data of the vehicle route and the external object routes;

modifying the risk data using the route condition data; and

generating route condition indicators, using the route condition data, for the vehicle route and the external object routes that satisfy the proximity criterion.

6. The method of claim 5, wherein the route condition data includes any of vehicle stoppage data, construction activity data, lane closure data, road surface condition data, pedestrian activity data, and traffic signal timing data.

7. The method of claim 1, further comprising:

determining, based on the external object route data, a time period during which a congestion event is occurring in a predetermined area including a portion of the vehicle route; and

in response to determining that a time the vehicle will enter the predetermined area corresponds to the time period during which the congestion event is occurring, generating one or more congestion indicators for external objects within the portion of the vehicle route.

8. A risk indicator apparatus comprising:

a memory; and

a processor configured to execute instructions stored in the memory to;

determine a vehicle route of a vehicle using vehicle route data including a current vehicle location, a current vehicle destination, and historical vehicle route data;

determine external object routes of external objects using external object route data including current external object locations, current external object destinations, and historical external object route data;

compare the vehicle route data to the external object route data to determine external object routes that satisfy a proximity criterion, the external object routes that satisfy the proximity criterion being within a predefined proximity of the vehicle route during a predefined time period that is determined using past time periods associated with the vehicle;

generate risk data for the vehicle based on a vehicle state of the vehicle and external object states of the external objects corresponding to the external object routes that satisfy the proximity criterion, the risk data including any of proximity information, severity information, and probability information associated with an adverse event; and

in response to determining that the risk data satisfies a risk criterion, generate at least one risk indicator including a risk type.

9. The risk indicator apparatus of claim 8, wherein the processor is further configured to execute instructions stored in the memory to:

generate traffic flow data based on the external object route data including velocities, densities, and relative locations of external objects corresponding to the external object routes that satisfy the proximity criterion; and

in response to determining that the traffic flow data satisfies a traffic flow criterion, generate external object indicators of the external objects corresponding to the external object routes that satisfy the proximity criterion.

10. The risk indicator apparatus of claim 8, wherein the processor is further configured to execute instructions stored in the memory to:

receive sensor data from sensors of any of the vehicle and the external objects, wherein any of the vehicle state and the external object states is based on the sensor data.

11. The risk indicator apparatus of claim 8, wherein the processor is further configured to execute instructions stored in the memory to:

generate traffic control device data based on a state of one or more traffic control devices located along the vehicle route; and

in response to determining, based on the traffic control device data, that at least one of the one or more traffic control devices is malfunctioning, determine a malfunction type of the one or more traffic control devices, the risk data including the malfunction type.

12. The risk indicator apparatus of claim 8, wherein the processor is further configured to execute instructions stored in the memory to:

determine a route condition using route condition data of the vehicle route and the external object routes;

modify the risk data using the route condition data; and

generate route condition indicators, using the route condition data, for the vehicle route and the external object routes that satisfy the proximity criterion.

13. The risk indicator apparatus of claim 12, wherein the route condition data includes any of vehicle stoppage data, construction activity data, lane closure data, road surface condition data, pedestrian activity data, and traffic signal timing data.

14. The risk indicator apparatus of claim 8, wherein the processor is further configured to execute instructions stored in the memory to:

determine, based on the external object route data, a time period during which a congestion event is occurring in a predetermined area including a portion of the vehicle route; and

in response to determining that a time the vehicle will enter the predetermined area corresponds to the time period during which the congestion event is occurring, generate one or more congestion indicators for external objects within the portion of the vehicle route.

15. A non-transitory computer-readable storage medium including program instructions executable by one or more processors of a risk indicator apparatus that, when executed, cause the one or more processors to perform operations, the operations comprising:

determining a vehicle route of a vehicle using vehicle route data including a current vehicle location, a current vehicle destination, and historical vehicle route data;

determining external object routes of external objects using external object route data including current external object locations, current external object destinations, and historical external object route data;

comparing the vehicle route data to the external object route data to determine external object routes that satisfy a proximity criterion, the external object routes that satisfy the proximity criterion being within a predefined proximity of the vehicle route during a predefined time period that is determined using past time periods associated with the vehicle;

generating risk data for the vehicle based on a vehicle state of the vehicle and external object states of the external objects corresponding to the external object routes that satisfy the proximity criterion, the risk data including any of proximity information, severity information, and probability information associated with an adverse event; and

in response to determining that the risk data satisfies a risk criterion, generating at least one risk indicator including a risk type.

16. The non-transitory computer-readable storage medium of claim 15, the operations further comprising:

generating traffic flow data based on the external object route data including velocities, densities, and relative locations of external objects corresponding to the external object routes that satisfy the proximity criterion; and

in response to determining that the traffic flow data satisfies a traffic flow criterion, generating external object indicators of the external objects corresponding to the external object routes that satisfy the proximity criterion.

17. The non-transitory computer-readable storage medium of claim 15, the operations further comprising:

receiving sensor data from sensors of any of the vehicle and the external objects, wherein any of the vehicle state and the external object states is based on the sensor data.

18. The non-transitory computer-readable storage medium of claim 15, the operations further comprising:

generating traffic control device data based on a state of one or more traffic control devices located along the vehicle route; and

in response to determining, based on the traffic control device data, that at least one of the one or more traffic control devices is malfunctioning, determining a malfunction type of the one or more traffic control devices, the risk data including the malfunction type.

19. The non-transitory computer-readable storage medium of claim 15, the operations further comprising:

determining a route condition using route condition data of the vehicle route and the external object routes;

modifying the risk data using the route condition data; and

generating route condition indicators, using the route condition data, for the vehicle route and the external object routes that satisfy the proximity criterion.

20. The non-transitory computer-readable storage medium of claim 15, the operations further comprising:

determining, based on the external object route data, a time period during which a congestion event is occurring in a predetermined area including a portion of the vehicle route; and

in response to determining that a time the vehicle will enter the predetermined area corresponds to the time period during which the congestion event is occurring, generating one or more congestion indicators for external objects within the portion of the vehicle route.