Autonomous vehicle monitoring using generated interfaces

• US 10,839,473 B2
• Filed: 11/30/2017
• Issued: 11/17/2020
• Est. Priority Date: 11/30/2016
• Status: Active Grant

• Current Assignee: Florida Institute For Human and Machine Cognition, Nissan Motor Co., Ltd.
• Original Assignee: Nissan North America Incorporated (Nissan Motor Co., Ltd.)
• Inventors: Pedersen, Liam, Sierhuis, Maarten, Utz, Hans, Penna, Mauro Della, Bradshaw, Jeffrey, Johnson, Matthew, Vignati, Michael, Bunch, Lawrence
• Primary Examiner(s): Antonucci, Anne Marie
• Application Number:
  US16/349,486
• Publication Number:
  US 20200193549A1
• Time in Patent Office: 1,083 Days
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 10/047 Optimisation of routes or paths, e.g. travelling salesman problem
• CPC Class Codes: G06Q 10/063 Operations research, analysis or management
• CPC Class Codes: G06Q 10/20 Administration of product repair or maintenance
• CPC Class Codes: G06Q 50/40 Business processes related to the transportation industry shipping G06Q10/083
• CPC Class Codes: G07C 5/008 communicating information to a remotely located station transmission systems for measured values G08C
• CPC Class Codes: G07C 5/0825 using optical means
• CPC Class Codes: G08G 1/0112 from the vehicle, e.g. floating car data [FCD]
• CPC Class Codes: G08G 1/052 with provision for determining speed or overspeed speed measuring in general G01P
• CPC Class Codes: G08G 1/20 Monitoring the location of vehicles belonging to a group, e.g. fleet of vehicles, countable or determined number of vehicles

Abstract

Methods, apparatuses, systems, and non-transitory computer readable storage media for monitoring vehicles including autonomous vehicles are described. The disclosed technology includes a vehicle monitoring system that receives vehicle data and external data associated with a vehicle and a corresponding predetermined area. The vehicle data includes a vehicle state of the vehicle and the external data includes external states of external objects. An issue type of the vehicle is determined based on the vehicle state and at least one of the external states. An indication of the issue type is generated for display on an interface.

Claims

1. A method for vehicle monitoring, the method comprising:

receiving vehicle data and external data associated with a vehicle and a corresponding predetermined area, wherein the vehicle data includes a vehicle state of the vehicle and the external data includes external states of external objects;

determining an issue type of the vehicle based on the vehicle state and at least one of the external states; and

generating an indication of the issue type for display on an interface.

2. The method of claim 1, further comprising:

generating route data based on location states and temporal states of the vehicle relative to expected location states and expected temporal states of the vehicle, wherein the issue type is based on the route data.

3. The method of claim 2, further comprising:

generating expected destinations of the vehicle based on the expected location states and the expected temporal states;

grouping the external objects into clusters according to the expected destinations;

determining cluster sizes of each of the clusters; and

generating a congestion indication of each cluster that has a cluster size exceeding a threshold cluster size.

4. The method of claim 3, further comprising:

modifying the expected temporal states of a portion of the external objects in the clusters that exceed the threshold cluster size, wherein the modifying comprises minimizing differences between the expected temporal states within each of the clusters.

5. The method of claim 1, further comprising:

generating traffic data based on any of a number and a location of the external objects; and

determining a traffic issue based on the traffic data, wherein the indication of the issue type includes the traffic issue.

6. The method of claim 1, further comprising:

sending a state query message to the vehicle, wherein the state query message includes a request for the vehicle data; and

in response to a predetermined time period elapsing before receiving a response to the state query message, retrieving object sensor data of any of the external objects located within the predetermined area.

7. The method of claim 1, further comprising:

determining a velocity of the vehicle based on the vehicle data; and

in response to determining that the vehicle is in an immobile state based on the velocity of the vehicle, determining that the issue type is an immobility issue of the vehicle, wherein the indication of the issue type includes the immobility issue.

8. The method of claim 1, wherein the vehicle data is based on vehicle sensor data of at least one sensor in the vehicle and further comprising:

determining an occupant state value based on the vehicle sensor data; and

in response to the occupant state value satisfying an occupant state condition, generating an occupant state indication.

9. The method of claim 1, wherein the external data further comprises any of a vehicle velocity, velocities of the external objects, a vehicle orientation, orientations of the external objects, an autonomous mode state for the vehicle, a vehicle operational state, a vehicle energy state, traffic signal data, road closure data, construction data, road condition data, emergency vehicle route data, and zoning data.

10. A vehicle monitoring apparatus comprising:

a non-transitory memory; and

a processor configured to execute instructions stored in the non-transitory memory to;

receive vehicle data and external data associated with a vehicle and a corresponding predetermined area, wherein the vehicle data includes a vehicle state of a vehicle and the external data includes external states of external objects;

determine an issue type of the vehicle based on the vehicle state and at least one of the external states; and

generate an indication of the issue type for display on an interface.

11. The apparatus of claim 10, wherein the processor is further configured to execute instructions stored in the memory to:

generate route data based on location states and temporal states of the vehicle relative to expected location states and expected temporal states of the vehicle, wherein the issue type is based on the route data.

12. The apparatus of claim 11, wherein the processor is further configured to execute instructions stored in the memory to:

generate expected destinations of the vehicle based on the expected location states and the expected temporal states;

group the external objects into clusters according to the expected destinations;

determine cluster sizes of each of the clusters; and

generate a congestion indication of each cluster that has a cluster size exceeding a threshold cluster size.

13. The apparatus of claim 10, wherein the processor is further configured to execute instructions stored in the memory to:

generate traffic data based on any of a number and a location of the external objects; and

determine a traffic issue based on the traffic data, wherein the indication of the issue type includes the traffic issue.

14. The apparatus of claim 10, wherein the external data comprises any of vehicle velocity, velocities of the external objects, a vehicle orientation, orientations of the external objects, an autonomous mode state for the vehicle, a vehicle operational state, a vehicle energy state, traffic signal data, road closure data, construction data, road condition data, emergency vehicle route data, and zoning data.

15. The apparatus of claim 10, wherein the processor is further configured to execute instructions stored in the memory to:

sending a state query message to the vehicle, wherein the state query message includes a request for the vehicle data; and

in response to a predetermined time period elapsing before receiving a response to the state query message, retrieving object sensor data of any of the external objects located within the predetermined area.

16. A non-transitory computer-readable storage medium including program instructions executable by one or more processors that, when executed, cause the one or more processors to perform operations, the operations comprising:

receiving vehicle data and external data associated with a vehicle and a corresponding predetermined area, wherein the vehicle data includes a vehicle state of a vehicle and the external data includes an external states of external objects;

determining an issue type of the vehicle based on the vehicle state and at least one of the external states; and

generating an indication of the issue type for display on an interface.

17. The non-transitory computer-readable storage medium of claim 16, the operations further comprising:

generating route data based on location states and temporal states of the vehicle relative to expected location states and expected temporal states of the vehicle, wherein the issue type is based on the route data.

18. The non-transitory computer-readable storage medium of claim 17, the operations further comprising:

generating expected destinations of the vehicle based on the expected location states and the expected temporal states;

grouping the external objects into clusters according to the expected destinations;

determining cluster sizes of each of the clusters; and

generating a congestion indication of each cluster that has a cluster size exceeding a threshold cluster size.

19. The non-transitory computer-readable storage medium of claim 16, the operations further comprising:

generate traffic data based on any of a number and a location of the external objects; and

determine a traffic issue based on the traffic data, wherein the indication of the issue type includes the traffic issue.

20. The non-transitory computer-readable storage medium of claim 16, wherein the external data comprises any of vehicle velocity, velocities of the external objects, a vehicle orientation, orientations of the external objects, an autonomous mode state for the vehicle, a vehicle operational state, a vehicle energy state, traffic signal data, road closure data, construction data, road condition data, emergency vehicle route data, and zoning data.