Systems and methods for prioritizing object prediction for autonomous vehicles

• US 10,216,189 B1
• Filed: 11/14/2017
• Issued: 02/26/2019
• Est. Priority Date: 08/23/2017
• Status: Active Grant

• Current Assignee: Aurora Operations Incorporated (Aurora Innovation, Inc.)
• Original Assignee: Uber Technologies, Inc.
• Inventors: Haynes, Galen Clark
• Primary Examiner(s): Hutchinson, Alan D
• Application Number:
  US15/811,865
• Publication Number:
  US 20190064815A1
• Time in Patent Office: 469 Days
• CPC Class Codes: B60W 2050/065 by reducing the computational load on the digital processor of the control computer
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2554/402 Type
• CPC Class Codes: B60W 30/095 Predicting travel path or likelihood of collision
• CPC Class Codes: B60W 30/0956 the prediction being responsive to traffic or environmental parameters
• CPC Class Codes: B60W 40/00 Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
• CPC Class Codes: B60W 50/06 Improving the dynamic response of the control system, e.g. improving the speed of regulation or avoiding hunting or overshoot
• CPC Class Codes: B60W 60/00276 for two or more other traffic participants
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0212 with means for defining a desired trajectory involving a plurality of land vehicles G05D1/0287
• CPC Class Codes: G05D 1/0221 involving a learning process
• CPC Class Codes: G05D 1/0231 using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G05D 1/0257 using a radar radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G08G 1/0112 from the vehicle, e.g. floating car data [FCD]
• CPC Class Codes: G08G 1/0129 for creating historical data or processing based on historical data
• CPC Class Codes: G08G 1/0133 : for classifying traffic situation

Abstract

Systems and methods for determining object prioritization and predicting future object locations for an autonomous vehicle are provided. A method can include obtaining, by a computing system comprising one or more processors, state data descriptive of at least a current or past state of a plurality of objects that are perceived by an autonomous vehicle. The method can further include determining, by the computing system, a priority classification for each object in the plurality of objects based at least in part on the respective state data for each object. The method can further include determining, by the computing system, an order at which the computing system determines a predicted future state for each object based at least in part on the priority classification for each object and determining, by the computing system, the predicted future state for each object based at least in part on the determined order.

Claims

1. A computer-implemented method, comprising:

obtaining, by a computing system comprising one or more processors, state data descriptive of at least a current or past state of a plurality of objects that are perceived by an autonomous vehicle;

determining, by the computing system, a priority classification for each object in the plurality of objects based at least in part on the respective state data for each object;

determining, by the computing system, an order at which the computing system determines a predicted future state for each object based at least in part on the priority classification for each object;

selecting, by the computing system, a future location prediction system based at least in part on a priority classification for at least one of the plurality of objects; and

determining, by the computing system, the predicted future state for each object based at least in part on the determined order, wherein determining the predicted future state for the at least one of the plurality of objects comprises using the selected future location prediction system.

2. The computer-implemented method of claim 1, further comprising:

determining, by the computing system, a motion plan for the autonomous vehicle based at least in part on the predicted future state for the at least one of the objects.

3. The computer-implemented method of claim 1, wherein determining, by the computing system, the priority classification for each object in the plurality based at least in part on the respective state data for each object comprises:

classifying, by the computing system, each object as either high-priority or low-priority.

4. The computer-implemented method of claim 3, wherein obtaining, by the computing system, state data descriptive of at least the current or past state of the plurality of objects that are perceived by the autonomous vehicle comprises:

obtaining, by the computing system for a plurality of consecutive time frames, state data descriptive of the current state of the plurality of objects that are perceived by the autonomous vehicle.

5. The computer-implemented method of claim 4, wherein determining, by the computing system, the predicted future state for an object classified as high-priority comprises:

determining, by the computing system, the predicted future state for the object based at least in part on state data obtained for the most recent time frame.

6. The computer-implemented method of claim 4, wherein determining, by the computing system, the predicted future state for an object classified as low-priority comprises:

determining, by the computing system, a predicted future state for the object classified as low-priority based on state data obtained for a previous sequential time frame.

7. The computer-implemented method of claim 1, wherein the future location prediction system comprises either a low-fidelity prediction system or a high-fidelity prediction system;

wherein the priority classification for each object comprises either a high-priority or low-priority classification;

wherein determining, by the computing system, the predicted future state for each low-priority object comprises determining, by the computing system, the predicted future state for the object using the low-fidelity prediction system; and

wherein determining, by the computing system, the predicted future state for each high-priority object comprises determining, by the computing system, the predicted future state for the object using the high-fidelity prediction system.

8. The computer-implemented method of claim 1, wherein determining, by the computing system, the priority classification for each object comprises:

determining, by the computing system, a ratio between a first number of the objects to be classified as high-priority objects versus a second number of the objects to be classified as low-priority objects;

wherein the ratio is a predetermined ratio associated with a threshold velocity or velocity range of the autonomous vehicle.

9. The computer-implemented method of claim 1, wherein determining, by the computing system, the priority classification for each object further comprises:

determining, by the computing system, the priority classification for each object based at least in part on one or more features of the object;

wherein the one or more features for each object comprise one or more of;

a position, a velocity, an acceleration, a heading, a yaw rate, a shape, a size, a type, a distance from the autonomous vehicle, a minimum path to interaction with the autonomous vehicle, or a minimum time duration to interaction with the autonomous vehicle.

10. The computer-implemented method of claim 1, wherein determining, by the computing system, the priority classification for each object based at least in part on the respective state data for each object comprises:

determining, by the computing system, the priority classification for each object using a machine-learned model.

11. The computer-implemented method of claim 10, wherein the machine-learned model has been trained based at least in part on training data that comprises annotated vehicle data logs that were previously collected during previous autonomous vehicle driving sessions.

12. The computer-implemented method of claim 10, wherein determining, by the computing system, the priority classification for each object using the machine-learned model comprises:

obtaining, by the computing system, data descriptive of the machine-learned model;

inputting, by the computing system, the respective state data for each object into the machine-learned model; and

receiving, by the computing system, data indicative of a respective priority classification for each object as an output of the machine-learned model.

13. The computer-implemented method of claim 1, wherein the computing system is on-board the autonomous vehicle.

14. A computing system, comprising:

a perception system comprising one or more processors, wherein the perception system is configured to generate, for each of a plurality of consecutive time frames, state data descriptive of at least a current state of each of a plurality of objects that are perceived by an autonomous vehicle;

a priority classification system comprising one or more processors, wherein the priority classification system is configured to, for each of the plurality of consecutive time frames, classify each object in the plurality of objects as either high-priority or low-priority based at least in part on the respective state data for each object; and

a prediction system comprising one or more processors, wherein the prediction system is configured to, for each of the plurality of consecutive time frames;

receive the priority classification for each respective object;

determine, for a current time frame, a predicted future state for each object classified as high-priority;

provide the predicted future state for each object classified as high-priority for the current time frame to a motion planning system implemented by the one or more processors; and

provide a predicted future state for each object classified as low-priority for a previous sequential time frame to the motion planning system;

wherein the predicted future state for each object classified as low-priority for the previous sequential time frame is provided to the motion planning system concurrently with the predicted future state for each object classified as high priority for the current time frame.

15. The computing system of claim 14, wherein following providing the predicted future state for each object classified as high-priority to the motion planning system, the prediction system is further configured to:

determine a predicted future state for the current time frame for each object classified as low-priority.

16. The computing system of claim 14, wherein the priority classification system comprises a machine-learned object priority classifier configured to classify each respective object as either high-priority or low-priority.

17. The computing system of claim 14, wherein the prediction system is further configured to perform a low-fidelity prediction for objects classified as low-priority;

and

wherein the prediction system is further configured to perform a high-fidelity prediction for objects classified as high-priority.

18. An autonomous vehicle, comprising:

one or more processors; and

one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising;

obtaining state data descriptive of at least a current or past state of a plurality of objects that are perceived by the autonomous vehicle;

determining a priority classification for each object in the plurality of objects based at least in part on the respective state data for each object, wherein determining the priority classification for each object comprises determining a ratio between a first number of the objects to be classified as high-priority objects versus a second number of the objects to be classified as low-priority objects based on a threshold velocity or velocity range of the autonomous vehicle;

determining an order at which the computing system determines a predicted future state for each object based at least in part on the priority classification for each object; and

determining the predicted future state for each object based at least in part on the determined order.

19. A computer-implemented method, comprising:

obtaining, by a computing system comprising one or more processors and for a plurality of consecutive time frames, state data descriptive of at least a current or past state of a plurality of objects that are perceived by an autonomous vehicle;

determining, by the computing system, a priority classification for each object in the plurality of objects based at least in part on the respective state data for each object, wherein determining the priority classification for the plurality of objects comprises classifying a first object as low-priority;

determining, by the computing system, an order at which the computing system determines a predicted future state for each object based at least in part on the priority classification for each object; and

determining, by the computing system, the predicted future state for each object based at least in part on the determined order, wherein determining the predicted future state for the first object classified as low-priority is based on state data for a previous sequential time frame.

20. A computing system, comprising:

a perception system comprising one or more processors, wherein the perception system is configured to generate, for each of a plurality of consecutive time frames, state data descriptive of at least a current state of each of a plurality of objects that are perceived by an autonomous vehicle;

a priority classification system comprising one or more processors, wherein the priority classification system is configured to, for each of the plurality of consecutive time frames, classify each object in the plurality of objects as either high-priority or low-priority based at least in part on the respective state data for each object; and

a prediction system comprising one or more processors, wherein the prediction system is configured to, for each of the plurality of consecutive time frames;

receive the priority classification for each respective object;

determine, for a current time frame, a predicted future state for each object classified as high-priority by performing a high-fidelity prediction; and

determine, for the current time frame, a predicted future state for each object classified as low-priority by performing a low-fidelity prediction.

21. The computing system of claim 20, wherein the prediction system is configured to:

provide the predicted future state for each object classified as high-priority for the current time frame to a motion planning system implemented by the one or more processors; and

provide the predicted future state for each object classified as low-priority for the current time frame to the motion planning system subsequent to providing the predicted future state for each object classified as high-priority for the current time frame.

22. The computing system of claim 21, wherein the prediction system is configured to:

provide a predicted future state for each object classified as low-priority for a previous sequential time frame to the motion planning system; and

wherein the predicted future state for each object classified as low-priority for the previous sequential time frame is provided to the motion planning system concurrently with the predicted future state for each object classified as high priority for the current time frame.

23. An autonomous vehicle, comprising:

one or more processors; and

one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising;

obtaining, for each of a plurality of consecutive time frames, state data descriptive of at least a current state of each of a plurality of objects that are perceived by the autonomous vehicle;

determining a priority classification for each object in the plurality of objects based at least in part on the respective state data for each object;

determining, for a current time frame, a predicted future state for each object in the plurality of objects; and

providing the predicted future state for each object classified as high-priority for the current time frame to a motion planning system prior to providing the predicted future state for each object classified as low-priority for the current time frame.

24. The autonomous vehicle of claim 23, wherein the operations further comprise:

determining, for a previous sequential time frame, the predicted future state for each object classified as low priority; and

providing, to the motion planning system, a predicted future state for each object classified as low-priority for the previous sequential time frame;

wherein the predicted future state for each object classified as low-priority for the previous sequential time frame is provided to the motion planning system concurrently with the predicted future state for each object classified as high priority for the current time frame.