Machine learning for event detection and classification in autonomous vehicles

• US 10,520,947 B2
• Filed: 03/27/2017
• Issued: 12/31/2019
• Est. Priority Date: 03/27/2017
• Status: Active Grant

• Current Assignee: Uatc, LLC (Uber Technologies, Inc.)
• Original Assignee: Uatc, LLC (Uber Technologies, Inc.)
• Inventors: Liu, Chenggang, Bares, John Evan
• Primary Examiner(s): Smith, Isaac G
• Application Number:
  US15/469,981
• Publication Number:
  US 20180275667A1
• Time in Patent Office: 1,009 Days
• Field of Search: 701 27, 701 322
• CPC Class Codes: B60W 2556/10 Historical data
• CPC Class Codes: B60W 30/095 Predicting travel path or likelihood of collision
• CPC Class Codes: B60W 50/029 Adapting to failures or work around with other constraints, e.g. circumvention by avoiding use of failed parts
• CPC Class Codes: G01C 21/26 specially adapted for navigation in a road network
• CPC Class Codes: G05D 1/0221 involving a learning process

Abstract

The present disclosure provides systems and methods for automatic event detection and classification for autonomous vehicles. One example method includes obtaining, by one or more computing devices, vehicle data descriptive of vehicle conditions associated with an autonomous vehicle during an autonomous driving session. The method includes extracting, by the one or more computing devices, a plurality of features from the vehicle data. The method includes determining, by the one or more computing devices using a machine-learned classifier, a classification for each of one or more candidate events based at least in part on one or more of the plurality of features that are respectively associated with the one or more candidate events. The method includes associating, by the one or more computing devices, the classification determined for each of the one or more candidate events with the vehicle data.

Claims

1. A computer-implemented method to detect uncomfortable driving events performed by autonomous vehicles, the method comprising:

obtaining, by one or more computing devices, training data that comprises vehicle data logs that were previously collected during previous autonomous vehicle driving sessions, each of the vehicle data logs annotated with event labels that were provided by human passengers during one of the previous autonomous vehicle driving sessions, each event label having a respective label time associated therewith;

analyzing, by the one or more computing devices, each vehicle data log to identify one or more potentially referenced events for each event label, wherein each of the one or more potentially referenced events identified for each event label has an associated event time that is included in a time window prior to the respective label time associated with such event label;

assigning, by the one or more computing devices, each event label to at least one of the one or more potentially referenced events identified for such event label, such that the event label is associated with vehicle data collected at the respective event time which is prior to the respective label time;

after assigning the event labels to the potentially referenced events, training, by the one or more computing devices, a machine-learned classifier using the training data comprising the event labels assigned to the potentially referenced events at the respective event times; and

after training the machine-learned classifier;

obtaining, by the one or more computing devices, vehicle data descriptive of vehicle conditions associated with an autonomous vehicle during an autonomous driving session;

extracting, by the one or more computing devices, a plurality of features from the vehicle data;

determining, by the one or more computing devices using the machine-learned classifier, a classification for each of one or more candidate events based at least in part on one or more of the plurality of features that are respectively associated with the one or more candidate events; and

associating, by the one or more computing devices, the classification determined for each of the one or more candidate events with the vehicle data.

2. The computer-implemented method of claim 1, wherein extracting, by the one or more computing devices, the plurality of features from the vehicle data comprises performing, by the one or more computing devices, a continuous wavelet transform on the vehicle data to extract the plurality of features.

3. The computer-implemented method of claim 2, wherein performing, by the one or more computing devices, the continuous wavelet transform comprises performing, by the one or more computing devices, the continuous wavelet transform on the vehicle data with a Mexican Hat wavelet to extract the plurality of features.

4. The computer-implemented method of claim 1, wherein the plurality of features comprise one or more of:

a speed feature, a steering angle feature, and a lateral acceleration feature.

5. The computer-implemented method of claim 1, wherein extracting, by the one or more computing devices, the plurality of features from the vehicle data comprises:

determining, by the one or more computing devices, a plurality of scale components for each of one or more vehicle data channels; and

computing, by the one or more computing devices for each of the one or more vehicle data channels, a set of statistics that describe the plurality of scale components for such vehicle data channel.

6. The computer-implemented method of claim 5, wherein computing, by the one or more computing devices for each of the one or more vehicle data channels, the set of statistics that describe the plurality of scale components for such vehicle data channel comprises computing, by the one or more computing devices for each of the plurality of features, one or more of:

a mean of the plurality of scale components, a standard deviation of the plurality of scale components, a maximum of the plurality of scale components, a minimum of the plurality of scale components, and a moment of the plurality of scale components.

7. The computer-implemented method of claim 1, wherein:

extracting, by the one or more computing devices, the plurality of features comprises determining, by the one or more computing devices, a plurality of scale components; and

the method further comprises identifying, by the one or more computing devices, one or more relative peaks associated with the plurality of scale components, each of the one or more relative peaks corresponding to one of the one or more candidate events.

8. The computer-implemented method of claim 1, wherein determining, by the one or more computing devices using a machine-learned classifier, the classification for each of the one or more candidate events comprises assigning, by the one or more computing devices, at least one of the candidate events to one or more of the following classifications:

a high deceleration classification, a high acceleration classification, a juke classification, a jerk classification, and a weaving classification.

9. The computer-implemented method of claim 1, wherein determining, by the one or more computing devices using the machine-learned classifier, the classification for each of the one or more candidate events comprises, for each candidate event:

inputting, by the one or more computing devices, the one or more features associated with the candidate event into the machine-learned classifier, wherein the machine-learned classifier comprises one or more of a logistic regression classifier, a support vector machine, and a neural network; and

receiving, by the one or more computing devices, the classification for the candidate event as an output of the machine-learned classifier.

10. The computer-implemented method of claim 1, wherein the one or more computing devices are on-board the autonomous vehicle and the method is iteratively performed in real-time as the autonomous vehicle operates to execute the autonomous driving session.

11. A computer system, comprising:

one or more processors; and

one or more tangible, non-transitory, computer readable media that collectively store;

at least one vehicle data log that was collected during a previous autonomous vehicle driving session, the vehicle data log descriptive of vehicle conditions associated with an autonomous vehicle during the previous autonomous vehicle driving session, the vehicle data log annotated with a plurality of event labels respectively at a plurality of label times; and

instructions that, when executed by the one or more processors, cause the computer system to;

analyze the vehicle data log to identify one or more potentially referenced events for each event label, wherein each of the one or more potentially referenced events identified for each event label has an associated event time that is included in a time window prior to the respective label time associated with such event label;

assign each event label to at least one of the one or more potentially referenced events identified for such event label;

extract, for the event time associated with each potentially referenced event to which one of the event labels has been assigned, one or more features from the vehicle data log;

associate each event label with the one or more features extracted from the vehicle data log for the event time associated with each potentially referenced event to which one of the event labels has been assigned, such that the event label is associated with features extracted from the vehicle data log collected at the respective event time which is prior to the respective label time; and

train a classifier model to perform event classification based at least in part on the plurality of event labels and the one or more features respectively associated therewith.

12. The computer system of claim 11, wherein the classifier model comprises one or more of a logistic regression classifier, a support vector machine, and a neural network.

13. The computer system of claim 11, wherein to train the classifier model to perform event classification based at least in part on the plurality of event labels and the one or more features respectively associated therewith, the instructions further cause the computer system to:

input, for each of the plurality of times, the one or more features extracted for such time into the classifier model;

receive, for each of the plurality of times, at least one classification as an output of the classifier model;

determine, for each of the plurality of times, an objective function that describes a difference between the at least one classification for such time and the event label associated with such time; and

train the classifier model based at least in part on the objective function.

14. The computer system of claim 11, wherein to extract, for the event time associated with each potentially referenced event to which one of the event labels has been assigned, the one or more features from the vehicle data log the instructions further cause the computer system to perform, for the event time associated with each potentially referenced event to which one of the event labels has been assigned, a continuous wavelet transform on the vehicle data log to extract the one or more features.

15. The computer system of claim 11, wherein to extract, for the event time associated with each potentially referenced event to which one of the event labels has been assigned, the one or more features from the vehicle data log the instructions further cause the computer system to:

determine, for the event time associated with each potentially referenced event to which one of the event labels has been assigned, a plurality of scale components for each of one or more vehicle data channels; and

compute, for the event time associated with each potentially referenced event to which one of the event labels has been assigned and for each of the one or more vehicle data channels, a set of statistics that describe the plurality of scale components for such vehicle data channel at such time.

16. A computer system, comprising:

one or more processors;

a machine-learned classifier model; and

one or more tangible, non-transitory, computer-readable media that store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising;

obtaining training data that comprises vehicle data logs that were previously collected during previous autonomous vehicle driving sessions, each of the vehicle data logs annotated with event labels that were provided by human passengers during one of the previous autonomous vehicle driving sessions, each event label having a respective label time associated therewith;

analyzing each vehicle data log to identify one or more potentially referenced events for each event label, wherein each of the one or more potentially referenced events identified for each event label has an associated event time that is included in a time window prior to the respective label time associated with such event label;

assigning each event label to at least one of the one or more potentially referenced events identified for such event label such that the event label is associated with vehicle data collected at the respective event time which is prior to the respective label time;

after assigning the event labels to the potentially referenced events, training a machine-learned classifier using the training data comprising the event labels assigned to the potentially referenced events at the respective event times; and

after training the machine-learned classifier;

obtaining vehicle data descriptive of vehicle conditions associated with an autonomous vehicle during a driving session;

extracting a plurality of features from the vehicle data;

identifying a plurality of candidate events;

inputting, for each of the plurality of candidate events, the plurality of features into the machine-learned classifier model;

receiving, for each of the plurality of candidate events, a classification for the candidate event as an output of the machine-learned classifier model; and

associating the classification provided for each of the plurality of candidate events with the vehicle data.

17. The computer system of claim 16, wherein:

extracting the plurality of features from the vehicle data comprises performing a continuous wavelet transform on the vehicle data to extract the plurality of features.