Machine learning systems and techniques to optimize teleoperation and/or planner decisions

• US 10,048,683 B2
• Filed: 12/28/2016
• Issued: 08/14/2018
• Est. Priority Date: 11/04/2015
• Status: Active Grant

• Current Assignee: Zoox Inc. (Amazon.com, Inc.)
• Original Assignee: Zoox Inc. (Amazon.com, Inc.)
• Inventors: Levinson, Jesse Sol, Sibley, Gabriel Thurston, Rege, Ashutosh Gajanan
• Primary Examiner(s): Lee, Tyler J
• Application Number:
  US15/393,228
• Publication Number:
  US 20180136644A1
• Time in Patent Office: 594 Days
• Field of Search: 701 24
• CPC Class Codes: B60Q 1/28 for indicating front of vehicle
• CPC Class Codes: B60Q 1/30 for indicating rear of vehicle, e.g. by means of reflecting surfaces
• CPC Class Codes: B60Q 1/503 using luminous text or symbol displays in or on the vehicle, e.g. static text
• CPC Class Codes: B60Q 1/507 specific to autonomous vehicles
• CPC Class Codes: B60Q 1/508 specific to vehicles driving in fleets or convoys
• CPC Class Codes: B60Q 1/549 for expressing greetings, gratitude or emotions
• CPC Class Codes: B60Q 2400/40 Welcome lights, i.e. specific or existing exterior lamps to assist leaving or approaching the vehicle
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/87 Combinations of radar systems, e.g. primary radar and secondary radar
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 2013/9316 combined with communication equipment with other vehicles or with base stations
• CPC Class Codes: G01S 2013/9322 using additional data, e.g. driver condition, road state or weather data
• CPC Class Codes: G01S 7/4972 Alignment of sensor
• CPC Class Codes: G05D 1/0022 characterised by the communication link data switching networks in general H04L12/00
• CPC Class Codes: G05D 1/0027 involving a plurality of vehicles, e.g. fleet or convoy travelling fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 7/01 : Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06Q 10/00 : Administration; Management
• CPC Class Codes: G06Q 10/06 : Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G08G 1/005 : including pedestrian guidance indicator
• CPC Class Codes: G08G 1/165 : for passive traffic, e.g. including static obstacles, trees
• CPC Class Codes: G08G 1/166 : for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: G08G 1/202 : Dispatching vehicles on the basis of a location, e.g. taxi dispatching
• CPC Class Codes: H04L 41/0816 : the condition being an adaptation, e.g. in response to network events
• CPC Class Codes: H04L 41/16 : using machine learning or artificial intelligence
• CPC Class Codes: H04L 67/10 : in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/12 : specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks

Abstract

A system, an apparatus or a process may be configured to implement an application that applies artificial intelligence and/or machine-learning techniques to predict an optimal course of action (or a subset of courses of action) for an autonomous vehicle system (e.g., one or more of a planner of an autonomous vehicle, a simulator, or a teleoperator) to undertake based on suboptimal autonomous vehicle performance and/or changes in detected sensor data (e.g., new buildings, landmarks, potholes, etc.). The application may determine a subset of trajectories based on a number of decisions and interactions when resolving an anomaly due to an event or condition. The application may use aggregated sensor data from multiple autonomous vehicles to assist in identifying events or conditions that might affect travel (e.g., using semantic scene classification). An optimal subset of trajectories may be formed based on recommendations responsive to semantic changes (e.g., road construction).

Claims

1. An autonomous vehicle comprising:

one or more processors;

a communication interface, the communication interface configured to establish a communications link between the autonomous vehicle and one or more other devices or autonomous vehicles;

one or more sensors; and

memory having stored thereon processor-executable instructions that, when executed by the one or more processors, configure the autonomous vehicle to perform operations comprising;

obtaining sensor data from the one or more sensors;

detecting, from the sensor data, an event, the event being associated with operation of the autonomous vehicle;

determining a type of the event;

requesting, via the communication interface, a teleoperator command based, at least in part, on the type of the event;

receiving via the communication interface a teleoperator command; and

controlling the autonomous vehicle based, at least in part, on the teleoperator command.

2. The autonomous vehicle as claim 1 recites, wherein the communication link comprises two or more networks or subnetworks, the two or more networks or subnetworks configured to provide the communication link and a redundant communication link, and further wherein each of the one or more networks or subnetworks comprise one or more of a cellular network or a data network.

3. The autonomous vehicle as claim 1 recites, wherein the communication interface is configured to implement a communications protocol, the communications protocol comprising one or more of a real time encryption protocol or a quality of service.

4. The autonomous vehicle as claim 1 recites, wherein the teleoperator command is determined based, at least in part, on policy data.

5. The autonomous vehicle as claim 4 recites, further comprising:

a communication event detector, the communication event detector configured to output a communication level, and

wherein the operations further comprise selecting, based at least in part, on the communication level updated policy data.

6. The autonomous vehicle of claim 5, wherein the updated policy data comprises instructions configured to control the autonomous vehicle to navigate out of a region.

7. The autonomous vehicle as claim 1 recites, the operations further comprising:

based at least in part on the type of the event, determining a confidence level that indicates a probability that the event will impact operation of the autonomous vehicle, the type of the event indicating a criticality of the event; and

wherein detecting the event or determining the type of the event includes one or more of;

determining that the autonomous vehicle is approaching a region,

determining a time of day,

determining a position of the sun,

determining that there is one or more of traffic or an accident,

determining that the autonomous vehicle has been immobile for a threshold amount of time,

determining that the confidence level is below a threshold, or

determining that previously determined confidence levels exhibit a decreasing trend.

8. A method comprising:

obtaining sensor data from one or more sensors, the one or more sensors located on an autonomous vehicle;

determining based, at least in part, on the sensor data that a type of event has occurred;

transmitting a request for a teleoperator command based, at least in part, on the type of event;

receiving, via a communication interface, a teleoperator command, the teleoperator command based, at least in part, on one or more of policy data or the type of event; and

controlling the autonomous vehicle based, at least in part, on the teleoperator command, wherein the communication interface is configured to send and receive one or more data messages over a communication link, wherein the type of event includes one or more of;

the autonomous vehicle approaching a region,

a time of day,

a position of the sun,

traffic,

an accident,

the autonomous vehicle has been immobile for a threshold amount of time, or

a confidence level determined from the sensor data is below a threshold.

9. The method of claim 8, wherein the communications link is configured to transmit and receive the one or more data messages between the autonomous vehicle and another device.

10. The method of claim 8, wherein the another device includes another autonomous vehicle.

11. The method of claim 8, wherein the one or more sensors comprise at least one of a LIDAR, a radar, a sonar, or an image capture device.

12. The method of claim 8, wherein the communications link is configured to provide a primary communication link and a redundant communication link, the redundant communication link being between the autonomous vehicle and another device.

13. The method of claim 8, wherein the one or more data message is encrypted.

14. The method of claim 8, wherein the policy data is updated to create updated policy data based, at least in part, on the type of event, the updated policy data comprising instructions to control the autonomous vehicle to avoid a region, and

wherein the region comprises a geographic location where a bandwidth of one or more networks does not meet a threshold bandwidth for forming the communications link or successfully transmitting data over the communications link.

15. The method of claim 8, wherein the communication link comprises one or more of a cellular network or a data network.

16. The method of claim 8, wherein the confidence level is calculated by the autonomous vehicle based, at least in part, on the type of event, and indicates a probability that a course of action chosen by the autonomous vehicle will result in safe operation of the autonomous vehicle.

17. The method of claim 8 further comprising directing an available bandwidth of the communications link to be adjusted to maintain at least a threshold bandwidth.

18. One or more non-transitory computer-readable media storing computer-executable instructions configured to program one or more computing devices to perform operations comprising:

monitoring operation of one or more autonomous vehicles based at least in part on data received from the one or more autonomous vehicles;

detecting, from the monitoring, an event at an autonomous vehicle of the one or more autonomous vehicles associated with operation of the autonomous vehicle; and

obtaining sensor data from the autonomous vehicle based, at least in part, on the detecting of the event, the obtaining including one or more of;

transmitting, via a communication interface, instructions to the autonomous vehicle to transmit the sensor data for reception by the one or more computing devices, or

receiving, via the communication interface, the sensor data contemporaneously with the event from the autonomous vehicle,

wherein the communication interface configured to send and receive one or more data messages over a communication link.

19. One or more non-transitory computer-readable media as claim 18 recites, wherein the operations further comprise:

determining a type of the event, the type indicating a probability that the event will impact nominal driving operation of the autonomous vehicle; and

wherein the obtaining is further based at least in part on the type of the event.

20. The one or more non-transitory computer-readable media of claim 18, the operations further comprising establishing, via the communication interface, the communication link via a first communication channel and a redundant communication link via an alternate communication channel,

wherein establishing the redundant communication link includes establishing the redundant communication link with another autonomous vehicle.

21. The one or more non-transitory computer-readable media of claim 20, wherein establishing the alternate communication link with the first autonomous vehicle includes dispatching a second autonomous vehicle to park at a location to establish a peer-to-peer network with the autonomous vehicle.