Teleoperation system and method for trajectory modification of autonomous vehicles

• US 9,507,346 B1
• Filed: 11/04/2015
• Issued: 11/29/2016
• Est. Priority Date: 11/04/2015
• Status: Active Grant

• Current Assignee: Zoox Inc. (Amazon.com, Inc.)
• Original Assignee: Zoox Inc. (Amazon.com, Inc.)
• Inventors: Levinson, Jesse Sol, Sibley, Gabriel Thurston, Gamara, Rachad Youssef, Rege, Ashutosh Gajanan, Linscott, Gary, Kentley, Timothy David
• Primary Examiner(s): Nguyen, John Q
• Assistant Examiner(s): Sood, Anshul
• Application Number:
  US14/932,966
• Time in Patent Office: 391 Days
• Field of Search: 701/2
• US Class Current: 1/1
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2554/20 Static objects
• CPC Class Codes: B60W 2554/4026 Cycles
• CPC Class Codes: B60W 2554/4029 Pedestrians
• CPC Class Codes: B60W 2554/4041 Position
• CPC Class Codes: B60W 2554/80 Spatial relation or speed relative to objects
• CPC Class Codes: B60W 2556/50 of positioning data, e.g. GPS [Global Positioning System] data
• CPC Class Codes: B60W 2710/18 Braking system
• CPC Class Codes: B60W 2710/20 Steering systems
• CPC Class Codes: B60W 2710/30 Auxiliary equipments
• CPC Class Codes: B60W 30/09 Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: B60W 30/0956 the prediction being responsive to traffic or environmental parameters
• CPC Class Codes: B60W 60/0016 of the vehicle or its occupants
• CPC Class Codes: B60W 60/0027 using trajectory prediction for other traffic participants
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/87 Combinations of radar systems, e.g. primary radar and secondary radar
• CPC Class Codes: G01S 2013/9316 combined with communication equipment with other vehicles or with base stations
• CPC Class Codes: G01S 2013/9322 : using additional data, e.g. driver condition, road state or weather data
• CPC Class Codes: G01S 7/4808 : Evaluating distance, position or velocity data
• CPC Class Codes: G05D 1/0011 : associated with a remote control arrangement
• CPC Class Codes: G05D 1/0027 : involving a plurality of vehicles, e.g. fleet or convoy travelling fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00
• CPC Class Codes: G05D 1/0044 : by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0276 : using signals provided by a source external to the vehicle involving a plurality of vehicles G05D1/0287; automatically controlling vehicle speed responsive to externally generated signals B60K31/0058
• CPC Class Codes: G05D 1/0291 : Fleet control monitoring fleets in traffic control systems for road vehicles G08G1/127, G08G1/127
• CPC Class Codes: G06F 3/04842 : Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 : Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/1454 : involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G06F 3/147 : using display panels
• CPC Class Codes: G09G 2354/00 : Aspects of interface with display user
• CPC Class Codes: G09G 2360/08 : Power processing, i.e. workload management for processors involved in display operations, such as CPUs or GPUs
• CPC Class Codes: G09G 2370/16 : Use of wireless transmission of display information
• CPC Class Codes: G09G 2380/10 : Automotive applications
• CPC Class Codes: G09G 5/14 : Display of multiple viewports
• CPC Class Codes: G09G 5/363 : Graphics controllers
• CPC Class Codes: H04W 4/44 : for communication between vehicles and infrastructures, e.g. vehicle-to-cloud [V2C] or vehicle-to-home [V2H]

Abstract

Various embodiments relate generally to autonomous vehicles and associated mechanical, electrical and electronic hardware, computer software and systems, and wired and wireless network communications to provide an autonomous vehicle fleet as a service. More specifically, systems, devices, and methods are configured to initiate modification of trajectories to influence navigation of autonomous vehicles. In particular, a method may include receiving a teleoperation message via a communication link from an autonomous vehicle, detecting data from the teleoperation message specifying an event associated with the autonomous vehicle, identifying one or more courses of action to perform responsive to detecting the data specifying the event, and generating visualization data to present information associated with the event to a display of a teleoperator computing device.

Claims

1. A method comprising:

receiving a teleoperation message via a communication link from an autonomous driverless vehicle;

detecting data from the teleoperation message specifying an event associated with the autonomous driverless vehicle;

identifying one or more courses of action to perform responsive to detecting the data specifying the event;

calculating corresponding ranks for the one or more courses of action;

simulating the one or more courses of action to produce one or more simulated courses of action;

calculating corresponding simulation values for the one or more simulated courses of action;

generating visualization data to present information associated with the event and at least a subset of the one or more courses of action in association with the corresponding ranks and simulation values to a display of a teleoperator computing device;

receiving data representing selection of a selected course of action from among the one or more courses of action presented on the display; and

transmitting the selected course of action to the autonomous driverless vehicle.

2. The method of claim 1, further comprising presenting the visualization data on the display to enable a human teleoperator to determine whether to intervene in operation of a planner of the autonomous driverless vehicle.

3. The method of claim 2, wherein presenting the visualization data comprises depicting a physical environment in which the autonomous driverless vehicle is traveling.

4. The method of claim 2, wherein presenting the visualization data comprises depicting the subset of the one or more courses of action in association with the corresponding ranks in a portion of the display configured to accept a user input representing a course of action from the human teleoperator.

5. The method of claim 1, further comprising determining failure to identify a teleoperation message from the autonomous driverless vehicle during an interval of time.

6. The method of claim 5, wherein determining failure to identify the message comprises detecting absence of a heartbeat-signal as part of the teleoperation message from the autonomous driverless vehicle, where the absence of the heartbeat-signal indicates a loss of communications via the communication link.

7. The method of claim 6, further comprising receiving a data signal from the teleoperator computing device configured to establish an alternate communication link with the autonomous driverless vehicle.

8. The method of claim 1, wherein generating the visualization data comprises adjusting content for presentation based on a bandwidth of the communication link.

9. A method comprising:

receiving, at a computing system, sensor data generated from multiple sensors on an autonomous driverless vehicle;

detecting, from the sensor data, degradation of at least one sensor on the autonomous driverless vehicle;

simulating one or more actions to produce one or more simulated actions;

calculating corresponding simulation values for the one or more simulated actions;

generating, for presentation on a display to a human teleoperator, visualization data pertaining to the sensor data received from the multiple sensors and an indication that the at least one sensor is degraded and the calculated simulation values;

receiving, at the computing system, input from the human teleoperator indicating an action to take to attempt correction of the at least one sensor; and

transmitting the action to the autonomous driverless vehicle.

10. The method of claim 9, wherein the action comprises an instruction to reduce effect of the degradation.

11. The method of claim 9, wherein the visualization data to present a visual representation of the sensor degradation.

12. A method comprising:

receiving, at a computing system from an autonomous driverless vehicle, sensor data indicative of an event associated with the autonomous driverless vehicle;

identifying one or more courses of action to perform responsive to the event;

simulating the one or more courses of action to produce one or more simulated courses of action;

calculating corresponding simulation values for the one or more simulated courses of action; and

generating visualization data to present, on a display, pertaining to the one or more courses of action in association with the simulation values;

receiving data representing selection of a selected course of action from among the one or more courses of action presented on the display, the selection being based in part on the simulation values corresponding to the one or more courses of action; and

transmitting the selected course of action to the autonomous driverless vehicle.

13. The method of claim 12, further comprising:

calculating corresponding ranks for the one or more courses of action; and

generating visualization data to present at least a subset of the one or more courses of action in association with the corresponding ranks.

14. The method of claim 13, further comprising:

receiving data representing selection of a selected course of action from among the one or more courses of action presented on the display, the selection being based in part on a combination of the ranks and the simulation values corresponding to the one or more courses of action; and

transmitting the selected course of action to the autonomous driverless vehicle.

15. The method of claim 12, further comprising presenting the visualization data on the display to enable a human teleoperator to select one of the one or more courses of action.

16. A method comprising:

presenting, on a display of a computing device, a first area containing first visualization data pertaining to an environment within which an autonomous driverless vehicle is operating;

presenting, on the display of the computing device, a second area containing second visualization data pertaining to a listing of actions that could be performed by the autonomous driverless vehicle while operating in the environment, the second visualization data further pertaining to a simulation value of a corresponding action in the listing of actions indicating a result of simulating the corresponding action in a simulation computer;

receiving, at the computing device, user selection of a selected action to be performed by the autonomous driverless vehicle; and

transmitting the selected action to the autonomous driverless vehicle.

17. The method of claim 16, wherein the first and second areas are presented simultaneously on the display and the second area is smaller than the first area.

18. The method of claim 16, further comprising additionally presenting visualization data pertaining to a rank of the corresponding action in the listing of actions in the second area.