Predictive teleoperator situational awareness

• US 10,268,191 B1
• Filed: 07/07/2017
• Issued: 04/23/2019
• Est. Priority Date: 07/07/2017
• Status: Active Grant

• Current Assignee: Zoox Inc. (Amazon.com, Inc.)
• Original Assignee: Zoox Inc. (Amazon.com, Inc.)
• Inventors: Lockwood, Amanda Lee Kelly, Gogna, Ravi, Linscott, Gary, Orecchio, Paul, Xie, Dan, Rege, Ashutosh Gajanan, Levinson, Jesse Sol
• Primary Examiner(s): Marc-Coleman, Marthe Y
• Application Number:
  US15/644,349
• Time in Patent Office: 655 Days
• Field of Search: 701 2, 701 1
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0022 characterised by the communication link data switching networks in general H04L12/00
• CPC Class Codes: G05D 1/0027 involving a plurality of vehicles, e.g. fleet or convoy travelling fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005

Abstract

A teleoperator device may be configured to obtain a request for teleoperator assistance from a driverless vehicle and obtain teleoperator data in response to the request. The teleoperator device may also be configured to record at least some of the teleoperator input and/or guidance transmitted to the driverless vehicle based on the teleoperator input. Upon receiving a subsequent request, the teleoperator device may be configured to reproduce at least part of the former teleoperator input and/or to provide an option to activate guidance associated with the teleoperator input. The teleoperator device may also be configured to train a model and/or use a model to determine from vehicle data an option for presentation via a teleoperator interface and/or a presentation configuration of the teleoperator interface.

Claims

1. A teleoperation system comprising:

at least one processor; and

at least one memory having stored thereon processor-executable instructions that, when executed by the system, configure the teleoperation system to;

obtain via a network interface a first request for teleoperator assistance from a first driverless vehicle and at least one of first operation state data or first sensor data from the first driverless vehicle;

obtain teleoperator input via a user interface;

transmit a first guidance signal to the first driverless vehicle via the network interface based at least in part on the teleoperator input;

store as historical teleoperation data at least part of the teleoperator input and a first descriptor, the first descriptor determined based at least part on at least one of the first request, the first operation state data, or the first sensor data;

obtain via the network interface a second request for teleoperator assistance from a second driverless vehicle and at least one of second operation state data or second sensor data;

determine a second descriptor based at least in part on the second request, the second operation state data, or the second sensor data;

determine a distance between the first descriptor and the second descriptor in a descriptor space; and

perform an action based at least in part on the distance.

2. The teleoperation system of claim 1, wherein the instructions further configure the teleoperation system to:

obtain an affirmation via the user interface of the action; and

transmit a second guidance signal to the second driverless vehicle based on the affirmation and the teleoperation input.

3. The teleoperation system of claim 1, wherein the instructions further configure the teleoperation system to:

cause presentation via the user interface, based at least in part on obtaining the first request, at least one of the first request, the first operation state data, or the first sensor data according to a presentation; and

associate and store the presentation configuration with the historical teleoperation data.

4. The teleoperation system of claim 3, wherein the instructions further configure the teleoperation system to:

determine that at least one of the second request, the second operation state data, or the second sensor data corresponds to at least a portion of the historical teleoperation data, based at least in part on obtaining the second request; and

cause presentation via the user interface of at least one of the second request, the second operation state data, or the second sensor data according to the presentation configuration of the historical teleoperation data.

5. The teleoperation system of claim 1, wherein the action comprises outputting one or more options for presentation by a teleoperator display, the options comprising at least one of a first option to accept a proposed guidance, a second option to reject the proposed guidance, or a third option to modify the proposed guidance.

6. The teleoperation system of claim 1, wherein the instructions further configure the teleoperation system to:

obtain via the network interface an indication that the first driverless vehicle received the first guidance signal and successfully operated in view of an event that caused the first driverless vehicle to transmit the first request; and

associate and store the indication with the historical teleoperation data corresponding to the event.

7. The teleoperation system of claim 6, wherein the instructions further configure the teleoperation system to learn a model from at least one of the indication, at least a portion of the historical teleoperation data, or the event, wherein the model is configured to:

receive at least one of a request for assistance, operation state data, teleoperation data or sensor data; and

output at least one of instructions for a presentation configuration of the user interface or instructions for presenting an option to perform an action.

8. The teleoperation system of claim 1, wherein determining the second at least one of the tensor or the string comprises populating at least one of a tensor or a vector with one or more values associated with the second request, the second operation state data, or the second sensor data, wherein the tensor or the vector is configured as input for a machine-learned model and the one or more values comprise a representation of at least one of the second request, the second operation state data, or the second sensor data that is formatted as input for the machine-learned model.

9. A non-transitory computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by a computer, cause the computer to:

cause presentation of vehicle data obtained responsive to a request for assistance from a driverless vehicle via a user interface;

capture one or more teleoperator interactions via the user interface;

transmit a guidance signal to the driverless vehicle based at least in part on the one or more teleoperator interactions;

determine, based at least in part on the vehicle data, a descriptor, the descriptor comprising at least one of a tensor or a vector; and

train a model based on at least one of the descriptor, the presentation, the one or more teleoperator interactions, or the guidance signal, the model configured to;

receive the descriptor as input; and

determine, based at least in part on the input, an output of at least one of;

an option for presentation via the user interface, or

a presentation configuration of the presentation.

10. The non-transitory computer-readable storage medium of claim 9, wherein the option, upon selection by a teleoperator, causes performance of an action.

11. The non-transitory computer-readable storage medium of claim 9, wherein the presentation includes at least one of the request, operation state data, or sensor data from the driverless vehicle and is configured according to a presentation configuration determined by the model.

12. The non-transitory computer-readable storage medium of claim 9, wherein the instructions further configure the teleoperation device to obtain an outcome from the driverless vehicle, the outcome indicating a result of reliance on the guidance signal by the driverless vehicle.

13. The non-transitory computer-readable storage medium of claim 12, wherein training the model is further based, at least in part, on the outcome.

14. The non-transitory computer-readable storage medium of claim 9, the instructions further configuring the teleoperation device to:

obtain second vehicle data corresponding to a second request for assistance;

determine at least one of an output option or an output presentation configuration of the model based at least in part on the model and at least one of the vehicle data or the second request; and

cause presentation via the user interface of at least one of at least a portion of the vehicle data according to the output option or the output presentation configuration.

15. The non-transitory computer-readable storage medium of claim 14, wherein the option includes causing presentation of an option to accept, reject, or modify a teleoperator action, selection of the acceptance of the teleoperation action causing the system to perform the teleoperation action.

16. The non-transitory computer-readable storage medium of claim 9, wherein the output presentation configuration includes causing presentation of a replay of vehicle data.

17. The non-transitory computer-readable storage medium of claim 9, wherein the option includes at least one of:

a command for the driverless vehicle to perform an operation;

a collaborative instruction, wherein the collaborative instruction includes at least one of;

classifying at least one of an object or a scene or

revising operation parameters of the driverless vehicle and

is useable by a processor of the driverless vehicle to determine an operation to perform;

an option to revise operation parameters of the driverless vehicle;

or

an option to confirm or deny a decision determined by the driverless vehicle.

18. A computer-implemented method comprising:

obtaining vehicle data from a driverless vehicle;

determining a descriptor based at least in part on a subset of the vehicle data, wherein the descriptor comprises a representation of the subset formatted such that the descriptor may be provided as input to a machine-learned model and the subset comprises at least one of sensor data, a detected object, event data, vehicle state data, or environmental data;

evaluating the machine-learned model using the descriptor to identify at least one of a teleoperator option or a presentation configuration that corresponds to the at least one of the tensor or the string via the machine-learned model; and

causing at least one of presentation of the teleoperator option via a user interface or configuration of the user interface according to the presentation configuration.

19. The computer-implemented method of claim 18, wherein the machine-learned model includes a neural network trained to receive a representation of vehicle data as an input and output at least one of the teleoperator option or the presentation configuration.

20. The computer-implemented method of claim 18, further comprising:

obtaining at least one of a teleoperator input that differs from the teleoperator option, a modification of a presentation at a user interface, a guidance signal, or an outcome from the driverless vehicle; and

modifying the machine-learned model based on at least one of the teleoperator input, the modification, the guidance signal, or the outcome from the driverless vehicle,

wherein the optimizing comprises an objective function that at least one of;

increases a likelihood that a teleoperator relies on the teleoperator option or the presentation configuration to provide guidance to the driverless vehicle, or

decreases a time between receiving a request and transmitting the guidance signal to the driverless vehicle.