Remote assistance for an autonomous vehicle in low confidence situations

• US 10,444,754 B2
• Filed: 09/06/2016
• Issued: 10/15/2019
• Est. Priority Date: 03/03/2014
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Fairfield, Nathaniel, Herbach, Joshua Seth
• Primary Examiner(s): Anwari, Maceeh
• Application Number:
  US15/257,021
• Publication Number:
  US 20160370801A1
• Time in Patent Office: 1,134 Days
• Field of Search: 701 2
• CPC Class Codes: B60W 2050/0215 Sensor drifts or sensor failures
• CPC Class Codes: B60W 2556/20 Data confidence level
• CPC Class Codes: B60W 2556/45 External transmission of data to or from the vehicle
• CPC Class Codes: G05D 1/0027 involving a plurality of vehicles, e.g. fleet or convoy travelling fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks

Abstract

Example systems and methods enable an autonomous vehicle to request assistance from a remote operator when the vehicle'"'"'s confidence in operation is low. One example method includes operating an autonomous vehicle in a first autonomous mode. The method may also include identifying a situation where a level of confidence of an autonomous operation in the first autonomous mode is below a threshold level. The method may further include sending a request for assistance to a remote assistor, the request including sensor data representative of a portion of an environment of the autonomous vehicle. The method may additionally include receiving a response from the remote assistor, the response indicating a second autonomous mode of operation. The method may also include causing the autonomous vehicle to operate in the second autonomous mode of operation in accordance with the response from the remote assistor.

Claims

1. A computer-implemented method comprising:

receiving a request for assistance from an autonomous vehicle, the request comprising;

a sensor data representation of an environment of the autonomous vehicle, the sensor data representation comprising a representation based on sensor data collected by the autonomous vehicle while operating in autonomous mode; and

a live video stream of a portion of the environment from one or more cameras on the autonomous vehicle;

providing for display of a graphical interface, wherein the graphical interface comprises a first sub-window showing the sensor data representation of the environment and the graphical interface further comprises a second sub-window showing the live video stream of the portion of the environment, wherein the first sub-window and the second sub-window are simultaneously displayed within the graphical interface to allow for comparison of detected objects in the sensor data representation with corresponding objects in the live video stream;

receiving, via the graphical interface, input data indicating a suggested mode of autonomous operation for the autonomous vehicle; and

sending a response to the autonomous vehicle, wherein the response includes the suggested mode of autonomous operation for the autonomous vehicle.

2. The method of claim 1, wherein the sensor data representation comprises a composite representation based on sensor data collected from a plurality of different types of sensors on the autonomous vehicle.

3. The method of claim 1, wherein the sensor data representation comprises abstracted geometric shapes corresponding to one or more objects in the environment of the autonomous vehicle.

4. The method of claim 3, wherein the abstracted geometric shapes comprise three-dimensional rectangular shapes.

5. The method of claim 1, wherein the first sub-window showing the sensor data representation of the environment is displayed horizontally adjacent to the second sub-window showing the live video stream of the portion of the environment within the graphical interface.

6. The method of claim 1, wherein the sensor data representation is overlayed on top of the live video stream within the graphical interface.

7. The method of claim 1, wherein the first sub-window and the second sub-window are separate windows of approximately a same size within the graphical interface.

8. The method of claim 1, further comprising:

receiving one or more timestamps indicative of when the sensor data was collected;

based on the one or more timestamps, determining a latency amount indicative of an age of the sensor data; and

providing for display of the latency amount within the graphical interface.

9. The method of claim 1, wherein the input data further comprises a focus region that indicates a particular portion of the environment for the autonomous vehicle to watch for a change within the environment in order to proceed within the suggested mode of autonomous operation, and wherein the response to the autonomous vehicle further comprises the focus region.

10. The method of claim 1, wherein the graphical interface is presented to a human operator that is remote from the autonomous vehicle.

11. A system comprising:

a display device; and

a control system configured to;

receive a request for assistance from an autonomous vehicle, the request comprising;

a sensor data representation of an environment of the autonomous vehicle, the sensor data representation comprising a representation based on sensor data collected by the autonomous vehicle while operating in autonomous mode; and

a live video stream of a portion of the environment from one or more cameras on the autonomous vehicle;

display a graphical interface on the display device, wherein the graphical interface comprises a first sub-window showing the sensor data representation of the environment and the graphical interface further comprises a second sub-window showing the live video stream of the portion of the environment, wherein the first sub-window and the second sub-window are simultaneously displayed within the graphical interface to allow for comparison of detected objects in the sensor data representation with corresponding objects in the live video stream;

receive, via the graphical interface, input data indicating a suggested mode of autonomous operation for the autonomous vehicle; and

send a response to the autonomous vehicle, wherein the response includes the suggested mode of autonomous operation for the autonomous vehicle.

12. The system of claim 11, wherein the sensor data representation comprises a composite representation based on sensor data collected from a plurality of different types of sensors on the autonomous vehicle.

13. The system of claim 11, wherein the sensor data representation comprises abstracted geometric shapes corresponding to one or more objects in the environment of the autonomous vehicle.

14. The system of claim 11, wherein the first sub-window and the second sub window are separate windows of approximately a same size within the graphical interface, and wherein the first sub-window showing the sensor data representation of the environment is displayed horizontally adjacent to the second sub-window showing the live video stream of the portion of the environment within the graphical interface.

15. The system of claim 11, wherein the input data further comprises a focus region that indicates a particular portion of the environment for the autonomous vehicle to watch for a change within the environment in order to proceed within the suggested mode of autonomous operation, and wherein the response to the autonomous vehicle further comprises the focus region.

16. The system of claim 11, wherein the graphical interface is presented to a human operator that is remote from the autonomous vehicle.

17. A non-transitory computer readable medium having stored therein instructions, that when executed by a computing system, cause the computing system to perform functions comprising:

receiving a request for assistance from an autonomous vehicle, the request comprising;

a sensor data representation of an environment of the autonomous vehicle, the sensor data representation comprising a representation based on sensor data collected by the autonomous vehicle while operating in autonomous mode; and

a live video stream of a portion of the environment from one or more cameras on the autonomous vehicle;

providing for display of a graphical interface, wherein the graphical interface comprises a first sub-window showing the sensor data representation of the environment and the graphical interface further comprises a second sub-window showing the live video stream of the portion of the environment, wherein the first sub-window and the second sub-window are simultaneously displayed within the graphical interface to allow for comparison of detected objects in the sensor data representation with corresponding objects in the live video stream;

receiving, via the graphical interface, input data indicating a suggested mode of autonomous operation for the autonomous vehicle; and

sending a response to the autonomous vehicle, wherein the response includes the suggested mode of autonomous operation for the autonomous vehicle.

18. The non-transitory computer readable medium of claim 17, wherein the sensor data representation comprises a composite representation based on sensor data collected from a plurality of different types of sensors on the autonomous vehicle.

19. The non-transitory computer readable medium of claim 17, wherein the sensor data representation comprises abstracted geometric shapes corresponding to one or more objects in the environment of the autonomous vehicle.

20. The non-transitory computer readable medium of claim 17, wherein the first sub-window and the second sub-window are separate windows of approximately a same size within the graphical interface, and wherein the first sub-window showing the sensor data representation of the environment is displayed horizontally adjacent to the second sub-window showing the live video stream of the portion of the environment within the graphical interface.