Remote assistance for an autonomous vehicle in low confidence situations

• US 9,465,388 B1
• Filed: 03/03/2014
• Issued: 10/11/2016
• Est. Priority Date: 03/03/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Fairfield, Nathaniel, Herbach, Joshua Seth
• Primary Examiner(s): Anwari, Maceeh
• Application Number:
  US14/195,663
• Time in Patent Office: 953 Days
• Field of Search: 701/2
• US Class Current: 1/1
• CPC Class Codes: B60W 2050/0215 Sensor drifts or sensor failures
• CPC Class Codes: B60W 2556/20 Data confidence level
• CPC Class Codes: B60W 2556/45 External transmission of data to or from the vehicle
• CPC Class Codes: G05D 1/0027 involving a plurality of vehicles, e.g. fleet or convoy travelling fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks

Abstract

Example systems and methods enable an autonomous vehicle to request assistance from a remote operator when the vehicle'"'"'s confidence in operation is low. One example method includes operating an autonomous vehicle in a first autonomous mode. The method may also include identifying a situation where a level of confidence of an autonomous operation in the first autonomous mode is below a threshold level. The method may further include sending a request for assistance to a remote assistor, the request including sensor data representative of a portion of an environment of the autonomous vehicle. The method may additionally include receiving a response from the remote assistor, the response indicating a second autonomous mode of operation. The method may also include causing the autonomous vehicle to operate in the second autonomous mode of operation in accordance with the response from the remote assistor.

Claims

1. A computer-implemented method comprising:

operating an autonomous vehicle in a first autonomous mode;

identifying, by a control system of the autonomous vehicle, a situation where a level of confidence of an autonomous operation in the first autonomous mode is below a threshold level;

sending, by the control system of the autonomous vehicle, a request for assistance to a remote assistor, the request comprising sensor data representative of a portion of an environment of the autonomous vehicle, the request further comprising a plurality of proposed autonomous modes of operation for the autonomous vehicle to proceed in the identified situation;

receiving, by the control system of the autonomous vehicle, a response from the remote assistor, the response indicating a second autonomous mode of operation, wherein the second autonomous mode of operation is selected from the plurality of proposed autonomous modes of operation for the autonomous vehicle to proceed in the identified situation; and

causing the autonomous vehicle to operate in the second autonomous mode of operation in accordance with the response from the remote assistor.

2. The method of claim 1, wherein identifying the situation where the level of confidence of the autonomous operation in the first autonomous mode is below the threshold level comprises determining when an amount of uncertainty associated with a position of the autonomous vehicle is above a threshold amount.

3. The method of claim 1, wherein identifying the situation where the level of confidence of the autonomous operation in the first autonomous mode is below the threshold level comprises determining when the autonomous vehicle is unable to move for a certain amount of time.

4. The method of claim 1, wherein identifying the situation where the level of confidence of the autonomous operation in the first autonomous mode is below the threshold level comprises determining when an amount of uncertainty associated with identification of one or more objects within the environment is above a threshold amount.

5. The method of claim 1, wherein identifying the situation where the level of confidence of the autonomous operation in the first autonomous mode is below the threshold level comprises receiving a warning signal from one or more sensor systems of the autonomous vehicle.

6. The method of claim 1, wherein the sensor data comprises:

a sensor data representation of the portion of the environment of the autonomous vehicle, the sensor data representation comprising a representation based on sensor data collected by the vehicle and used by the vehicle to determine the first mode of operation; and

a live video stream of the portion of the environment from one or more cameras on the autonomous vehicle.

7. The method of claim 1, wherein sending the request for assistance comprises requesting assistance from a passenger of the autonomous vehicle.

8. The method of claim 1, wherein sending the request for assistance comprises sending one or more timestamps indicative of when the sensor data was collected to the remote assistor.

9. The method of claim 1, wherein:

operating the autonomous vehicle in the first autonomous mode comprises operating the autonomous vehicle with a first level of caution, wherein the first level of caution indicates an amount of certainty in an autonomous operation required by the vehicle to execute the autonomous operation; and

operating the autonomous vehicle in the second autonomous mode comprises operating the autonomous vehicle with a second level of caution, wherein the second level of caution indicates a different amount of certainty in an autonomous operation required by the vehicle to execute the autonomous operation.

10. The method of claim 1, wherein:

operating the autonomous vehicle in the first autonomous mode comprises operating the autonomous vehicle within a first range of speeds; and

operating the autonomous vehicle in the second autonomous mode comprises operating the autonomous vehicle within a second range of speeds.

11. A computer-implemented method comprising:

receiving a request for assistance from an autonomous vehicle, the request comprising;

a sensor data representation of an environment of the autonomous vehicle, the sensor data representation comprising a representation based on sensor data collected by the vehicle;

a live video stream of a portion of the environment from one or more cameras on the autonomous vehicle; and

a plurality of proposed autonomous modes of operation for the autonomous vehicle;

providing for display of a graphical interface, wherein the graphical interface comprises;

a first sub-window showing the sensor data representation of the environment;

a second sub-window showing the live video stream of the portion of the environment; and

a list of the plurality of proposed autonomous modes of operation;

receiving a selected proposed mode of autonomous operation for the autonomous vehicle from the list via the graphical interface; and

sending a response to the autonomous vehicle indicating to switch to the selected proposed mode of autonomous operation.

12. The method of claim 11, wherein the graphical interface is presented to a human operator.

13. The method of claim 11, wherein the sensor data representation is overlayed on top of the live video stream within the graphical interface.

14. The method of claim 11, further comprising:

receiving one or more timestamps indicative of when the sensor data was collected;

based on the one or more timestamps, determining a latency amount indicative of an age of the sensor data; and

providing for display of the latency amount within the graphical interface.

15. The method of claim 11, wherein the response comprises an area of focus, wherein the area of focus comprises a particular portion of the environment for the autonomous vehicle to watch for a change within the environment in order to proceed within the selected proposed mode of autonomous operation.

16. An autonomous vehicle, comprising:

a control system configured to;

operate the autonomous vehicle in a first autonomous mode;

identify a situation where a level of confidence of an autonomous operation in the first autonomous mode is below a threshold level;

send a request for assistance to a remote assistor, the request comprising sensor data representative of a portion of an environment of the autonomous vehicle, the request further comprising a plurality of proposed autonomous modes of operation for the autonomous vehicle to proceed in the identified situation;

receive a response from the remote assistor, the response indicating a second autonomous mode of operation, wherein the second autonomous mode of operation is selected from the plurality of proposed autonomous modes of operation for the autonomous vehicle to proceed in the identified situation; and

cause the autonomous vehicle to operate in the second autonomous mode of operation in accordance with the response from the remote assistor.

17. The vehicle of claim 16, wherein the control system is configured to identify the situation where the level of confidence of the autonomous operation in the first autonomous mode is below the threshold level by determining when an amount of uncertainty associated with a position of the autonomous vehicle is above a threshold amount.

18. The vehicle of claim 16, wherein the control system is configured to identify the situation where the level of confidence of the autonomous operation in the first autonomous mode is below the threshold level by determining when the autonomous vehicle is unable to move for a certain amount of time.

19. The vehicle of claim 16, wherein the control system is configured to identify the situation where the level of confidence of the autonomous operation in the first autonomous mode is below the threshold level by receiving a warning signal from one or more sensor systems of the autonomous vehicle.