Remote assistance for autonomous vehicles in predetermined situations

• US 9,720,410 B2
• Filed: 03/03/2014
• Issued: 08/01/2017
• Est. Priority Date: 03/03/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Furman, Vadim, Herbach, Joshua Seth, Fairfield, Nathaniel
• Primary Examiner(s): KLEINMAN, LAIL A
• Application Number:
  US14/195,682
• Publication Number:
  US 20150248131A1
• Time in Patent Office: 1,247 Days
• Field of Search: None
• CPC Class Codes: B60W 2540/049 Number of occupants
• CPC Class Codes: B60W 2556/45 External transmission of data to or from the vehicle
• CPC Class Codes: B60W 30/00 Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
• CPC Class Codes: B60W 60/001 Planning or execution of driving tasks
• CPC Class Codes: B60W 60/00253 Taxi operations
• CPC Class Codes: G05D 1/0011 associated with a remote control arrangement
• CPC Class Codes: G05D 1/0027 involving a plurality of vehicles, e.g. fleet or convoy travelling fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00
• CPC Class Codes: G05D 1/0033 by having the operator tracking the vehicle either by direct line of sight or via one or more cameras located remotely from the vehicle
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005

Abstract

Example systems and methods enable an autonomous vehicle to request assistance from a remote operator in certain predetermined situations. One example method includes determining a representation of an environment of an autonomous vehicle based on sensor data of the environment. Based on the representation, the method may also include identifying a situation from a predetermined set of situations for which the autonomous vehicle will request remote assistance. The method may further include sending a request for assistance to a remote assistor, the request including the representation of the environment and the identified situation. The method may additionally include receiving a response from the remote assistor indicating an autonomous operation. The method may also include causing the autonomous vehicle to perform the autonomous operation.

Claims

1. A computer-implemented method comprising:

determining a representation of an environment of an autonomous vehicle based on sensor data of the environment;

based on the representation, identifying that the autonomous vehicle is in a given situation from a predetermined set of situations for which the autonomous vehicle will request remote assistance;

sending a request for verification of an autonomous operation to take in the given situation to a remote computing device that comprises a user interface to display the representation of the environment and an indication of the given situation;

receiving a response from the remote computing device indicating verification of the autonomous operation; and

after receiving the response, causing the autonomous vehicle to perform the autonomous operation.

2. The method of claim 1, wherein sending the request comprises sending a video stream of a portion of the environment for display by the remote computing device.

3. The method of claim 1, wherein the user interface of the remote computing device is displayed to a passenger of the autonomous vehicle.

4. The method of claim 1, wherein sending the request comprises sending one or more timestamps indicative of when the sensor data was captured to the remote computing device.

5. The method of claim 1, further comprising causing the autonomous vehicle to hold position until the response indicating verification of the autonomous operation from the remote computing device is received.

6. The method of claim 1, wherein:

the identified situation comprises an intersection with a green light and no green arrow;

the identified situation comprises a lane which allows for oncoming traffic in an opposite direction; and

the autonomous operation comprises a left turn at the intersection.

16. The method of claim 1, wherein the predetermined set of situations comprises one or more situations in which the autonomous vehicle is stopped at a traffic intersection.

17. The method of claim 1, wherein the predetermined set of situations comprises at least unprotected left turns, wherein an unprotected left turn comprises a situation in which the autonomous vehicle intends to make a left turn at an intersection with a green light and no green arrow.

18. The method of claim 1, wherein the predetermined set of situations comprises at least a situation in which the autonomous vehicle intends to make a right turn at an intersection in which the autonomous vehicle is facing a red light.

7. A computer-implemented method comprising:

receiving a request for verification of an autonomous operation from an autonomous vehicle, the request comprising a representation of an environment of the autonomous vehicle and an identified situation from a predetermined set of situations;

receiving sensor data representative of the environment of the autonomous vehicle;

providing for display of a user interface that shows the representation of the environment, at least a portion of the sensor data, and an indication of the identified situation;

receiving, via the user interface, verification of the autonomous operation for the autonomous vehicle to perform in the identified situation; and

after receiving verification of the autonomous operation, sending a response to the request to the autonomous vehicle to perform the autonomous operation in the identified situation.

8. The method of claim 7, wherein receiving the sensor data representative of the environment comprises receiving a video stream of a portion of the environment.

9. The method of claim 7, wherein determining the autonomous operation comprises:

receiving one or more timestamps indicative of when the sensor data was captured;

based on the one or more timestamps, determining one or more latency amounts indicative of an age of the sensor data; and

providing for display of the one or more latency amounts within the user interface.

10. The method of claim 7, wherein:

the identified situation comprises an intersection with a red light and a lane which allows for cross traffic; and

the autonomous operation comprises a right turn.

11. An autonomous vehicle, comprising:

a control system configured to;

determine a representation of an environment of the autonomous vehicle based on sensor data of the environment;

based on the representation, identify that the autonomous vehicle is in a given situation from a predetermined set of situations for which the autonomous vehicle will request remote assistance;

send a request for verification of an autonomous operation to take in the given situation to a remote computing device that comprises a user interface to display the representation of the environment and an indication of the given situation;

receive a response from the remote computing device indicating verification of the autonomous operation; and

after receiving the response, cause the autonomous vehicle to perform the autonomous operation.

12. The vehicle of claim 11, wherein the control system is further configured to send a video stream of a portion of the environment to the remote computing device.

13. The vehicle of claim 11, wherein the user interface of the remote computing device is displayed to a passenger of the autonomous vehicle.

14. The vehicle of claim 11, wherein the control system is configured to send the request by sending one or more timestamps indicating when the sensor data was captured to the remote computing device.

15. The vehicle of claim 11, wherein the control system is further configured to cause the autonomous vehicle to hold position until the response indicating verification of the autonomous operation from the remote computing device is received.