Remote Assistance for Autonomous Vehicles in Predetermined Situations

• US 20150248131A1
• Filed: 03/03/2014
• Published: 09/03/2015
• Est. Priority Date: 03/03/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Fairfield, Nathaniel, Herbach, Joshua Seth, Furman, Vadim
• Granted Patent: US 9,720,410 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B60W 2540/049 Number of occupants
• CPC Class Codes: B60W 2556/45 External transmission of data to or from the vehicle
• CPC Class Codes: B60W 30/00 Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
• CPC Class Codes: B60W 60/001 Planning or execution of driving tasks
• CPC Class Codes: B60W 60/00253 Taxi operations
• CPC Class Codes: G05D 1/0011 associated with a remote control arrangement
• CPC Class Codes: G05D 1/0027 involving a plurality of vehicles, e.g. fleet or convoy travelling fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00
• CPC Class Codes: G05D 1/0033 by having the operator tracking the vehicle either by direct line of sight or via one or more cameras located remotely from the vehicle
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005

Abstract

Example systems and methods enable an autonomous vehicle to request assistance from a remote operator in certain predetermined situations. One example method includes determining a representation of an environment of an autonomous vehicle based on sensor data of the environment. Based on the representation, the method may also include identifying a situation from a predetermined set of situations for which the autonomous vehicle will request remote assistance. The method may further include sending a request for assistance to a remote assistor, the request including the representation of the environment and the identified situation. The method may additionally include receiving a response from the remote assistor indicating an autonomous operation. The method may also include causing the autonomous vehicle to perform the autonomous operation.

Claims

1. A computer-implemented method comprising:

determining a representation of an environment of an autonomous vehicle based on sensor data of the environment;

based on the representation, identifying a situation from a predetermined set of situations for which the autonomous vehicle will request remote assistance;

sending a request for assistance to a remote assistor, the request comprising the representation of the environment and the identified situation;

receiving a response from the remote assistor indicating an autonomous operation; and

causing the autonomous vehicle to perform the autonomous operation.

2. The method of claim 1, wherein sending the request for assistance comprises sending a video stream of a portion of the environment to the remote assistor.

3. The method of claim 1, wherein sending the request for assistance comprises sending a plurality of suggested operations to the remote assistor.

4. The method of claim 1, wherein sending the request for assistance comprises requesting assistance from a passenger of the autonomous vehicle.

5. The method of claim 1, wherein sending the request for assistance comprises sending one or more timestamps indicative of when the sensor data was captured to the remote assistor.

6. The method of claim 1, further comprising causing the autonomous vehicle to hold position until the response from the remote assistor is received.

7. The method of claim 1, wherein:

the identified situation comprises an intersection with a green light and no green arrow;

the identified situation comprises a lane which allows for oncoming traffic in an opposite direction; and

the autonomous operation comprises a left turn.

8. A computer-implemented method comprising:

receiving a request for assistance from an autonomous vehicle, the request comprising a representation of an environment of the autonomous vehicle and an identified situation from a predetermined set of situations;

receiving sensor data representative of the environment of the autonomous vehicle;

based on the representation of the environment and the sensor data, determining an autonomous operation for the autonomous vehicle to perform in the identified situation; and

sending a response to the request for assistance to the autonomous vehicle to perform the autonomous operation in the identified situation.

9. The method of claim 8, wherein receiving the sensor data representative of the environment comprises receiving a video stream of a portion of the environment.

10. The method of claim 8, wherein determining the autonomous operation comprises receiving a selection of the autonomous operation from a human operator.

11. The method of claim 8, wherein determining the autonomous operation comprises:

presenting a plurality of suggested operations to a human operator; and

receiving a selection of the autonomous operation from the human operator.

12. The method of claim 8, wherein determining the autonomous operation comprises:

receiving one or more timestamps indicative of when the sensor data was captured;

based on the one or more timestamps, determining one or more latency amounts indicative of an age of the sensor data; and

using sensor data which has a latency amount below a certain amount to determine the autonomous operation.

13. The method of claim 8, wherein:

the identified situation comprises an intersection with a red light and a lane which allows for cross traffic; and

the autonomous operation comprises a right turn.

14. An autonomous vehicle, comprising:

a control system configured to;

determine a representation of an environment of the autonomous vehicle based on sensor data of the environment;

based on the representation, identify a situation from a predetermined set of situations for which the autonomous vehicle will request remote assistance;

send a request for assistance to a remote assistor, the request comprising the representation of the environment and the identified situation;

receive a response from the remote assistor indicating an autonomous operation; and

cause the autonomous vehicle to perform the autonomous operation.

15. The vehicle of claim 14, wherein the control system is configured to send the request for assistance by sending a video stream of a portion of the environment to the remote assistor.

16. The vehicle of claim 14, wherein the control system is configured to send the request for assistance by sending a plurality of suggested operations to the remote assistor.

17. The vehicle of claim 14, wherein the control system is configured to send the request for assistance by requesting assistance from a passenger of the autonomous vehicle.

18. The vehicle of claim 14, wherein the control system is configured to send the request for assistance by sending one or more timestamps indicating when the sensor data was captured to the remote assistor.

19. The vehicle of claim 14, wherein the control system is further configured to cause the autonomous vehicle to hold position until the response from the remote assistor is received.

20. The vehicle of claim 14, wherein:

the identified situation comprises a detected stop sign that is not present on a map of the environment used by the vehicle for navigation; and

the autonomous operation comprises stopping the vehicle.