Methods and systems for keeping remote assistance operators alert

• US 10,761,542 B1
• Filed: 08/22/2017
• Issued: 09/01/2020
• Est. Priority Date: 07/11/2017
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Fairfield, Nathaniel, Herbach, Joshua
• Primary Examiner(s): Wiltey, Nicholas K
• Application Number:
  US15/683,386
• Time in Patent Office: 1,106 Days
• Field of Search: None
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2556/65 Data transmitted between vehicles
• CPC Class Codes: B60W 2900/00 Indexing codes relating to the purpose of, or problem solved of road vehicle drive control systems not otherwise provided for in groups B60W30/00
• CPC Class Codes: B60W 30/182 Selecting between different operative modes, e.g. comfort and performance modes
• CPC Class Codes: B60W 60/001 Planning or execution of driving tasks
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0022 characterised by the communication link data switching networks in general H04L12/00
• CPC Class Codes: G05D 1/0027 involving a plurality of vehicles, e.g. fleet or convoy travelling fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0282 generated in a local control room
• CPC Class Codes: G05D 1/0285 using signals transmitted via a public communication network, e.g. GSM network
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04W 4/44 for communication between vehicles and infrastructures, e.g. vehicle-to-cloud [V2C] or vehicle-to-home [V2H]

Abstract

Examples described may enable provision of remote assistance for an autonomous vehicle. An example method includes a computing system operating by default in a first mode and periodically transitioning from operation in the first mode to operation in a second mode. In the first mode, the system may receive environment data provided by the vehicle and representing object(s) having a detection confidence below a threshold, where the detection confidence is indicative of a likelihood of correct identification of the object(s), and responsive to the object(s) having a confidence below the threshold, provide remote assistance data comprising an instruction to control the vehicle and/or a correct identification of the object(s). In the second mode, the system may trigger user interface display of remote assistor alertness data based on pre-stored data related to an environment in which the pre-stored data was acquired, and receive a response relating to the alertness data.

Claims

1. A method of providing remote assistance for an autonomous vehicle, the method comprising:

operating, by a computing system, in one of two modes including a first mode and a second mode,

wherein operating in one of two modes comprises operation by default in the first mode and predefined periodic transitioning from operation in the first mode to operation in the second mode,

wherein, in the first mode, the computing system (i) receives environment data provided by an autonomous vehicle operating in an environment, wherein the environment data represents at least one object of the environment having a detection confidence below a threshold, wherein the detection confidence is indicative of a likelihood that the at least one object is correctly identified in the environment, and (ii) in response to the at least one object having a detection confidence below the threshold, provides remote assistance data comprising one or more of (a) an instruction to control the autonomous vehicle or (b) a correct identification of the at least one object, and

wherein, in the second mode, the computing system (i) triggers user interface display of remote assistor alertness data based on a set of pre-stored data related to an environment in which the pre-stored data was acquired, and (ii) receives a response relating to the remote assistor alertness data.

2. The method of claim 1, wherein the predefined periodic transitioning from the operation in the first mode to the operation in the second mode comprises periodically:

determining that a period of time has expired since last providing remote assistance data; and

in response to determining that the period of time has expired, transitioning from the operation in the first mode to the operation in the second mode.

3. The method of claim 2, wherein the period of time is the same for every transition from the operation in the first mode to the operation in the second mode.

4. The method of claim 2, wherein each period of time is a random value in a range from 1 minute to 10 minutes.

5. The method of claim 1, wherein the remote assistor alertness data is associated with at least one object having a detection confidence below the threshold.

6. The method of claim 1, wherein the remote assistor alertness data is associated with at least one portion of the environment data for which no remote assistance data was provided.

7. The method of claim 1, wherein the remote assistor alertness data is associated with at least one object that was previously and correctly identified.

8. The method of claim 1, wherein the remote assistor alertness data includes at least one natural-language question.

9. The method of claim 1, wherein the set of pre-stored data comprises a video.

10. The method of claim 1, wherein the set of pre-stored data comprises an audio signal.

11. The method of claim 1, wherein the computing system is selected from the group consisting of a remote operator computing system and a server computing system communicatively linked to the remote operator computing system.

12. A computing system comprising:

a processor,

wherein the processor is configured for performing operations comprising;

operating in one of two modes including a first mode and a second mode,

wherein operating in one of two modes comprises operation by default in the first mode and predefined periodic transitioning from operation in the first mode to operation in the second mode,

wherein, in the first mode, the computing system (i) receives environment data provided by an autonomous vehicle operating in an environment, wherein the environment data represents at least one object of the environment having a detection confidence below a threshold, wherein the detection confidence is indicative of a likelihood that the at least one object is correctly identified in the environment, and (ii) in response to the at least one object having a detection confidence below the threshold, provides remote assistance data comprising one or more of (a) an instruction to control the autonomous vehicle or (b) a correct identification of the at least one object, and

wherein, in the second mode, the computing system (i) triggers user interface display of remote assistor alertness data based on a set of pre-stored data related to an environment in which the pre-stored data was acquired, and (ii) receives a response relating to the remote assistor alertness data.

13. The computing system of claim 12, wherein the predefined periodic transitioning from the operation in the first mode to the operation in the second mode comprises periodically:

determining that a period of time has expired since last providing remote assistance data; and

in response to determining that the period of time has expired, transitioning from the operation in the first mode to the operation in the second mode.

14. The computing system of claim 13, wherein the period of time is the same for every transition from the operation in the first mode to the operation in the second mode.

15. The computing system of claim 13, wherein each period of time is a random value in a range from 1 minute to 10 minutes.

16. The computing system of claim 12, wherein the computing system is a server computing system,

wherein triggering user interface display of remote assistor alertness data comprises transmitting, to a remote operator computing device, instructions for the remote operator computing device to display, via a user interface of the remote operator computing device, the remote assistor alertness data, and

wherein receiving the response relating to the remote assistor alertness data comprises receiving the response from the remote operator computing device.

17. The computing system of claim 12, wherein triggering user interface display of the remote assistor alertness data comprises triggering user interface display of the pre-stored data with the remote assistor alertness data.

18. An article of manufacture including a non-transitory computer-readable medium having stored thereon instructions that, when executed by a processor in a computing system, causes the computing system to perform operations comprising:

operating in one of two modes including a first mode and a second mode,

wherein operating in one of two modes comprises operation by default in the first mode and predefined periodic transitioning from operation in the first mode to operation in the second mode,

wherein, in the first mode, the computing system (i) receives environment data provided by an autonomous vehicle operating in an environment, wherein the environment data represents at least one object of the environment having a detection confidence below a threshold, wherein the detection confidence is indicative of a likelihood that the at least one object is correctly identified in the environment, and (ii) in response to the at least one object having a detection confidence below the threshold, provides remote assistance data comprising one or more of (a) an instruction to control the autonomous vehicle or (b) a correct identification of the at least one object, and

wherein, in the second mode, the computing system (i) triggers user interface display of remote assistor alertness data based on a set of pre-stored data related to an environment in which the pre-stored data was acquired, and (ii) receives a response relating to the remote assistor alertness data.

19. The article of manufacture of claim 18, wherein the remote assistor alertness data includes at least one natural-language question that requests confirmation of at least one object that was previously and correctly identified.

20. The article of manufacture of claim 18, wherein the predefined periodic transitioning from the operation in the first mode to the operation in the second mode comprises periodically:

determining that a period of time has expired since last providing remote assistance data; and

in response to determining that the period of time has expired, transitioning from the operation in the first mode to the operation in the second mode.