Assisted perception for autonomous vehicles

• US 9,864,378 B1
• Filed: 05/12/2017
• Issued: 01/09/2018
• Est. Priority Date: 11/27/2013
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Ferguson, David Ian Franklin, Lo, Wan-Yen, Fairfield, Nathaniel
• Primary Examiner(s): Oh, Harry
• Application Number:
  US15/594,191
• Time in Patent Office: 242 Days
• Field of Search: 701 28
• CPC Class Codes: B60W 2050/0075 Automatic parameter input, automatic initialising or calibrating means
• CPC Class Codes: B60W 2050/0215 Sensor drifts or sensor failures
• CPC Class Codes: B60W 2556/45 External transmission of data to or from the vehicle
• CPC Class Codes: B60W 40/04 Traffic conditions
• CPC Class Codes: G05D 1/00 Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots drive control systems specially adapted for autonomous road vehicles B60W60/00
• CPC Class Codes: G05D 1/0022 characterised by the communication link data switching networks in general H04L12/00
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G07C 5/008 communicating information to a remotely located station transmission systems for measured values G08C
• CPC Class Codes: H04L 67/125 involving control of end-device applications over a network

Abstract

Disclosed herein are systems and methods for providing supplemental identification abilities to an autonomous vehicle system. The sensor unit of the vehicle may be configured to receive data indicating an environment of the vehicle, while the control system may be configured to operate the vehicle. The vehicle may also include a processing unit configured to analyze the data indicating the environment to determine at least one object having a detection confidence below a threshold. Based on the at least one object having a detection confidence below a threshold, the processor may communicate at least a subset of the data indicating the environment for further processing. The vehicle is also configured to receive an indication of an object confirmation of the subset of the data. Based on the object confirmation of the subset of the data, the processor may alter the control of the vehicle by the control system.

Claims

1. An apparatus comprising:

a communication system configured to communicate with a remote computer system, wherein the communication system is configured to;

transmit image data of an environment of the apparatus;

transmit a preliminary object identification of at least one object of an environment of the apparatus that has a detection confidence below a threshold;

receive an object confirmation indicating an object identification that is different than the preliminary object identification;

a control system, configured to operate the apparatus in an autonomous operation mode based on instructions to control the apparatus; and

one or more processors configured to;

determine, based on data from one or more sensors, the preliminary object identification of at least one object of the environment that has a detection confidence below a threshold, wherein the detection confidence indicates a likelihood that the determined object is present in the environment;

provide instructions to the control system based on the preliminary object identification; and

provide instructions to the control system based on an altered instruction when an object confirmation indicating an object identification that is different than the preliminary object identification is received.

2. The apparatus according to claim 1, wherein the remote computer system is configured to receive an input from a human operator.

3. The apparatus according to claim 1, wherein the one or more processors are further configured to:

alter instructions to control the apparatus by the control system based on the preliminary object identification;

operate the apparatus based on the altered instructions when an object confirmation indicating an object identification confirming the preliminary object identification is received from the remote computer system; and

operate the apparatus based on the altered instructions that are further altered when an object confirmation indicating an object identification that is different than the preliminary object identification is received from the remote computer system.

4. The apparatus according to claim 1, wherein the one or more sensors comprises a camera and the transmitted image data comprises an image.

5. The apparatus according to claim 1, wherein the one or more sensors comprises a microphone and the transmitted image data comprises a video.

6. The apparatus according to claim 1, wherein the preliminary object identification is based on one of radar and laser data.

7. The apparatus according to claim 1, wherein the one or more processors are further configured to:

provide a bounding box around the object having the preliminary object identification; and

cause the transmitted image data to include the bounding box.

8. The apparatus of claim 1, wherein the detection confidence is based on a predetermined time threshold.

9. A method comprising:

determining, by a processor, based data from one or more sensors, a preliminary object identification of at least one object of the environment that has a detection confidence below a threshold, wherein the detection confidence indicates a likelihood that the determined object is present in the environment;

operating an apparatus by a control system in an autonomous operation mode based on an instruction to control the apparatus, wherein the instruction to control the apparatus is based on the preliminary object identification;

communicating image data of an environment of the apparatus to a remote computer system;

communicating the preliminary object identification of at least one object of the environment of the apparatus that has the detection confidence below a threshold;

receive an object confirmation indicating an object identification that is different than the preliminary object identification; and

operating the apparatus based on an altered instruction when an object confirmation indicating an object identification that is different than the preliminary object identification is received.

10. The method according to claim 9, wherein the remote computer system is configured to receive an input from a human operator.

11. The method according to claim 9, wherein altering the instruction to control of the apparatus comprises switching the apparatus to a human-controlled mode.

12. The method according to claim 9, wherein the one or more sensors comprises a microphone and the received data comprises an audio signal.

13. The method according to claim 9, wherein the one or more sensors comprises a camera and the received data comprises an image.

14. The method according to claim 9, wherein the preliminary object identification is based on one of radar and laser data.

15. An article of manufacture including a non-transitory computer-readable medium having stored thereon program instructions that, if executed by a processor in a vehicle system, causes the vehicle system to perform operations comprising:

determining, by a processor, based on data from one or more sensors, a preliminary object identification of at least one object of the environment that has a detection confidence below a threshold, wherein the detection confidence indicates a likelihood that the determined object is present in the environment;

operating an apparatus by a control system in an autonomous operation mode based on an instruction to control the apparatus, wherein the instruction to control the apparatus is based on the preliminary object identification;

communicating image data of an environment of the apparatus to a remote computer system responsive to the preliminary object identification;

communicating the preliminary object identification of at least one object of the environment of the apparatus that has the detection confidence below a threshold;

receiving an object confirmation indicating an object identification that is different than the preliminary object identification; and

operating the apparatus based on an altered instruction when an object confirmation indicating an object identification that is different than the preliminary object identification is received.

16. The article of manufacture according to claim 15, wherein the remote computer system is configured to receive an input from a human operator.

17. The article of manufacture according to claim 15, wherein altering the instruction to control of the apparatus comprises switching the apparatus to a human-controlled mode.

18. The article of manufacture according to claim 15, wherein the one or more sensors comprises a microphone and the received data comprises an audio signal.

19. The article of manufacture according to claim 15, wherein the one or more sensors comprises a camera and the received data comprises an image.

20. The article of manufacture according to claim 15, wherein the preliminary object identification is based on one of radar and laser data.