Assisted perception for autonomous vehicles

• US 9,201,421 B1
• Filed: 04/16/2014
• Issued: 12/01/2015
• Est. Priority Date: 11/27/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Fairfield, Nathaniel, Herbach, Josh Seth, Furman, Vadim
• Primary Examiner(s): Beaulieu, Yonel
• Application Number:
  US14/254,810
• Time in Patent Office: 594 Days
• Field of Search: 701/23, 701/24, 701/28
• US Class Current: 1/1
• CPC Class Codes: B60W 2050/0075 Automatic parameter input, automatic initialising or calibrating means
• CPC Class Codes: B60W 2050/0215 Sensor drifts or sensor failures
• CPC Class Codes: B60W 2556/45 External transmission of data to or from the vehicle
• CPC Class Codes: B60W 40/04 Traffic conditions
• CPC Class Codes: G05D 1/00 Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots drive control systems specially adapted for autonomous road vehicles B60W60/00
• CPC Class Codes: G05D 1/0022 characterised by the communication link data switching networks in general H04L12/00
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G07C 5/008 communicating information to a remotely located station transmission systems for measured values G08C
• CPC Class Codes: H04L 67/125 involving control of end-device applications over a network

Abstract

Disclosed herein are systems and methods for providing supplemental identification abilities to an autonomous vehicle system. The sensor unit of the vehicle may be configured to receive data indicating an environment of the vehicle, while the control system may be configured to operate the vehicle. The vehicle may also include a processing unit configured to analyze the data indicating the environment to determine at least one object having a detection confidence below a threshold. Based on the at least one object having a detection confidence below a threshold, the processor may communicate at least a subset of the data indicating the environment for further processing. The vehicle is also configured to receive an indication of an object confirmation of the subset of the data. Based on the object confirmation of the subset of the data, the processor may alter the control of the vehicle by the control system.

Claims

1. An autonomous vehicle comprising:

a set of sensors configured to receive data, wherein the data represents objects of an environment of the autonomous vehicle;

a control system, configured to operate the autonomous vehicle; and

one or more processors configured to;

determine, based on the received data, at least one object of the environment that has a detection confidence below a threshold, wherein the detection confidence is indicative of a likelihood that the determined object is correctly identified in the environment; and

based on the at least one object having a detection confidence below the threshold;

communicate, from the received data, image data of the at least one object for further processing to a remote operator,

determine at least one natural-language question associated with the at least one object having a detection confidence below the threshold,

communicate the at least one natural-language question to the remote operator,

receive an answer to the at least one natural-language question from the remote operator providing an object indication, and

provide instructions to control the autonomous vehicle by the control system based on the object indication.

2. The apparatus according to claim 1, wherein the one or more processors are further configured to:

provide a bounding box substantially around the at least one object in the image data; and

communicate the image data of the least one object including the bounding box substantially around the at least one object for further processing to the remote operator.

3. The apparatus according to claim 1, wherein the one or more processors are further configured to:

perform object detection of objects within image data in the received data; and

determine that the at least one object has the detection confidence below the threshold based on being unable to identify the at least one object with a detection confidence above the threshold.

4. The apparatus according to claim 3, wherein the one or more processors are further configured to:

determine the at least one natural-language question associated with the at least one object having the detection confidence below the threshold based on a result of the object detection of the at least one object, wherein the at least one natural-language question indicates a possible identification of the at least one object and requests confirmation.

5. The apparatus according to claim 1, wherein the set of sensors comprises a LIDAR sensor and the data indicating the environment comprises LIDAR data.

6. The apparatus according to claim 1, wherein the set of sensors comprises a camera and the data indicating the environment comprises an image.

7. The apparatus according claim 1, wherein the set of sensors comprises a microphone and the data indicating the environment comprises an audio signal.

8. The apparatus according claim 1, wherein providing the instructions to control the autonomous vehicle by the control system based on the object indication comprises providing instructions to switch a mode of the autonomous vehicle to a human-controlled mode.

9. A method comprising:

receiving data by a set of sensors, wherein the data represents objects of an environment of an autonomous vehicle;

operating the autonomous vehicle by a control system;

determining, by a processor, based on the received data, at least one object of the environment that has a detection confidence below a threshold, wherein the detection confidence is indicative of a likelihood that the determined object is correctly identified in the environment; and

based on the at least one object having a detection confidence below the threshold, the processor;

communicating, from the received data, image data of the at least one object for further processing to a remote operator,

determining at least one natural-language question associated with the at least one object having a detection confidence below the threshold,

communicating the at least one natural-language question to the remote operator,

receiving an answer to the at least one natural-language question from the remote operator providing an object indication, and

providing instructions to control the autonomous vehicle by the control system based on the object indication.

10. The method according to claim 9, further comprising:

providing a bounding box substantially around the at least one object in the image data; and

communicating the image data of the least one object including the bounding box substantially around the at least one object for further processing to the remote operator.

11. The method according to claim 9, further comprising:

performing object detection of objects within image data in the received data; and

determining that the at least one object has the detection confidence below the threshold based on being unable to identify the at least one object with a detection confidence above the threshold.

12. The method according to claim 11, further comprising:

determining the at least one natural-language question associated with the at least one object having the detection confidence below the threshold based on a result of the object detection of the at least one object, wherein the at least one natural-language question indicates a possible identification of the at least one object and requests confirmation.

13. The method according to claim 9, wherein the set of sensors comprises a LIDAR sensor and the data indicating the environment comprises LIDAR data.

14. The method according to claim 9, wherein the set of sensors comprises a camera and the data indicating the environment comprises an image.

15. The method according claim 9, wherein the set of sensors comprises a microphone and the data indicating the environment comprises an audio signal.

16. The method according claim 9, wherein providing the instructions to control the autonomous vehicle by the control system based on the object indication comprises providing instructions to switch a mode of the autonomous vehicle to a human-controlled mode.

17. An article of manufacture including a non-transitory computer-readable medium having stored thereon instructions that, when executed by a processor in a vehicle system, causes the vehicle system to perform operations comprising:

receiving data by a set of sensors, wherein the data represents objects of an environment of an autonomous vehicle;

determining based on the received data, at least one object of the environment that has a detection confidence below a threshold, wherein the detection confidence is indicative of a likelihood that the determined object is correctly identified in the environment; and

based on the at least one object having a detection confidence below the threshold;

communicating, from the received data, image data of the at least one object for further processing to a remote operator,

determining at least one natural-language question associated with the at least one object having a detection confidence below the threshold,

communicating the at least one natural-language question to the remote operator,

receiving an answer to the at least one natural-language question from the remote operator providing an object indication, and

providing instructions to control the autonomous vehicle by the control system based on the object indication.

18. The article of manufacture according to claim 17, wherein the operations further comprise:

providing a bounding box substantially around the at least one object in the image data; and

communicating the image data of the least one object including the bounding box substantially around the at least one object for further processing to the remote operator.

19. The article of manufacture according to claim 17, wherein the operations further comprise:

performing object detection of objects within image data in the received data; and

determining that the at least one object has the detection confidence below the threshold based on being unable to identify the at least one object with a detection confidence above the threshold.

20. The article of manufacture according to claim 17, wherein the operations further comprise:

determining the at least one natural-language question associated with the at least one object having the detection confidence below the threshold based on a result of the object detection of the at least one object, wherein the at least one natural-language question indicates a possible identification of the at least one object and requests confirmation.

21. A method comprising:

determining, by a processor, based on received data, a preliminary identification of at least one object of the environment;

operating an autonomous vehicle by a control system, based on the preliminary identification; and

determining, by a processor the at least one object of the environment that has a detection confidence below a threshold, wherein the detection confidence indicates a likelihood that the determined object is present in the environment; and

based on the at least one object having a detection confidence below the threshold, the processor;

communicating, from the received data, image data of the at least one object for further processing to a remote operator,

determining at least one natural-language question associated with the at least one object having a detection confidence below the threshold,

communicating the at least one natural-language question to the remote operator,

receiving an answer to the at least one natural-language question from the remote operator providing a corrected identification, and

providing instructions to control the autonomous vehicle by the control system based on the corrected identification.

22. The method according to claim 21, further comprising:

performing object detection of objects within image data in the received data; and

determining that the at least one object has the detection confidence below the threshold based on being unable to identify the at least one object with a detection confidence above the threshold.

23. The method according to claim 21, further comprising:

determining the at least one natural-language question associated with the at least one object having the detection confidence below the threshold based on a result of the object detection of the at least one object, wherein the at least one natural-language question indicates a possible identification of the at least one object and requests confirmation.

24. The method according to claim 21, wherein altering the instructions to control of the autonomous vehicle comprises switching the autonomous vehicle to a human-controlled mode.