Assisted perception for autonomous vehicles

• US 9,563,199 B1
• Filed: 04/16/2014
• Issued: 02/07/2017
• Est. Priority Date: 11/27/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Ferguson, David Ian Franklin, Lo, Wan-Yen, Fairfield, Nathaniel
• Primary Examiner(s): Oh, Harry
• Application Number:
  US14/254,830
• Time in Patent Office: 1,028 Days
• Field of Search: 701/28
• US Class Current: 1/1
• CPC Class Codes: B60W 2050/0075 Automatic parameter input, automatic initialising or calibrating means
• CPC Class Codes: B60W 2050/0215 Sensor drifts or sensor failures
• CPC Class Codes: B60W 2556/45 External transmission of data to or from the vehicle
• CPC Class Codes: B60W 40/04 Traffic conditions
• CPC Class Codes: G05D 1/00 Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots drive control systems specially adapted for autonomous road vehicles B60W60/00
• CPC Class Codes: G05D 1/0022 characterised by the communication link data switching networks in general H04L12/00
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G07C 5/008 communicating information to a remotely located station transmission systems for measured values G08C
• CPC Class Codes: H04L 67/125 involving control of end-device applications over a network

Abstract

Disclosed herein are systems and methods for providing supplemental identification abilities to an autonomous vehicle system. The sensor unit of the vehicle may be configured to receive data indicating an environment of the vehicle, while the control system may be configured to operate the vehicle. The vehicle may also include a processing unit configured to analyze the data indicating the environment to determine at least one object having a detection confidence below a threshold. Based on the at least one object having a detection confidence below a threshold, the processor may communicate at least a subset of the data indicating the environment for further processing. The vehicle is also configured to receive an indication of an object confirmation of the subset of the data. Based on the object confirmation of the subset of the data, the processor may alter the control of the vehicle by the control system.

Claims

1. An apparatus comprising:

a set of sensors configured to receive data, wherein the data represents objects of an environment of the apparatus;

a control system, configured to operate the apparatus in an autonomous operation mode based on instructions to control the apparatus; and

one or more processors configured to;

determine, based on the received data, a preliminary object identification of at least one object of the environment that is in partial view of a sensor of the set of sensors that has a detection confidence below a threshold, wherein the detection confidence indicates a likelihood that the determined object is present in the environment;

alter instructions to control the apparatus by the control system based on the preliminary object identification;

communicate at least a subset of the received data for further processing to a remote computer system responsive to the preliminary object identification;

operate the apparatus based on the altered instructions when an object confirmation indicating an object identification confirming the preliminary object identification is received from the remote computer system; and

operate the apparatus based on the altered instructions that are further altered when an object confirmation indicating an object identification that is different than the preliminary object identification is received from the remote computer system.

2. The apparatus according to claim 1, wherein the remote computer system is configured to receive an input from a human operator.

3. The apparatus according to claim 1, wherein the one or more processors are further configured to provide instructions to the control system to control the apparatus based on the object confirmation.

4. The apparatus according to claim 1, wherein the set of sensors comprises a camera and the received data comprises an image.

5. The apparatus according to claim 1, wherein the set of sensors comprises a microphone and the received data comprises an audio signal.

6. The apparatus according to claim 1, wherein altering the instructions to control of the apparatus comprises switching the apparatus to a human-controlled mode.

7. A method comprising:

receiving data by a set of sensors, wherein the data represents objects of an environment of an apparatus;

operating the apparatus by a control system in an autonomous operation mode based on instructions to control the apparatus; and

determining, by a processor, based on the received data, a preliminary object identification of at least one object of the environment that is in partial view of a sensor of the set of sensors that has a detection confidence below a threshold, wherein the detection confidence indicates a likelihood that the determined object is present in the environment;

alter instructions to control the apparatus by the control system based on the preliminary object identification;

communicating at least a subset of the received data for further processing to a remote computer system responsive to the preliminary object identification;

operating the apparatus based on the altered instructions when an object confirmation indicating an object identification confirming the preliminary object identification is received from the remote computer system; and

operating the apparatus based on the altered instructions that are further altered when an object confirmation indicating an object identification that is different than the preliminary object identification is received from the remote computer system.

8. The method according to claim 7, wherein the remote computer system is configured to receive an input from a human operator.

9. The method according to claim 7, further comprising providing instructions to the control system to control the apparatus based on the object confirmation.

10. The method according to claim 7, wherein the set of sensors comprises a camera and the received data comprises an image.

11. The method according to claim 7, wherein the set of sensors comprises a microphone and the received data comprises an audio signal.

12. The method according to claim 7, wherein altering the instructions to control of the apparatus comprises switching the apparatus to a human-controlled mode.

13. An article of manufacture including a non-transitory computer-readable medium having stored thereon program instructions that, if executed by a processor in a vehicle system, causes the vehicle system to perform operations comprising:

receiving data by a set of sensors, wherein the data represents objects of an environment of the vehicle system; and

determining, based on the received data, a preliminary object identification of at least one object of the environment that is in partial view of a sensor of the set of sensors that has a detection confidence below a threshold, wherein the detection confidence indicates a likelihood that the determined object is present in the environment;

altering instructions to control the vehicle system based on the preliminary object identification;

communicating at least a subset of the received data for further processing to a remote computer system responsive to the preliminary object identification;

operating the apparatus based on the altered instructions when an object confirmation indicating an object identification confirming the preliminary object identification is received from the remote computer system; and

operating the apparatus based on the altered instructions that are further altered when an object confirmation indicating an object identification that is different than the preliminary object identification is received from the remote computer system.

14. The article of manufacture according to claim 13, wherein the remote computer system is configured to receive an input from a human operator.

15. The article of manufacture according to claim 13, further comprising providing instructions to the control system to control the vehicle system based on the object confirmation.

16. The article of manufacture according to claim 13, wherein the set of sensors comprises a camera and the received data comprises an image.

17. The article of manufacture according to claim 13, wherein altering the instructions to control of the vehicle system comprises switching the vehicle system to a human-controlled mode.