Cross-validating sensors of an autonomous vehicle

• US 9,555,740 B1
• Filed: 11/20/2015
• Issued: 01/31/2017
• Est. Priority Date: 09/27/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Zhu, Jiajun, Dolgov, Dmitri A., Urmson, Christopher Paul
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): Tarko, Asmamaw G
• Application Number:
  US14/946,967
• Time in Patent Office: 438 Days
• Field of Search: 348/148
• US Class Current: 1/1
• CPC Class Codes: B60Q 9/008 for anti-collision purposes
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: B60W 2050/0043 Signal treatments, identification of variables or parameters, parameter estimation or state estimation
• CPC Class Codes: B60W 2050/0215 Sensor drifts or sensor failures
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 50/0205 Diagnosing or detecting failures; Failure detection models
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: G01D 18/00 Testing or calibrating apparatus or arrangements provided for in groups G01D1/00 - G01D15/00
• CPC Class Codes: G01S 13/723 by using numerical data
• CPC Class Codes: G01S 13/862 Combination of radar systems with sonar systems
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 7/40 Means for monitoring or calibrating
• CPC Class Codes: G01S 7/497 Means for monitoring or calibrating
• CPC Class Codes: G06V 10/98 Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G08G 1/163 : involving continuous checking
• CPC Class Codes: G08G 1/166 : for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

Methods and systems are disclosed for cross-validating a second sensor with a first sensor. Cross-validating the second sensor may include obtaining sensor readings from the first sensor and comparing the sensor readings from the first sensor with sensor readings obtained from the second sensor. In particular, the comparison of the sensor readings may include comparing state information about a vehicle detected by the first sensor and the second sensor. In addition, comparing the sensor readings may include obtaining a first image from the first sensor, obtaining a second image from the second sensor, and then comparing various characteristics of the images. One characteristic that may be compared are object labels applied to the vehicle detected by the first and second sensor. The first and second sensors may be different types of sensors.

Claims

1. A system for validating the accuracy of a sensor, the system comprising:

one or more computing devices having one or more processors configured to;

receive first sensor information from a first sensor configured to detect objects in a vehicle'"'"'s environment;

receive second sensor information from a second sensor configured to detect objects in a vehicle'"'"'s environment;

identify a first object and a first characteristic value of the first object using the first sensor information;

identify the first object and a second characteristic value of the first object using the second sensor information, wherein the first characteristic and the second characteristic are of a same type;

determine a deviation value based on the first characteristic value with the second characteristic value; and

use the deviation value to determine whether the first sensor is accurately detecting the object by comparing the deviation value to a deviation threshold value identifying an acceptable amount of difference between the first and second characteristic values; and

when the comparison indicates that the deviation value does not meet the deviation threshold value, determine that the first sensor has had an operational failure.

2. The system of claim 1, further comprising the first sensor and the second sensor.

3. The system of claim 2, wherein the first sensor includes a laser and the second sensor includes a camera.

4. The system of claim 1, wherein the first and second characteristic values each defines a dimension of the object.

5. The system of claim 1, wherein the first and second characteristic values each defines a position of the object.

6. The system of claim 1, wherein the first and second characteristic values each define a heading of the object.

7. The system of claim 1, further comprising the vehicle.

8. The system of claim 1, wherein the one or more processors are further configured to:

memory storing a failure count value for each of the first and second sensors; and

wherein the one or more processors are further configured to change the failure count value for the first sensor based upon the determination that the first sensor has had an operational failure.

9. The system of claim 8, wherein the vehicle includes an autonomous driving mode and a manual driving mode, and the one or more processors are further configured to use the failure count value to determine whether to switch from the autonomous driving mode to a manual driving mode.

10. The system of claim 1, wherein the one or more processors are further configured to select the deviation threshold value from a plurality of deviation threshold values based on the type of the first and second characteristic values.

11. The system of claim 1, wherein the first and second characteristic values are numerical values.

12. A method for validating the accuracy of a sensor, the method comprising:

receiving, by one or more processors, first sensor information from a first sensor configured to detect objects in a vehicle'"'"'s environment;

receiving, by the one or more processors, second sensor information from a second sensor configured to detect objects in a vehicle'"'"'s environment;

identifying, by the one or more processors, a first object and first characteristic value of the first object using the first sensor information;

identifying, by the one or more processors, the first object and second characteristic value of the first object using the second sensor information, wherein the first characteristic and the second characteristic are of a same type;

determining, by the one or more processors, a deviation value based on the first characteristic value with the second characteristic value; and

using, by the one or more processors, the deviation value to determine whether the first sensor is accurately detecting the object by comparing the deviation value to a deviation threshold value identifying an acceptable amount of difference between the first and second characteristic values; and

when the comparison indicates that the deviation value does not meet the deviation threshold value, determining that the first sensor has had an operational failure.

13. The method of claim 12, further comprising changing a stored failure count value for the first sensor based upon the determination that the first sensor has had an operational failure.

14. The method of claim 13, wherein the vehicle includes an autonomous driving mode and a manual driving mode, and the method further comprises, using the failure count value to determine whether to switch from the autonomous driving mode to a manual driving mode.

15. The method of claim 12, further comprising selecting the deviation threshold value from a plurality of deviation threshold values based on the type of the first and second characteristic values.

16. The system of method 12, wherein the first sensor includes a laser and the second sensor includes a camera.

17. A non-transitory computer readable medium on which instructions are stored, the instructions, when executed by one or more processors, cause the one or more processors to perform a method for validating the accuracy of a sensor, the method comprising:

receiving first sensor information from a first sensor configured to detect objects in a vehicle'"'"'s environment;

receiving second sensor information from a second sensor configured to detect objects in a vehicle'"'"'s environment;

identifying a first object and first characteristic value of the first object using the first sensor information;

identifying the first object and second characteristic value of the first object using the second sensor information, wherein the first characteristic and the second characteristic are of a same type;

determining a deviation value based on the first characteristic value with the second characteristic value;

selecting a deviation threshold value from a plurality of deviation threshold values based on the type of the first and second characteristic values;

using the deviation value to determine whether the first sensor is accurately detecting the object by comparing the deviation value to a deviation threshold value identifying an acceptable amount of difference between the first and second characteristic values; and

when the comparison indicates that the deviation value does not meet the deviation threshold value, determine that the first sensor has had an operational failure.

18. The medium of claim 17, wherein the method further comprises changing a stored failure count value for the first sensor based upon the determination that the first sensor has had an operational failure.

19. The medium of claim 18, wherein the vehicle includes an autonomous driving mode and a manual driving mode, and the method further comprises, using the failure count value to determine whether to switch from the autonomous driving mode to a manual driving mode.

20. The medium of claim 17, wherein the first sensor includes a laser and the second sensor includes a camera.