Cross-validating sensors of an autonomous vehicle

• US 9,868,446 B1
• Filed: 12/19/2016
• Issued: 01/16/2018
• Est. Priority Date: 09/27/2012
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Zhu, Jiajun, Dolgov, Dmitri A., Urmson, Christopher Paul
• Primary Examiner(s): Rahaman, Mohammed
• Assistant Examiner(s): Tarko, Asmamaw G
• Application Number:
  US15/383,198
• Time in Patent Office: 393 Days
• CPC Class Codes: B60Q 9/008 for anti-collision purposes
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: B60W 2050/0043 Signal treatments, identification of variables or parameters, parameter estimation or state estimation
• CPC Class Codes: B60W 2050/0215 Sensor drifts or sensor failures
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 50/0205 Diagnosing or detecting failures; Failure detection models
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: G01D 18/00 Testing or calibrating apparatus or arrangements provided for in groups G01D1/00 - G01D15/00
• CPC Class Codes: G01S 13/723 by using numerical data
• CPC Class Codes: G01S 13/862 Combination of radar systems with sonar systems
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 7/40 Means for monitoring or calibrating
• CPC Class Codes: G01S 7/497 Means for monitoring or calibrating
• CPC Class Codes: G06V 10/98 Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G08G 1/163 : involving continuous checking
• CPC Class Codes: G08G 1/166 : for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

Methods and systems are disclosed for cross-validating a second sensor with a first sensor. Cross-validating the second sensor may include obtaining sensor readings from the first sensor and comparing the sensor readings from the first sensor with sensor readings obtained from the second sensor. In particular, the comparison of the sensor readings may include comparing state information about a vehicle detected by the first sensor and the second sensor. In addition, comparing the sensor readings may include obtaining a first image from the first sensor, obtaining a second image from the second sensor, and then comparing various characteristics of the images. One characteristic that may be compared are object labels applied to the vehicle detected by the first and second sensor. The first and second sensors may be different types of sensors.

Claims

1. A method for validating a sensor configured to detect objects, the method comprising:

receiving, by one or more processors, laser sensor data from a laser-based sensor configured to detect objects in an environment of the laser-based sensor;

receiving, by the one or more processors, an image of the environment captured by a second sensor configured to detect objects in the environment;

identifying, by the one or more processors, a vehicle in the image;

assigning, by the one or more processors, a first object value to the vehicle in the image including a first bounding box for the vehicle in the image, the first object value identifying a first dimension of the first bounding box defining a first size of the first bounding box;

detecting, by the one or more processors, the vehicle from the laser sensor data;

assigning, by the one or more processors, a second object value to the vehicle from the laser sensor data including a second bounding box for the vehicle in the image, the second object value identifying a second dimension of the second bounding box defining a second size of the second bounding box; and

validating, by the one or more processors, accuracy of the laser-based sensor by comparing the first dimension to the second dimension.

2. The method of claim 1, wherein the first bounding box defines a location of the vehicle in the image.

3. The method of claim 1, wherein the first bounding box is defined as a rectangle.

4. The method of claim 1, wherein the second sensor is a second laser-based sensor.

5. The method of claim 1, wherein the second sensor is a camera-based sensor that captured the image.

6. The method of claim 1, wherein the second sensor is a radar-based sensor.

7. The method of claim 1, further comprising:

determining, using the laser sensor data, a first speed value for the vehicle from the laser sensor data; and

determining, using the image, a second speed value for the vehicle in the image, wherein validating the accuracy of the laser-based sensor is further based on a comparison of the first speed value and the second speed value.

8. The method of claim 7, wherein validating the accuracy of the laser-based sensor further includes:

determining a difference between the first speed value and the second speed value; and

comparing the difference to a threshold speed value.

9. The method of claim 1, further comprising:

determining, using the laser sensor data, a first heading value for a heading of the vehicle from the laser sensor data; and

determining, using the image, a second heading value for a heading of the vehicle in the image, wherein validating the accuracy of the laser-based sensor is further based on a comparison of the first heading value and the second heading value.

10. The method of claim 1, wherein assigning the first object value includes assigning a first geographic position for the vehicle in the image, assigning the second object value includes assigning a second geographic position for the vehicle from the laser sensor data, and the method further comprising:

determining a distance between the first geographic position and the second geographic position, and wherein validating the accuracy of the laser-based sensor further includes comparing the distance to a positional threshold value.

11. The method of claim 1, further comprising:

based on validating the accuracy of the laser-based sensor, determining whether a failure threshold has been met; and

when the failure threshold has been met, displaying an error message.

12. The method of claim 1, further comprising:

based on validating the accuracy of the laser-based sensor, determining whether a failure threshold has been met; and

when the failure threshold has been met, provide a request that a human operator take control of the vehicle.

13. The method of claim 1, further comprising:

based on validating the accuracy of the laser-based sensor, increase a count value corresponding to a number of failures for the laser-based sensor;

comparing the count value to a predetermined failure value; and

based on the comparison of the count value to the predetermined failure value, displaying an error message.

14. The method of claim 1, wherein the at least one dimension includes a height estimate of the vehicle and a width estimate of the vehicle.

19. The method of claim 1, wherein the first dimension of the first bounding box is at least one of a height dimension or width dimension.

20. The method of claim 1, wherein the first dimension of the first bounding box includes a pixel coordinate that defines a location of the first bounding box.

15. A system for validating a sensor configured to detect objects, the system comprising:

one or more processors configured to;

receive laser sensor data from a laser-based sensor configured to detect objects in an environment of the laser-based sensor;

receive an image of the environment captured by a second sensor configured to detect objects in the environment;

identify a vehicle in the image;

assign a first object value to the vehicle in the image, the first object value identifying at least one dimension of the vehicle in the image;

detect the vehicle from the laser sensor data;

assign a second object value to the vehicle from the laser sensor data, the second object value identifying the at least one dimension of the vehicle from the laser sensor data;

assign a first object value to the vehicle in the image including a first bounding box for the vehicle in the image, the first object value identifying a first dimension of the first bounding box defining a first size of the first bounding box;

assign a second object value to the vehicle in the image including a second bounding box for the vehicle in the image, the second object value identifying a second dimension of the second bounding box defining a second size of the second bounding box; and

validate accuracy of the laser-based sensor by comparing the first dimension to the second dimension.

16. The system of claim 15, wherein the second sensor is a second laser-based sensor, and the system further comprises the laser-based sensor and the second sensor.

17. The system of claim 15, wherein the one or more processors are further configured to:

determine, using the laser sensor data, a first speed value for the vehicle from the laser sensor data; and

determine, using the image, a second speed value for the vehicle in the image, wherein validating the accuracy of the laser-based sensor is further based on a comparison of the first speed value and the second speed value.

18. The system of claim 17, wherein the one or more processors are further configured to validate the accuracy of the laser-based sensor by:

determining a difference between the first speed value and the second speed value; and

comparing the difference to a threshold speed value.