Cross-validating sensors of an autonomous vehicle

• US 9,221,396 B1
• Filed: 09/27/2012
• Issued: 12/29/2015
• Est. Priority Date: 09/27/2012
• Status: Expired due to Fees

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Zhu, Jiajun, Dolgov, Dmitri A., Urmson, Christopher Paul
• Primary Examiner(s): Pe, Geepy
• Assistant Examiner(s): Tarko, Asmamaw G
• Application Number:
  US13/628,986
• Time in Patent Office: 1,188 Days
• Field of Search: 348/148
• US Class Current: 1/1
• CPC Class Codes: B60Q 9/008 for anti-collision purposes
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: B60W 2050/0043 Signal treatments, identification of variables or parameters, parameter estimation or state estimation
• CPC Class Codes: B60W 2050/0215 Sensor drifts or sensor failures
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 50/0205 Diagnosing or detecting failures; Failure detection models
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: G01D 18/00 Testing or calibrating apparatus or arrangements provided for in groups G01D1/00 - G01D15/00
• CPC Class Codes: G01S 13/723 by using numerical data
• CPC Class Codes: G01S 13/862 Combination of radar systems with sonar systems
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 7/40 Means for monitoring or calibrating
• CPC Class Codes: G01S 7/497 Means for monitoring or calibrating
• CPC Class Codes: G06V 10/98 Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G08G 1/163 : involving continuous checking
• CPC Class Codes: G08G 1/166 : for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

Methods and systems are disclosed for cross-validating a second sensor with a first sensor. Cross-validating the second sensor may include obtaining sensor readings from the first sensor and comparing the sensor readings from the first sensor with sensor readings obtained from the second sensor. In particular, the comparison of the sensor readings may include comparing state information about a vehicle detected by the first sensor and the second sensor. In addition, comparing the sensor readings may include obtaining a first image from the first sensor, obtaining a second image from the second sensor, and then comparing various characteristics of the images. One characteristic that may be compared are object labels applied to the vehicle detected by the first and second sensor. The first and second sensors may be different types of sensors.

Claims

1. An apparatus comprising:

a first sensor configured to detect a vehicle in a driving environment;

a second sensor configured to detect the vehicle in the driving environment;

memory that stores;

detailed map information for the driving environment;

first sensor parameters for determining whether the vehicle is present in a first image of the driving environment;

second sensor parameters for determining whether the vehicle is present in a second image of the driving environment; and

a failure count threshold identifying an action to be performed when the failure count threshold is exceeded; and

one or more processors in communication with the memory, the first sensor and the second sensor, the one or more processors being configured to;

receive first sensor information from the first sensor, the first sensor information being used to produce the first image;

receive second sensor information from the second sensor, the second sensor information being used to produce the second image;

determine whether the vehicle is in both the first image and the second image;

determine a deviation value by comparing the first image with the second image when it is determined that the vehicle is present in the first image and the second image; and

compare the determined deviation value with a deviation threshold to determine whether the first or second sensor has a problem detecting the vehicle, wherein the comparison of the determined deviation value with the deviation threshold causes the failure count threshold to be exceeded; and

when the failure count threshold is exceeded, perform the action identified by the failure count threshold.

2. The apparatus of claim 1, wherein the first sensor is a first type of imaging sensor and the second sensor is a second type of imaging sensor, the first type of imaging sensor being different than the second type of imaging sensor.

3. The apparatus of claim 1, wherein the one or more processors are further configured to select the first sensor as a reference sensor for determining whether the second sensor has the problem detecting the vehicle.

4. The apparatus of claim 3, wherein the one or more processors are configured to selects the first sensor based on a predetermined time interval.

5. The apparatus of claim 3, wherein the one or more processors are configured to selects the second sensor for cross-validation with the first sensor based on an amount of time that has elapsed since the second sensor was last cross-validated.

6. The apparatus of claim 1, wherein:

the memory further stores a multi-level failure threshold comprising a plurality of levels, each level of the plurality of levels is associated with an action for the processor to perform, the failure count threshold being a first level of the plurality of levels is associated with a condition of the deviation value exceeding the deviation threshold; and

the one or more processors are further configured to, prior to performing the action, identify the first level and the action associated with the failure count threshold when the condition of the deviation value exceeding the deviation threshold is met.

7. The apparatus of claim 1, wherein:

the first image comprises a plurality of images determined from the first sensor information, the first sensor information being received during a predetermined time interval;

the second image comprises a plurality of images determined from the second sensor information, the second sensor information being received during the predetermined time interval; and

the one or more processors are further configured to;

determine first state information about the vehicle from the first plurality of images;

determine second state information about the vehicle from the second plurality of images; and

compare the first state information with the second state information to determine the deviation value.

8. The apparatus of claim 7, wherein:

the first state information comprises at least one of a first vehicle speed, a first vehicle position, or a first vehicle heading;

the second state information comprises at least one of a second vehicle speed, a second vehicle position, or a second vehicle heading; and

the one or more processors are further configured to compares at least one of the first vehicle speed, first vehicle position or first vehicle heading with at least one of the second vehicle speed, second vehicle position, or second vehicle heading.

9. The apparatus of claim 1, wherein:

the first image comprises a first object label encompassing the vehicle, the first object label comprising a first parameter that defines a characteristic of the first object label;

the second image comprises a second object label encompassing the vehicle, the second object label comprising a second parameter that defines a characteristic of the second object label; and

the one or more processors are configured to compare the first image with the second image to determine the deviation value by comparing the first object label with the second object label.

10. The apparatus of claim 9, wherein the processor compares the first object label with the second object label by comparing the first parameter with the second parameter.

11. A method comprising:

receiving, with one or more processors in communication with a first sensor, first sensor information, the first sensor information being used to produce a first image of a driving environment;

receiving, with the one or more processors in communication with the second sensor, second sensor information, the second sensor information being used to produce the second image of the driving environment;

determining, with the one or more processors, whether a vehicle is present in both the first image and the second image;

determining, with the one or more processors, a deviation value by comparing the first image with the second image when it is determined that the vehicle is present in the first image and the second image;

comparing, with the one or more processors, the determined deviation value with a deviation threshold to determine whether the first or second sensor has a problem detecting the vehicle, wherein the comparison of the determined deviation value with the deviation threshold causes a failure count threshold to be exceeded, the failure count threshold identifying an action to be performed when the failure count threshold is exceeded; and

when the failure count threshold is exceeded, perform the action identified by the failure count threshold.

12. The method of claim 11, wherein the first sensor is a first type of imaging sensor and the second sensor is a second type of imaging sensor, the first type of imaging sensor being different than the second type of imaging sensor.

13. The method of claim 11, further comprising:

selecting, with the one or more processors, the first sensor as a reference sensor for determining whether the second sensor has the problem detecting the vehicle.

14. The method of claim 13, further comprising:

selecting, with the one or more processors, the first sensor based on a predetermined time interval.

15. The method of claim 13, further comprising:

selecting, with the one or more processors, the second sensor for cross-validation with the first sensor based on an amount of time that has elapsed since the second sensor was last cross-validated.

16. The method of claim 11, further comprising:

determining whether the deviation value exceeds a deviation threshold associated with a first level of a multi-level failure threshold, each level of the plurality of levels is associated with an action for the processor to perform, the failure count threshold being the first level of the plurality of levels associated with a condition of the deviation value exceeding the deviation threshold; and

prior to performing the action, identifying the first level and the action associated with the failure count threshold when the condition of the deviation value exceeding the deviation threshold is met.

17. The method of claim 11, wherein:

the first image comprises a plurality of images determined from the first sensor information, the first sensor information being received during a predetermined time interval;

the second image comprises a plurality of images determined from the second sensor information, the second sensor information being received during the predetermined time interval; and

the method further comprises;

determining, with the one or more processors, the first state information about the vehicle from the first plurality of images;

determining, with the one or more processors, second state information about the vehicle from the second plurality of images; and

comparing, with the one or more processors, the first state information with the second state information to determine the deviation value.

18. The method of claim 17, wherein:

the first state information comprises at least one of a first vehicle speed, a first vehicle position, or a first vehicle heading;

the second state information comprises at least one of a second vehicle speed, a second vehicle position, or a second vehicle heading; and

the method further comprises;

comparing, with the one or more processors, at least one of the first vehicle speed, first vehicle position or first vehicle heading with at least one of the second vehicle speed, second vehicle position, or second vehicle heading.

19. The method of claim 11, wherein:

the first image comprises a first object label encompassing the vehicle, the first object label comprising a first parameter that defines a characteristic of the first object label;

the second image comprises a second object label encompassing the vehicle, the second object label comprising a second parameter that defines a characteristic of the second object label; and

the method further comprises;

comparing, with the one or more processors, the first image with the second image to determine the deviation value by comparing the first object label with the second object label.

20. The method of claim 19, wherein comparing the first object label with the second object label comprises comparing the first parameter with the second parameter.