Calibration validation for autonomous vehicle operations

• US 10,268,203 B2
• Filed: 04/20/2017
• Issued: 04/23/2019
• Est. Priority Date: 04/20/2017
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Wisniowski, Matthias
• Primary Examiner(s): Soofi, Yazan A
• Application Number:
  US15/492,910
• Publication Number:
  US 20180307238A1
• Time in Patent Office: 733 Days
• Field of Search: 701 28
• CPC Class Codes: B60R 2300/30 characterised by the type of image processing
• CPC Class Codes: B60W 30/00 Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/87 Combinations of radar systems, e.g. primary radar and secondary radar
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 2013/9316 combined with communication equipment with other vehicles or with base stations
• CPC Class Codes: G01S 2013/9323 Alternative operation using light waves
• CPC Class Codes: G01S 2013/93273 on the top of the vehicles
• CPC Class Codes: G01S 7/497 Means for monitoring or calibrating
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30

Abstract

Systems and method are provided for controlling a vehicle. In one embodiment, a vehicle includes: an imaging device, a reference data source, one or more sensors, one or more actuators, and a controller. The controller detects a stationary condition based at least in part on the sensor output, obtains an image during the stationary condition, identifies a reference position for a reference object based on reference data for a field of view of the imaging device during the stationary condition, identifies the reference object at a second position within the image, validates a transformation associated with the imaging device based on a difference between the reference position and the second position, and thereafter autonomously operates the one or more actuators in a manner that is influenced by the validated transformation.

Claims

1. A method of controlling a vehicle, the method comprising:

detecting, by a control module onboard the vehicle, a stationary condition of the vehicle based at least in part on output of a sensor system onboard the vehicle;

in response to the stationary condition, obtaining, by the control module, an image from an imaging device onboard the vehicle during the stationary condition; and

obtaining, by the control module from a map database, a subset of mapping data corresponding to a vehicle pose during the stationary condition based on a pose of the imaging device relative to the vehicle, wherein the subset of mapping data corresponds to the field of view of the imaging device;

identifying, by the control module, an expected position for a reference object having a fixed position based on the subset of mapping data and a transformation associated with the imaging device;

identifying, by the control module, the reference object at an observed position within the image;

validating, by the control module, the transformation associated with the imaging device based on a difference between the expected position and the observed position; and

operating, by the control module, one or more actuators onboard the vehicle in a manner that is influenced by the validated transformation.

2. The method of claim 1, further comprising, after validating the transformation:

assigning, by the control module, one or more attributes to the image based at least in part on the transformation, wherein operating the one or more actuators comprises autonomously operating, by the control module, one or more actuators onboard the vehicle in a manner that is influenced by the one or more attributes assigned to the image.

3. The method of claim 1, further comprising identifying the reference object within the subset of the mapping data.

4. The method of claim 3, the reference object having the fixed position within the subset of the mapping data, wherein identifying the expected position comprises determining the expected position of the reference object in a coordinate frame associated with the imaging device based on the transformation.

5. The method of claim 4, wherein identifying the reference object at the observed position comprises:

determining a region of the image encompassing the expected position of the reference object; and

searching the region of the image for the reference object.

6. The method of claim 5, further comprising initiating, by the control module, a remedial action with respect to the imaging device in response to a failure to identify the reference object within the region of the image.

7. The method of claim 5, wherein searching the region of the image of the reference object comprises applying a convolutional neural network to the region of the image to locate the reference object within the region.

8. The method of claim 4, wherein validating the transformation comprises:

calculating a validity metric value based on the difference between the expected position and the observed position; and

validating the transformation when the validity metric value is less than a threshold.

9. The method of claim 4, further comprising initiating, by the control module, a remedial action with respect to the imaging device when the difference between the expected position and the observed position is greater than a threshold.

10. A vehicle, comprising:

an imaging device onboard the vehicle;

a map database storing mapping data;

one or more sensors onboard the vehicle;

one or more actuators onboard the vehicle; and

a controller that, by a processor, detects a stationary condition based at least in part on output of the one or more sensors, obtains an image from the imaging device during the stationary condition, obtains a subset of the mapping data corresponding to a field of view of the imaging device at a vehicle pose during the stationary condition based on a pose of the imaging device relative to the vehicle from the mapping data, identifies an expected position for a reference object having a fixed position based on the subset of the mapping data and a transformation associated with the imaging device, identifies the reference object at an observed position within the image, validates the transformation associated with the imaging device based on a difference between the expected position and the observed position, and thereafter autonomously operates the one or more actuators onboard the vehicle in a manner that is influenced by the validated transformation.

11. The vehicle of claim 10, wherein:

the expected position comprises an expected position of the reference object in a coordinate frame associated with the imaging device based on the transformation and the fixed position within the subset of the mapping data.

12. The vehicle of claim 10, wherein the controller determines a region of the image encompassing the expected position of the reference object and searches the region of the image for the reference object to identify the reference object at the observed position within the image.

13. The vehicle of claim 12, wherein the controller initiates a remedial action with respect to the imaging device in response to an absence of the reference object within the region of the image.

14. The vehicle of claim 10, further comprising a ranging device onboard the vehicle providing ranging data, wherein the controller autonomously operates the one or more actuators onboard the vehicle in a manner that is influenced by the validated transformation by:

assigning one or more attributes to the image based at least in part on the ranging data and one or more calibrated parameter values associated with the transformation;

identifying an object based at least in part on the one or more attributes; and

determining a command for the one or more actuators in a manner that is influenced by the object.

15. The vehicle of claim 14, wherein the imaging device comprises a camera and the ranging device comprises a lidar device.

16. A method of controlling a vehicle, the method comprising:

obtaining first data from a sensing device onboard the vehicle during a stationary condition of the vehicle;

identifying an expected position for a reference object within the first data based on reference data corresponding to a field of view of the sensing device and one or more calibrated transformation parameter values associated with the sensing device, wherein identifying the expected position comprises;

obtaining mapping data corresponding to a vehicle pose during the stationary condition from a map database;

identifying a subset of the mapping data corresponding to the field of view of the sensing device based on a pose of the sensing device relative to the vehicle;

identifying the reference object at a fixed position within the subset of the mapping data;

converting the fixed position to the expected position in a coordinate frame associated with the sensing device based on the one or more calibrated transformation parameter values;

identifying the reference object at an observed position within the first data; and

initiating a remedial action with respect to the sensing device when a difference between the expected position and the observed position is greater than a threshold;

validating the one or more calibrated transformation parameter values when the difference between the expected position and the observed position is less than the threshold; and

operating one or more actuators onboard the vehicle in a manner that is influenced by the calibrated transformation parameter values after validation.