Method for image-based vehicle localization

• US 9,884,623 B2
• Filed: 07/13/2016
• Issued: 02/06/2018
• Est. Priority Date: 07/13/2015
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Fasola, Juan, Glaser, Stuart, Branson, Elliot, Vogt, Kyle
• Primary Examiner(s): Badii, Behrang
• Assistant Examiner(s): Testardi, David A
• Application Number:
  US15/209,088
• Publication Number:
  US 20170015317A1
• Time in Patent Office: 573 Days
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2552/00 Input parameters relating to infrastructure
• CPC Class Codes: B60W 2552/53 Road markings, e.g. lane marker or crosswalk
• CPC Class Codes: B60W 30/12 Lane keeping
• CPC Class Codes: B60W 40/10 related to vehicle motion
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects

Abstract

A method for image-based vehicle localization includes measuring, at a vehicle, a position and a heading of the vehicle; capturing, at an image capture system of the vehicle, road surface image data of a road surface; processing the road surface image data to correct for distortion in the road surface image data due to pitch and roll of the vehicle; performing feature detection on the processed road surface image data to detect lane markers on the road surface; generating a local map based on the detected lane markers; wherein generating the local map comprises identifying a lane demarcated by the detected lane markers; and controlling the vehicle according to the local map.

Claims

1. A method for image-based localization of a vehicle, comprising:

capturing, at an image capture system of the vehicle, road surface image data of a road surface;

processing, by a dynamic model, a vehicle speed and a vehicle wheel angle to determine an expected time distortion contribution;

performing, by the dynamic model, feature detection on the road surface image data to detect lane markers on the road surface based on an estimated pitch of the vehicle, an estimated roll of the vehicle, and the expected time distortion contribution;

generating a local map comprising a lane demarcated by the detected lane markers; and

controlling the vehicle according to the local map.

2. The method of claim 1, further comprising retrieving pre-existing map data based on a position and a heading of the vehicle, and wherein the performing the feature detection is based on the pre-existing map data.

3. The method of claim 2, wherein the pre-existing map data includes a set of pre-existing road surface images that are linked to a set of known locations.

4. The method of claim 2, wherein the performing feature detection further comprises generating expected lane marker locations from the the pre-existing map data and using the expected lane marker locations to aid in the feature detection of lane markers on the road surface.

5. The method of claim 1, wherein the generating the local map comprises computing a set of estimated lane boundaries, each of the set generated from road surface image data captured at different times, and identifying the lane from an average of the set of estimated lane boundaries.

6. The method of claim 1, further comprising generating the estimated pitch and the estimated roll using an inertial measurement unit coupled to the vehicle.

7. The method of claim 1, wherein the feature detection comprises detecting the lane markers using multiple binary feature detectors operating on pixels of the road surface image data.

8. A method for image-based localization of a vehicle, comprising:

capturing, at an image capture system of the vehicle, road surface image data of a road surface;

processing the road surface image data to correct for distortion in the road surface image data due to an estimated pitch and an estimated roll of the vehicle, resulting in processed road surface image data;

performing feature detection using multiple binary feature detectors operating on pixels of the processed road surface image data to detect lane markers on the road surface;

generating a local map comprising a lane demarcated by the detected lane markers; and

controlling the vehicle according to the local map,

wherein a first binary feature detector of the multiple binary feature detectors returns a true value for a first pixel if an intensity difference between the first pixel and a second pixel is greater than a first threshold value and a false value for the first pixel otherwise, wherein the first pixel and second pixel are separated by a transformed lane marker width, and wherein the method further comprises generating the transformed lane marker width from an expected lane marker width and the location of the first pixel within the road surface image data.

9. The method of claim 8, wherein a second binary feature detector of the multiple binary feature detectors returns a true value for the first pixel if an intensity difference between the first pixel and a third pixel is greater than a second threshold value and a false value for the first pixel otherwise, wherein the first pixel and third pixel are separated by a doubled transformed lane marker width, and wherein the method further comprises generating the doubled transformed lane marker width by doubling the transformed lane marker width.

10. The method of claim 9, further comprising generating a first binary map using the first binary feature detector, generating a second binary map using the second binary feature detector, and combining the first and second binary maps.

11. The method of claim 9, wherein the combining the first and second binary maps comprises merging the first and second binary maps using a region-growing technique, and wherein the merging the first and second binary maps comprises including regions of true values of the second binary map if and only if the regions couple or overlap with regions of true values of the first binary map.

12. The method of claim 11, wherein the region-growing technique is initiated at pixels of the road surface image data corresponding to a smallest distance from the image capture system.

13. The method of claim 8, further comprising modifying the first threshold value based on environmental conditions of an environment external to the vehicle.

14. The method of claim 13, wherein the modifying the first threshold value based on environmental conditions comprises scaling the first threshold value based on an average intensity of the road surface image data.

15. A system for image-based localization of a vehicle, comprising:

an image capture system configured to capture road surface image data of a road surface;

a computer readable medium that stores a dynamic model that is configured to, by a processor, process a vehicle speed and a vehicle wheel angle to determine an expected time distortion contribution, and perform feature detection on the road surface image data to detect lane markers on the road surface based on an estimated pitch of the vehicle, an estimated roll of the vehicle, and the expected time distortion contribution;

a controller configured to generate a local map comprising a lane demarcated by the detected lane markers, and control the vehicle according to the local map.

16. The system of claim 15, further comprising an inertial measurement unit that provides the estimated pitch of the vehicle and the estimated roll of the vehicle.

17. The system of claim 15, wherein the computer readable medium that stores pre-existing map data that includes a set of pre-existing road surface images that are linked to a set of known locations, and wherein the feature detection is based on the pre-existing road surface images.