METHOD FOR IMAGE-BASED VEHICLE LOCALIZATION

• US 20170015317A1
• Filed: 07/13/2016
• Published: 01/19/2017
• Est. Priority Date: 07/13/2015
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: Cruise Automation, Inc. (General Motors Company)
• Inventors: Fasola, Juan, Branson, Elliot, Glaser, Stuart, Vogt, Kyle
• Granted Patent: US 9,884,623 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2552/00 Input parameters relating to infrastructure
• CPC Class Codes: B60W 2552/53 Road markings, e.g. lane marker or crosswalk
• CPC Class Codes: B60W 30/12 Lane keeping
• CPC Class Codes: B60W 40/10 related to vehicle motion
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects

Abstract

A method for image-based vehicle localization includes measuring, at a vehicle, a position and a heading of the vehicle; capturing, at an image capture system of the vehicle, road surface image data of a road surface; processing the road surface image data to correct for distortion in the road surface image data due to pitch and roll of the vehicle; performing feature detection on the processed road surface image data to detect lane markers on the road surface; generating a local map based on the detected lane markers; wherein generating the local map comprises identifying a lane demarcated by the detected lane markers; and controlling the vehicle according to the local map.

Claims

1. A method for image-based vehicle localization comprising:

generating, at a vehicle, an estimated pitch and an estimated roll of the vehicle from inertial measurement unit data;

capturing, at an image capture system of the vehicle, road surface image data of a road surface;

detecting, using a dynamic model that accounts for distortion in the road surface image data due to pitch and roll of the vehicle, a set of lane markers based on the road surface image data, the estimated pitch, and the estimated roll;

generating a local map based on the set of detected lane markers;

wherein generating the local map comprises identifying a lane demarcated by the set of detected lane markers; and

controlling the vehicle according to the local map;

wherein controlling the vehicle comprises keeping the vehicle within the lane during vehicular travel.

2. The method of claim 1, wherein the dynamic model further accounts for changes between frames of the road surface image data due to a vehicle speed and a vehicle wheel angle and further detects the set of lane markers based on the vehicle speed and the vehicle wheel angle.

3. The method of claim 2, wherein detecting the set of lane markers further comprises detecting the set of lane markers using first and second binary feature detectors operating on pixels of the road surface image data;

wherein the first binary feature detector operates on an intensity difference between two pixels separated by a first pixel width;

wherein the second binary feature detector operates on an intensity difference between two pixels separated by a second pixel width;

wherein the second pixel width is greater than the first pixel width.

4. A method for image-based vehicle localization comprising:

measuring, at a vehicle, a position and a heading of the vehicle;

capturing, at an image capture system of the vehicle, road surface image data of a road surface;

processing the road surface image data to correct for distortion in the road surface image data due to pitch and roll of the vehicle, resulting in processed road surface image data;

performing feature detection on the processed road surface image data to detect lane markers on the road surface;

generating a local map based on the detected lane markers;

wherein generating the local map comprises identifying a lane demarcated by the detected lane markers; and

controlling the vehicle according to the local map;

wherein controlling the vehicle comprises keeping the vehicle within the lane during vehicular travel.

5. The method of claim 4, further comprising retrieving pre-existing map data based on the position and heading of the vehicle.

6. The method of claim 5, wherein the pre-existing map data includes a set of pre-existing road surface images, linked to a set of known locations;

wherein measuring the position and the heading of the vehicle comprises generating a position estimate using a GNSS receiver, generating a comparison of the road surface image data to the set of pre-existing road surface images, and refining the position estimate based on the comparison and the linked set of known locations.

7. The method of claim 5, wherein performing feature detection further comprises generating expected lane marker locations from the set of pre-existing road surface images and using the expected lane marker locations to aid in detection of lane markers on the road surface.

8. The method of claim 4, wherein identifying a lane demarcated by the detected lane markers comprises computing a set of estimated lane boundaries, each of the set generated from road surface image data captured at different times, and identifying the lane from an average of the set of estimated lane boundaries.

9. The method of claim 4, wherein both of processing the road surface image data and performing feature detection are performed using a dynamic model;

wherein processing the road surface image data and performing feature detection comprises;

at the dynamic model, taking as input the road surface image data, an estimated pitch of the vehicle, and an estimated roll of the vehicle; and

detecting the lane markers based on the road surface image data, the estimated pitch, and the estimated roll.

10. The method of claim 9, further comprising generating the estimated pitch and the estimated roll using an inertial measurement unit coupled to the vehicle.

11. The method of claim 9, wherein processing the road surface image data and performing feature detection further comprises at the dynamic model, taking as input a vehicle speed and a vehicle wheel angle;

wherein detecting the lane markers comprises generating an expected time distortion contribution based on the vehicle speed and vehicle wheel angle and detecting the lane markers based on the road surface image data, the estimated pitch, the estimated roll, and the expected time distortion contribution.

12. The method of claim 9, wherein detecting the lane markers further comprises detecting the lane markers using multiple binary feature detectors operating on pixels of the road surface image data.

13. The method of claim 12, wherein a first binary feature detector of the multiple binary feature detectors returns a true value for a first pixel if an intensity difference between the first pixel and a second pixel is greater than a first threshold value and a false value for the first pixel otherwise;

wherein the first pixel and second pixel are separated by a transformed lane marker width;

wherein the method further comprises generating the transformed lane marker width from an expected lane marker width and the location of the first pixel within the road surface image data.

14. The method of claim 13, wherein a second binary feature detector of the multiple binary feature detectors returns a true value for the first pixel if an intensity difference between the first pixel and a third pixel is greater than a second threshold value and a false value for the first pixel otherwise;

wherein the first pixel and third pixel are separated by a doubled transformed lane marker width;

wherein the method further comprises generating the doubled transformed lane marker width by doubling the transformed lane marker width.

15. The method of claim 14, wherein detecting the lane markers using multiple binary feature detectors comprises:

generating a first binary map using the first binary feature detector, generating a second binary map using the second binary feature detector, and combining the first and second binary maps.

16. The method of claim 14, wherein combining the first and second binary maps comprises merging the first and second binary maps using a region-growing technique;

wherein merging the first and second binary maps comprises including regions of true values of the second binary map if and only if the regions couple or overlap with regions of true values of the first binary map.

17. The method of claim 16, wherein the region-growing technique is initiated at pixels of the road surface image data corresponding to a smallest distance from the image capture system.

18. The method of claim 13, further comprising modifying the first threshold value based on environmental conditions of an environment external to the vehicle.

19. The method of claim 18, wherein modifying the first threshold value based on environmental conditions comprises scaling the first threshold value based on an average intensity of the road surface image data.

20. The method of claim 4, wherein generating the local map further comprises constraining local map generation according to known maximum allowed road curvature values.