System and method for image based vehicle localization

• US 9,727,793 B2
• Filed: 12/15/2015
• Issued: 08/08/2017
• Est. Priority Date: 12/15/2015
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Taalimi, Ali, Heisele, Bernd
• Primary Examiner(s): NAKHJAVAN, SHERVIN K
• Application Number:
  US14/970,251
• Publication Number:
  US 20170169300A1
• Time in Patent Office: 602 Days
• Field of Search: None
• CPC Class Codes: G01C 21/28 with correlation of data from several navigational instruments
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects

Abstract

Methods and systems for determining a vehicle location and pose are provided. A vehicle localization system may determine a location and pose estimate for the vehicle. The system may retrieve three-dimensional map data based on the location and pose estimate. The system may obtain a camera image from a vehicle mounted camera. The system may project the three-dimensional map data onto the camera image based on the location and pose estimate. The system may compare the projected three-dimensional map data to the camera image. The system may then determine the vehicle location and pose based on the comparing.

Claims

1. A method of determining a vehicle location and pose, comprising:

determining a location and pose estimate;

retrieving three-dimensional map data based on the location and pose estimate;

obtaining a camera image from a vehicle mounted camera;

projecting the three-dimensional map data onto the camera image based on the location and pose estimate;

comparing the projected three-dimensional map data to the camera image; and

determining the vehicle location and pose based on the comparing, wherein projecting the three-dimensional map data onto the camera image comprises;

transforming the three-dimensional map data from a map coordinate system to a vehicle coordinate system;

transforming the map data from the vehicle coordinate system to a camera coordinate system; and

projecting the three-dimensional map data from the camera coordinate system onto the camera image.

2. The method of claim 1, wherein transforming the map data from the vehicle coordinate system to a camera coordinate system is based on a camera calibration.

3. The method of claim 1, wherein comparing the projected three-dimensional map data to the camera image includes matching edges of the camera image with edges of the projected three-dimensional map data.

4. The method of claim 3, wherein the matching edges comprises:

performing edge detection on the camera image to determine a set of edges of the camera image; and

determining a shortest distance between each edge of the set of edges of the camera image and an edge of the projected map data.

5. The method of claim 4, wherein an orientation of the edge of the camera image is within a threshold angle of the edge of the projected map data.

6. The method of claim 1, further comprising:

generating multiple particles for the location and pose estimate of the vehicle, wherein comparing the projected three-dimensional map data to the camera image includes;

determining a matching metric for each particle; and

selecting a particle with a best matching metric.

7. The method of claim 1, wherein the matching metric is a chamfer distance metric.

8. The method of claim 1, further comprising applying a mask to one or both of the three-dimensional map data or the camera image, the mask reducing a number of edges for the comparing.

9. The method of claim 1, wherein projecting the three-dimensional map data onto the camera image includes:

determining a gradient of a line in the projected map data;

assigning the gradient to any pixels along the line; and

extending the line to an edge of the camera image using the gradient.

10. A system for determining a location and pose of a vehicle, comprising:

a global positioning system receiver configured to determine a location and pose estimate for the vehicle;

a map database configured to provide three-dimensional map data based on the location and pose estimate;

a vehicle mounted camera configured to obtain a camera image;

a processor communicatively coupled to a memory, the processor configured to;

project the three-dimensional map data onto the camera image based on the location and pose estimate;

compare the projected three-dimensional map data to the camera image; and

determine the vehicle location and pose based on the comparing, wherein to project the three-dimensional map data, the processor is further configured to;

transform the three-dimensional map data from a map coordinate system to a vehicle coordinate system;

transform the map data from the vehicle coordinate system to a camera coordinate system; and

project the three-dimensional map data from the camera coordinate system onto the camera image.

11. The system of claim 10, wherein the processor is further configured to transform the map data from the vehicle coordinate system to the camera coordinate system based on a camera calibration.

12. The system of claim 10, wherein the processor is further configured to compare the projected three-dimensional map data to the camera image by matching edges of the camera image with edges of the projected three-dimensional map data.

13. The system of claim 12, wherein processor is further configured to:

perform edge detection on the camera image to determine a set of edges of the camera image; and

determine a shortest distance between each edge of the set of edges of the camera image and an edge of the projected map data.

14. The system of claim 13, wherein the processor is further configured to determine that an orientation of the edge of the camera image is within a threshold angle of the edge of the projected map data.

15. The system of claim 10, wherein the processor is further configured to:

generate multiple particles for the location and pose estimate of the vehicle, determine a matching metric for each particle; and

selecting a particle with a best matching metric.

16. The system of claim 10, wherein the matching metric is a chamfer distance metric.

17. The method of claim 10, wherein the processor is further configured to apply a mask to one or both of the three-dimensional map data or the camera image, the mask reducing a number of edges for the comparing.

18. A non-transitory computer-readable medium storing computer executable code for determining a vehicle location and pose, comprising code for:

determining a location and pose estimate;

retrieving three-dimensional map data based on the location and pose estimate;

obtaining a camera image from a vehicle mounted camera;

projecting the three-dimensional map data onto the camera image based on the location and pose estimate;

comparing the projected three-dimensional map data to the camera image; and

determining the vehicle location and pose based on the comparing, wherein projecting the three-dimensional map data onto the camera image comprises;

transforming the three-dimensional map data from a map coordinate system to a vehicle coordinate system;

transforming the map data from the vehicle coordinate system to a camera coordinate system; and

projecting the three-dimensional map data from the camera coordinate system onto the camera image.