System and method for image based mapping, localization, and pose correction of a vehicle with landmark transform estimation

• US 9,443,309 B2
• Filed: 03/21/2016
• Issued: 09/13/2016
• Est. Priority Date: 02/08/2014
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Menashe, Jacob V., Murarka, Aniket, Ranganathan, Ananth
• Primary Examiner(s): ANSARI, TAHMINA N
• Application Number:
  US15/076,164
• Publication Number:
  US 20160203373A1
• Time in Patent Office: 176 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G01S 5/163 Determination of attitude using inertial means G01C9/00; control of attitude G05D1/49
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30248 Vehicle exterior or interior
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 3/20 Linear translation of whole images or parts thereof, e.g. panning
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 30/2504 Coarse or fine approaches, e.g. resolution of ambiguities or multiscale approaches

Abstract

A system and method for mapping, localization and pose correction including, determining a current position of a vehicle along a travel route and a set of currently observable landmarks along the travel route relative to the current position, the set of currently observable landmarks extracted from one or more stereo images obtained from an imaging device, and querying a survey landmark database to identify a subset of surveyed landmarks relative to the current position of the vehicle. The method including determining one or more two-dimensional transform estimates between the set of currently observable landmarks and the subset of surveyed landmarks and identifying a best transform estimate from the one or more two-dimensional transform estimates that minimizes distances between the set of currently observable landmarks and the subset of surveyed landmarks. The method including correcting a pose of the vehicle based on the best transform estimate.

Claims

1. A computer-implemented method for mapping, localization, and pose correction, comprising:

determining a current position of a vehicle along a travel route and a set of currently observable landmarks along the travel route relative to the current position, the set of currently observable landmarks extracted from one or more stereo images obtained from an imaging device;

receiving a subset of surveyed landmarks from a survey landmark database based on a query identifying the current position, the query executed at the survey landmark database, the survey landmark database including a set of surveyed landmarks previously stored;

determining one or more two-dimensional transform estimates between the set of currently observable landmarks and the subset of surveyed landmarks for one or more depth levels;

identifying a best transform estimate from the one or more two-dimensional transform estimates, wherein the best transform estimate minimizes distances between the set of currently observable landmarks and the subset of surveyed landmarks; and

correcting a pose of the vehicle based on the best transform estimate.

2. The computer-implemented method of claim 1, including determining a centroid of each landmark of the set of currently observable landmarks.

3. The computer-implemented method of claim 2, wherein the subset of surveyed landmarks stores a centroid for each landmark in the subset of surveyed landmarks.

4. The computer-implemented method of claim 3, wherein identifying the best transform estimate includes determining from the one or more two-dimensional transform estimates corresponding centroid distances between the set of currently observable landmarks and the subset of surveyed landmarks, and determining the best transform estimate as a two-dimensional transform estimate of the one or more two-dimensional transform estimates having the smallest centroid distances between the set of currently observable landmarks and the subset of surveyed landmarks.

5. The computer-implemented method of claim 3, wherein determining one or more two-dimensional transform estimates between the set of currently observable landmarks and the subset of surveyed landmarks includes determining one or more two-dimensional transform estimates at one or more transform scales for one or more depth levels.

6. The computer-implemented method of claim 1, including determining a transform cost of each of the one or more two-dimensional transform estimates, wherein the transform cost is a difference between corresponding centroids of the set of currently observable landmarks and the subset of surveyed landmarks.

7. The computer-implemented method of claim 6, determining a best transform estimate is based on the transform cost of each of the one or more two-dimensional transform estimates.

8. The computer-implemented method of claim 1, wherein identifying a best transform estimate from the one or more two-dimensional transform estimates includes identifying a best transform estimate from the one or more two-dimensional transform estimates for each of the one or more depth levels, wherein the one or more two-dimensional transform estimates are determined at one or more transform scales for the one or more depth levels.

9. The computer-implemented method of claim 8, including determining one or more clusters of best transform estimates based on a difference between each best transform estimate for each of the one or more depth levels.

10. A system for mapping, localization and pose correction, comprising:

a processor; and

a memory operatively connected for computer communication to the processor, the memory storing a visual odometry module, a localization module, and a pose correction module with instructions for execution by the processor, wherein;

the visual odometry module causes the processor to determine a current position of a vehicle along a travel route, wherein the visual odometry module causes the processor to determine a set of currently observable landmarks along the travel route relative to the current position, wherein the set of currently observable landmarks are extracted from one or more stereo images received from an imaging device;

the localization module causes the processor to execute a query including the current position at a survey landmark database, the survey landmark database including a set of surveyed landmarks previously stored, wherein the localization module receives a subset of surveyed landmarks from the survey landmark database as a result of the query;

the pose correction module causes the processor to determine one or more two-dimensional transform estimates between the set of currently observable landmarks and the subset of surveyed landmarks, identify a best transform estimate from the one or more two-dimensional transform estimates, wherein the best transform estimate is one of the one or more two-dimensional transform that minimizes distances between the set of currently observable landmarks and the subset of surveyed landmarks, and correct a pose of the vehicle based on the best transform estimate.

11. The system of claim 10, wherein the localization module causes the processor to determine a centroid of each landmark of the set of currently observable landmarks.

12. The system of claim 11, wherein the pose correction module causes the processor to identify the best transform estimate as a two-dimensional transform estimate of the one or more two-dimensional transform estimates that minimizes corresponding distances between the centroid of each landmark of the set of currently observable landmarks and a centroid of each landmark of the subset of surveyed landmarks.

13. The system of claim 10, wherein the visual odometry module causes the processor to determine the one or more two-dimensional transform estimates at one or more transform scales for one or more depth levels.

14. The system of claim 13, wherein the pose correction module causes the processor to identify the best transform estimate based on a comparison between a best transform estimate at each of the one or more depth levels.

15. The system of claim 10, wherein the pose correction module causes the processor to determine a transform cost for each of the two-dimensional transform estimates.

16. The system of claim 15, wherein the pose correction module causes the processor to identify the best transform estimate as one of the two-dimensional transform estimates having a lowest transform cost.

17. A non-transitory computer readable medium comprising instructions that when executed by a processor perform a method for mapping, localization and pose correction, comprising:

determining a current position of a vehicle along a travel route and a set of currently observable landmarks along the travel route relative to the current position, the set of currently observable landmarks extracted from one or more stereo images obtained from an imaging device;

receiving a subset of surveyed landmarks from a survey landmark database based on a query identifying the current position, the query executed at the survey landmark database, the survey landmark database includes a set of surveyed landmarks previously stored;

determining one or more two-dimensional transform estimates between the set of currently observable landmarks and the subset of surveyed landmarks;

identifying a best transform estimate from the one or more two-dimensional transform estimates that minimizes distances between the set of currently observable landmarks and the subset of surveyed landmarks; and

correcting a pose of the vehicle based on the best transform estimate.

18. The non-transitory computer readable medium of claim 17, comprising determining a centroid of each landmark of the set of currently observable landmarks and wherein the subset of surveyed landmarks comprises a centroid for each landmark in the subset of surveyed landmarks.

19. The non-transitory computer readable medium of claim 18, wherein the best transform estimate minimizes corresponding centroid distances between the set of currently observable landmarks and the subset of surveyed landmarks.

20. The non-transitory computer readable medium of claim 17, wherein determining one or more two-dimensional transform estimates between the set of currently observable landmarks and the subset of surveyed landmarks comprises determining one or more two-dimensional transform estimates at one or more transform scales for one or more depth levels and determining a best transform estimate based on a transform cost of each of the one or more transform estimates.