SIGN BASED LOCALIZATION

• US 20170046580A1
• Filed: 08/11/2015
• Published: 02/16/2017
• Est. Priority Date: 08/11/2015
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Lu, Yan, Murarka, Aniket, Fujimura, Kikuo, Ranganathan, Ananth
• Granted Patent: US 10,395,126 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/247 by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/582 of traffic signs
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control

Abstract

Systems and techniques for sign based localization are provided herein. Sign based localization may be achieved by capturing an image of an operating environment around a vehicle, extracting one or more text candidates from the image, detecting one or more line segments within the image, defining one or more quadrilateral candidates based on one or more of the text candidates, one or more of the line segments, and one or more intersections of respective line segments, determining one or more sign candidates for the image based on one or more of the quadrilateral candidates and one or more of the text candidates, matching one or more of the sign candidates against one or more reference images, and determining a location of the vehicle based on a match between one or more of the sign candidates and one or more of the reference images.

Claims

1. A system for sign based localization, comprising:

a sensor component mounted to a vehicle, the sensor component capturing an image of an operating environment around the vehicle;

an extraction component;

extracting one or more text candidates from the image;

detecting one or more line segments within the image; and

defining one or more quadrilateral candidates based on one or more of the text candidates, one or more of the line segments, and one or more intersections of respective line segments;

a candidate component determining one or more sign candidates for the image based on one or more of the quadrilateral candidates and one or more of the text candidates;

a communication component matching one or more of the sign candidates against one or more reference images of a localization database; and

a localization component determining a location of the vehicle based on a match between one or more of the sign candidates and one or more of the reference images.

2. The system of claim 1, wherein the extraction component extracts one or more of the text candidates based on a stroke width transform (SWT).

3. The system of claim 1, wherein the candidate component filters one or more of the quadrilateral candidates based on a ground plane.

4. The system of claim 1, wherein the extraction component extracts one or more scale-invariant feature transform (SIFT) features from one or more of the sign candidates.

5. The system of claim 1, wherein the candidate component filters one or more of the sign candidates based on a number of scale-invariant feature transform (SIFT) features associated with respective sign candidates.

6. The system of claim 1, wherein the extraction component filters one or more of the line segments based on a line segment length criteria or a threshold proximity to text candidate distance.

7. The system of claim 1, wherein the candidate component filters one or more line segments based on a vertical tolerance range or a horizontal tolerance range.

8. The system of claim 1, wherein the extraction component performs line segment linking based on two or more collinear line segments having endpoints which are less than a threshold proximity away from one another.

9. The system of claim 1, wherein the candidate component filters one or more of the quadrilateral candidates based on a text candidate area to quadrilateral area threshold ratio.

10. The system of claim 1, wherein the candidate component filters one or more of the sign candidates based on a number of maximum stable extreme regions (MSER) features associated with respective sign candidates.

11. The system of claim 1, wherein the communication component matches one or more of the sign candidates against one or more of the reference images based on putative scale-invariant feature transform (SIFT) matching.

12. The system of claim 1, wherein the communication component removes one or more outliers from the matching based on homography based random sample consensus (RANSAC).

13. The system of claim 1, wherein the localization component determines the location of the vehicle using a perspective-n-point (PnP) solver module.

14. The system of claim 1, comprising a navigation component generating one or more navigation instructions based on the location of the vehicle.

15. A method for sign based localization, comprising:

capturing an image of an operating environment around a vehicle;

extracting one or more text candidates from the image;

detecting one or more line segments within the image;

defining one or more quadrilateral candidates based on one or more of the text candidates, one or more of the line segments, and one or more intersections of respective line segments;

determining one or more sign candidates for the image based on one or more of the quadrilateral candidates and one or more of the text candidates;

matching one or more of the sign candidates against one or more reference images; and

determining a location of the vehicle based on a match between one or more of the sign candidates and one or more of the reference images.

16. The method of claim 15, comprising extracting one or more of the text candidates based on a stroke width transform.

17. The method of claim 15, comprising:

extracting one or more scale-invariant feature transform (SIFT) features from one or more of the sign candidates; and

filtering one or more of the sign candidates based on a number of SIFT features associated with respective sign candidates.

18. The method of claim 15, comprising filtering one or more of the line segments based on a line segment length criteria or a threshold proximity to text candidate distance.

19. The method of claim 15, comprising filtering one or more of the sign candidates based on a number of maximum stable extreme regions (MSER) features associated with respective sign candidates.

20. A system for sign based localization, comprising:

a sensor component mounted to a vehicle, the sensor component capturing an image of an operating environment around the vehicle;

an extraction component;

extracting one or more text candidates from the image based on a stroke width transform (SWT);

detecting one or more line segments within the image; and

defining one or more quadrilateral candidates based on one or more of the text candidates, one or more of the line segments, and one or more intersections of respective line segments;

a candidate component determining one or more sign candidates for the image based on one or more of the quadrilateral candidates and one or more of the text candidates;

a communication component matching one or more of the sign candidates against one or more reference images of a localization database based on putative scale-invariant feature transform (SIFT) matching; and

a localization component determining a location of the vehicle based on a match between one or more of the sign candidates and one or more of the reference images.