Road marking detection and recognition

• US 9,053,372 B2
• Filed: 08/15/2012
• Issued: 06/09/2015
• Est. Priority Date: 06/28/2012
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Ranganathan, Ananth, Wu, Tao
• Primary Examiner(s): Alavi, Amir
• Assistant Examiner(s): Lesnick, Alexander J
• Application Number:
  US13/586,086
• Publication Number:
  US 20140003709A1
• Time in Patent Office: 1,028 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road

Abstract

A system and method are disclosed for detecting road marking in a video using learned road marking templates. The system comprises a template learning module configured to learn the feature-based road marking templates from a set of training images. The template learning module is configured to rectify each training image, detect multiple regions of interest, and for each detected region of interest, detect multiple key points. The template learning module extracts feature vectors for the detected key points and builds the road marking templates from the feature vectors. The system also includes a road marking detection module for detecting road markings in a video at runtime using the learned road marking templates. During runtime, these templates are matched using a two-step process of first selecting promising feature matches and subsequently performing a structural matching to account for the shape of the road markings.

Claims

1. A computer implemented method for detecting road markings in a video, the method comprising:

receiving a video comprising multiple digital representations of video frames,

wherein each video frame contains zero, one or more road markings;

for each video frame;

detecting one or more regions of interest on the video frame;

detecting one or more feature description key points in each detected region of interest;

extracting one or more feature vectors for each detected key point;

selecting one or more road marking candidates based on a comparison of the detected key points of the video frame and key points of a plurality of road marking templates;

detecting each road marking in the video frame based on the selected road marking candidates;

determining whether the video frame is qualified as a new road marking template based on detected road marking in the video frame, comprising;

determining amount of a road marking template of the plurality of the road marking template detected in the video frame based on the detected road marking in the video frame;

comparing a cost of adding the video frame as a road marking template with a threshold value; and

determining that the video frame is qualified as a road marking template responsive to the amount of the road marking template detected in the video frame exceeding a matching threshold and to the cost of adding the video frame exceeding a cost threshold; and

updating the plurality of the road marking templates by adding the video frame to the plurality of the road marking templates responsive to the video frame being qualified as a new road marking template.

2. The method of claim 1, further comprising:

rectifying each video frame of the video by compensating lens and perspective distortions of a camera that captures the video frame.

3. The method of claim 2, wherein rectifying a video frame comprises:

transforming the video frame to a corresponding image in a different viewpoint.

4. The method of claim 1, wherein detecting one or more regions of interest on the video frame comprises:

dividing the video frame into two sets of portions, wherein the pixels in the two sets of portions are different in pixel intensity; and

selecting the set of portions as the regions of interest based on rate of change of pixel intensity in the portions.

5. The method of claim 2, further comprising applying a maximally stable extremal regions analysis of the rectified video frame.

6. The method of claim 1, wherein extracting one or more feature vectors for detected key point comprises:

extracting the feature description key points using neighboring pixels of each key point.

7. The method of claim 1, wherein selecting one or more road marking candidates comprises:

measuring a distance between a set of detected key point of the video frame and a set of key points of each of a plurality of road marking templates based on the feature vectors of the key points; and

dividing the key points of a road marking template of the plurality of the road marking template into a matched subset and a non-matched subset based on the distance measure.

8. The method of claim 1, wherein detecting each road marking in the video frame based on the selected road marking candidates comprises:

comparing a shape formed by the selected key points of the video frame and a shape formed by the corresponding key points of one of the road marking template of the plurality of the road marking templates;

determining the types of road markings contained in the video frame based on the comparison of the shapes; and

computing a cost of selecting the shape formed by the selected key points of the video frame as a road marking contained in the video frame.

9. The method of claim 1, further comprising:

receiving a set of training images, each training image contains a type of road marking and identification of the road marking; and

learning a plurality of road marking templates from the set of training images.

10. The method of claim 9, wherein learning the plurality of road marking templates comprises:

for each training image;

rectifying the training image by compensating lens and distortion of a camera that captures the training image;

detecting one or more regions of interest on the training image;

detecting one or more feature description key points in each detected region of interest;

extracting one or more feature vectors for each detected key point;

generating the plurality of road marking templates from the extracted feature vectors of the detected key points; and

storing the plurality of road marking templates for runtime road marking detection.

13. The computer-readable storage-medium of claim 1, wherein the computer program code for rectifying a video frame comprises computer program code for:

transforming the video frame to a corresponding image in a different viewpoint.

15. The computer-readable storage-medium of claim 13, further comprising computer program code for applying a maximally stable extremal regions analysis of the rectified video frame.

11. A non-transitory computer-readable medium containing executable computer program code for detecting road markings in a video, the computer program code comprising code for:

receiving a video comprising multiple digital representations of video frames,

wherein each video frame contains zero, one or more road markings;

for each video frame;

detecting one or more regions of interest on the video frame;

detecting one or more feature description key points in each detected region of interest;

extracting one or more feature vectors for each detected key point;

selecting one or more road marking candidates based on a comparison of the detected key points of the video frame and key points of a plurality of road marking templates;

detecting each road marking in the video frame based on the selected road marking candidates;

determining whether the video frame is qualified as a new road marking template based on detected road marking in the video frame, comprising;

determining amount of a road marking template of the plurality of the road marking template detected in the video frame based on the detected road marking in the video frame;

comparing a cost of adding the video frame as a road marking template with a threshold value; and

determining that the video frame is qualified as a road marking template responsive to the amount of the road marking template detected in the video frame exceeding a matching threshold and to the cost of adding the video frame exceeding a cost threshold; and

updating the plurality of the road marking templates by adding the video frame to the plurality of the road marking templates responsive to the video frame being qualified as a new road marking template.

12. The computer-readable storage-medium of claim 11, further comprising computer program code for:

rectifying each video frame of the video by compensating lens and perspective distortions of a camera that captures the video frame.

14. The computer-readable storage-medium of claim 11, wherein the computer program code for detecting one or more regions of interest on the video frame comprises computer program code for:

dividing the video frame into two sets of portions, wherein the pixels of the two sets of portions are different in pixel intensity; and

selecting the set of portions as the regions of interest based on rate of change of pixel intensity in the portions.

16. The computer-readable storage-medium of claim 11, wherein the computer program code for extracting one or more feature vectors for detected key point comprises computer program code for:

extracting the feature description key points using neighboring pixels of each key point.

17. The computer-readable storage-medium of claim 11, wherein the computer program code for selecting one or more road marking candidates comprises computer program code for:

measuring a distance between a set of detected key point of the video frame and a set of key points of each of a plurality of road marking templates based on the feature vectors of the key points; and

dividing the key points of a road marking template of the plurality of the road marking template into a matched subset and a non-matched subset based on the distance measure.

18. The computer-readable storage-medium of claim 11, wherein the computer program code for detecting each road marking in the video frame based on the selected road marking candidates comprises computer program code for:

comparing a shape formed by the selected key points of the video frame and a shape formed by the corresponding key points of one of the road marking template of the plurality of the road marking templates;

determining the types of road markings contained in the video frame based on the comparison of the shapes; and

computing a cost of selecting the shape formed by the selected key points of the video frame as a road marking contained in the video frame.

19. The computer-readable storage-medium of claim 11, further comprising computer program code for:

receiving a set of training images, each training image contains a type of road marking and identification of the road marking; and

learning a plurality of road marking templates from the set of training images.

20. The computer-readable storage-medium of claim 19, wherein the computer program code for learning the plurality of road marking templates comprises computer program code for:

for each training image;

rectifying the training image by compensating lens and distortion of a camera that captures the training image;

detecting one or more regions of interest on the training image;

detecting one or more feature description key points in each detected region of interest;

extracting one or more feature vectors for each detected key point;

generating the plurality of road marking templates from the extracted feature vectors of the detected key points; and

storing the plurality of road marking templates for runtime road marking detection.

21. A computer system for detecting road markings in a video, the system comprising:

a road marking detection module configured to;

receive a video comprising multiple digital representations of video frames,

wherein each video frame contains zero, one or more road markings;

for each video frame;

detect one or more regions of interest on the video frame;

detect one or more feature description key points in each detected region of interest;

extract one or more feature vectors for each detected key point;

select one or more road marking candidates based on a comparison of the detected key points of the video frame and key points of a plurality of road marking templates;

detect each road marking in the video frame based on the selected road marking candidates;

determine whether the video frame is qualified as a new road marking template based on detected road marking in the video frame, comprising;

determine amount of a road marking template of the plurality of the road marking template detected in the video frame based on the detected road marking in the video frame;

compare a cost of adding the video frame as a road marking template with a threshold value; and

determine that the video frame is qualified as a road marking template responsive to the amount of the road marking template detected in the video frame exceeding a matching threshold and to the cost of adding the video frame exceeding a cost threshold; and

update the plurality of the road marking templates by adding the video frame to the plurality of the road marking templates responsive to the video frame being qualified as a new road marking template; and

a computer processor configured to execute the road marking detection module.

22. The computer system of claim 21, further comprising:

a template learning module configured to;

receive a set of training images, each training image contains a type of road marking and identification of the road marking; and

learn a plurality of road marking templates from the set of training images.

23. The computer system of claim 22, wherein the template learning module is further configured to:

for each training image;

rectify the training image by compensating lens and distortion of a camera that captures the training image;

detect one or more regions of interest on the training image;

detect one or more feature description key points in each detected region of interest;

extract one or more feature vectors for each detected key point;

generate the plurality of road marking templates from the extracted feature vectors of the detected key points; and

store the plurality of road marking templates for runtime road marking detection.