ROAD MARKING DETECTION AND RECOGNITION

• US 20140003709A1
• Filed: 08/15/2012
• Published: 01/02/2014
• Est. Priority Date: 06/28/2012
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Ranganathan, Ananth, Wu, Tao
• Granted Patent: US 9,053,372 B2
• Time in Patent Office: Days
• US Class Current: 382/159
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road

Abstract

A system and method are disclosed for detecting road marking in a video using learned road marking templates. The system comprises a template learning module configured to learn the feature-based road marking templates from a set of training images. The template learning module is configured to rectify each training image, detect multiple regions of interest, and for each detected region of interest, detect multiple key points. The template learning module extracts feature vectors for the detected key points and builds the road marking templates from the feature vectors. The system also includes a road marking detection module for detecting road markings in a video at runtime using the learned road marking templates. During runtime, these templates are matched using a two-step process of first selecting promising feature matches and subsequently performing a structural matching to account for the shape of the road markings.

Claims

1. A computer implemented method for detecting road markings in a video, the method comprising:

receiving a video comprising multiple digital representations of video frames, wherein each video frame contains zero, one or more road markings;

for each video frame;

detecting one or more regions of interest on the video frame;

detecting one or more feature description key points in each detected region of interest;

extracting one or more feature vectors for each detected key point;

selecting one or more road marking candidates based on a comparison of the detected key points of the video frame and key points of a plurality of road marking templates;

identify each detected road marking in the video frame based on the selected road marking candidates; and

updating the plurality of the road marking templates.

2. The method of claim 1, further comprising:

rectifying each video frame of the video by compensating lens and perspective distortions of a camera that captures the video frame.

3. The method of claim 2, wherein rectifying a video frame comprises:

transforming the video frame to a corresponding image in a different viewpoint.

4. The method of claim 1, wherein detecting one or more regions of interest on the video frame comprises:

dividing the video frame into two sets of portions, wherein the pixels in the two sets of portions are different in pixel intensity; and

selecting the set of portions as the regions of interest based on rate of change of pixel intensity in the portions.

5. The method of claim 4, further comprising applying a maximally stable extremal regions analysis of the rectified video frame.

6. The method of claim 1, wherein extracting one or more feature vectors for detected key point comprises:

extracting the feature description key points using neighboring pixels of each key point.

7. The method of claim 1, wherein selecting one or more road marking candidates comprises:

measuring a distance between a set of detected key point of the video frame and a set of key points of each of a plurality of road marking templates based on the feature vectors of the key points; and

dividing the key points of a road marking template of the plurality of the road marking template into a matched subset and a non-matched subset based on the distance measure.

8. The method of claim 1, wherein identifying each detected road marking in the video frame based on the selected road marking candidates comprises:

comparing a shape formed by the selected key points of the video frame and a shape formed by the corresponding key points of one of the road marking template of the plurality of the road marking templates; and

determining the types of road markings contained in the video frame based on the comparison of the shapes.

9. The method of claim 8, further comprising:

computing a cost of selecting a road marking candidate as a part of a road marking contained in the video frame.

10. The method of claim 1, further comprising:

receiving a set of training images, each training image contains a type of road marking and identification of the road marking; and

learning a plurality of road marking templates from the set of training images.

11. The method of claim 10, wherein learning the plurality of road marking templates comprises:

for each training image;

rectifying the training image by compensating lens and distortion of a camera that captures the training image;

detecting one or more regions of interest on the training image;

detecting one or more feature description key points in each detected region of interest;

extracting one or more feature vectors for each detected key point;

generating the plurality of road marking templates from the extracted feature vectors of the detected key points; and

storing the plurality of road marking templates for runtime road marking detection.

14. The computer-readable storage-medium of claim 1, wherein the computer program code for rectifying a video frame comprises computer program code for:

transforming the video frame to a corresponding image in a different viewpoint.

12. A non-statutory computer-readable medium containing executable computer program code for detecting road markings in a video, the computer program code comprising code for:

receiving a video comprising multiple digital representations of video frames, wherein each video frame contains zero, one or more road markings;

for each video frame;

detecting one or more regions of interest on the video frame;

detecting one or more feature description key points in each detected region of interest;

extracting one or more feature vectors for each detected key point;

selecting one or more road marking candidates based on a comparison of the detected key points of the video frame and key points of a plurality of road marking templates;

identifying each detected road marking in the video frame based on the selected road marking candidates; and

updating the plurality of the road marking templates.

13. The computer-readable storage-medium of claim 12, further comprising computer program code for:

rectifying each video frame of the video by compensating lens and perspective distortions of a camera that captures the video frame.

15. The computer-readable storage-medium of claim 12, wherein the computer program code for detecting one or more regions of interest on the video frame comprises computer program code for:

dividing the video frame into two sets of portions, wherein the pixels of the two sets of portions are different in pixel intensity; and

selecting the set of portions as the regions of interest based on rate of change of pixel intensity in the portions.

16. The computer-readable storage-medium of claim 15, further comprising computer program code for applying a maximally stable extremal regions analysis of the rectified video frame.

17. The computer-readable storage-medium of claim 12, wherein the computer program code for extracting one or more feature vectors for detected key point comprises computer program code for:

extracting the feature description key points using neighboring pixels of each key point.

18. The computer-readable storage-medium of claim 12, wherein the computer program code for selecting one or more road marking candidates comprises computer program code for:

measuring a distance between a set of detected key point of the video frame and a set of key points of each of a plurality of road marking templates based on the feature vectors of the key points; and

dividing the key points of a road marking template of the plurality of the road marking template into a matched subset and a non-matched subset based on the distance measure.

19. The computer-readable storage-medium of claim 12, wherein the computer program code for identifying each detected road marking in the video frame based on the selected road marking candidates comprises computer program code for:

comparing a shape formed by the selected key points of the video frame and a shape formed by the corresponding key points of one of the road marking template of the plurality of the road marking templates; and

determining the types of road markings contained in the video frame based on the comparison of the shapes.

20. The computer-readable storage-medium of claim 12, further comprising computer program code for:

receiving a set of training images, each training image contains a type of road marking and identification of the road marking; and

learning a plurality of road marking templates from the set of training images.

21. The computer-readable storage-medium of claim 20, wherein the computer program code for learning the plurality of road marking templates comprises computer program code for:

for each training image;

rectifying the training image by compensating lens and distortion of a camera that captures the training image;

detecting one or more regions of interest on the training image;

detecting one or more feature description key points in each detected region of interest;

extracting one or more feature vectors for each detected key point;

generating the plurality of road marking templates from the extracted feature vectors of the detected key points; and

storing the plurality of road marking templates for runtime road marking detection.

22. A computer system for detecting road markings in a video, the system comprising:

a road marking detection module configured to;

receive a video comprising multiple digital representations of video frames, wherein each video frame contains zero, one or more road markings;

for each video frame;

detect one or more regions of interest on the video frame;

detect one or more feature description key points in each detected region of interest;

extract one or more feature vectors for each detected key point;

select one or more road marking candidates based on a comparison of the detected key points of the video frame and key points of a plurality of road marking templates;

identify each detected road marking in the video frame based on the selected road marking candidates;

update the plurality of the road marking templates; and

a computer processor configured to execute the road marking detection module.

23. The computer system of claim 22, further comprising:

a template learning module configured to;

receive a set of training images, each training image contains a type of road marking and identification of the road marking; and

learn a plurality of road marking templates from the set of training images.

24. The computer system of claim 23, wherein the template learning module is further configured to:

for each training image;

rectify the training image by compensating lens and distortion of a camera that captures the training image;

detect one or more regions of interest on the training image;

detect one or more feature description key points in each detected region of interest;

extract one or more feature vectors for each detected key point;

generate the plurality of road marking templates from the extracted feature vectors of the detected key points; and

store the plurality of road marking templates for runtime road marking detection.