Pavement marker recognition device, pavement marker recognition method and pavement marker recognition program

• US 9,123,242 B2
• Filed: 12/03/2008
• Issued: 09/01/2015
• Est. Priority Date: 12/05/2007
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Takahashi, Katsuhiko
• Primary Examiner(s): Ustaris, Joseph
• Assistant Examiner(s): Navas, Jr., Edemio
• Application Number:
  US12/746,407
• Publication Number:
  US 20100259609A1
• Time in Patent Office: 2,463 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06V 10/768 using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G08G 1/096716 where the received information does not generate an automatic action on the vehicle control
• CPC Class Codes: G08G 1/096791 where the origin of the information is another vehicle
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

A pavement marker recognition device for recognizing a pavement marker accurately is provided, the device being capable of recognizing an object even when it is difficult to be recognized only from tones or color information. A pavement marker candidate extraction unit extracts an image area having a feature similar to that of a pavement marker from an image captured by a car-mounted camera as a pavement marker candidate area, a road shape analysis unit analyzes the road shape from the image captured by the car-mounted camera, and a pavement marker verification unit determines whether the road shape analyzed by the road shape analysis unit and the pavement marker candidate area extracted by the pavement marker candidate extraction unit are in a correspondence relationship indicated by a road shape and pavement marker correspondence table, and based on the determination result, recognizes the pavement marker candidate area as a true pavement marker.

Claims

1. A pavement marker recognition device mounted on a vehicle for recognizing a plurality of types of pavement markers of a road based on an image obtained by capturing the surface of the road ahead of the vehicle, the pavement marker recognition device comprising:

a correspondence table data storage unit which stores, in advance, a road shape and pavement marker correspondence table showing correspondence relationships between the plurality of types of pavement markers and road shapes;

a car-mounted camera which captures simultaneously a road shape and a surface of the road on which the vehicle is actually traveling; and

a computer, wherein the computer operates;

a road shape analysis unit which is configured to analyze the road shape from an image of the shape of the road captured by the car-mounted camera;

a pavement marker candidate extraction unit which is configured to extract, in addition to an image area having a feature coincides with a feature of a pavement marker, an image area having a feature similar to a feature of the pavement marker although it is worn off from the image of the road shape captured by the car-mounted camera as a pavement marker candidate area; and

a pavement marker verification unit which is configured to determine whether or not the pavement marker candidate area extracted by the pavement marker candidate extraction unit and the road shape analyzed by the road shape analysis unit are in a correspondence relationship shown in the road shape and pavement marker correspondence table of the correspondence table data storage unit, and based on the determination result, recognize the pavement marker candidate area as a true pavement marker, wherein

the pavement marker candidate extraction unit is further configured to create a pavement marker recognition dictionary from a plurality of pavement marker patterns by a statistical learning method and extract an image area having the highest consistency with the pavement marker recognition dictionary from the image captured by the car-mounted camera.

2. The pavement marker recognition device, according to claim 1, wherein

the road shape analysis unit and the pavement marker candidate extraction unit configured to respectively use images captured from different angles,

the pavement marker recognition device includes a road shape storage unit configured to store information of the road shape analyzed by the road shape analysis unit, and

the pavement marker verification unit configured to acquire the information of the road shape by reading the information from the road shape storage unit.

3. The pavement marker recognition device, according to claim 2, further comprising a travel amount measurement unit configured to measure a travel amount of a vehicle provided with a car-mounted camera, wherein

the road shape analysis unit configured to modify the road shape information stored in the road shape storage unit based on the measured travel amount, and additionally configured to write information of the road shape analyzed from an image of the car-mounted camera.

4. The pavement marker recognition device, according to claim 3, wherein

the pavement marker candidate extraction unit configured to extract the pavement marker candidate area from a captured image by a car-mounted camera facing downward of back of the vehicle.

5. The pavement marker recognition device, according to claim 4, wherein

the road shape analysis unit configured to analyze a road shape from a captured image by a car-mounted camera of at least one of a car-mounted camera facing front of the vehicle, a car-mounted camera facing a side of the vehicle, and an all-round car-mounted camera.

6. The pavement marker recognition device, according to claim 1, further comprising a pavement marker recognition unit in place of the pavement marker candidate extraction unit and the pavement marker verification unit, wherein

the pavement marker recognition unit configured to identify a type of a pavement marker corresponding to a road shape analyzed by the road shape analysis unit based on the road shape and pavement marker correspondence table, and configured to recognize a pavement marker of the identified type from the image captured by the car-mounted camera.

7. The pavement marker recognition device, according to claim 1, wherein

the road shape and pavement marker correspondence table is information indicating at least one of road shapes of places where the pavement markers are present, road shapes of places where the pavement markers are absent, and both.

8. A pavement marker recognition method for recognizing a plurality of types of pavement markers of a road based on an image obtained by capturing the surface of the road ahead of a vehicle, the pavement marker recognition method comprising:

storing, in a correspondence table data storage unit, in advance, a road shape and pavement marker correspondence table showing correspondence relationships between the plurality of types of pavement markers and road shapes;

capturing, by a car-mounted camera, simultaneously a road shape and a surface of the road on which the vehicle is actually traveling;

analyzing, by a computer, in a road shape analyzing step, the road shape from an image of the shape of the road captured by the car-mounted camera;

extracting, by the computer, in a pavement marker candidate area extracting step, in addition to an image area having a feature coincides with a feature of a pavement marker, an image area having a feature similar to a feature of the pavement marker although it is worn off from the image of the road shape captured by the car-mounted camera as a pavement marker candidate area;

determining, by the computer, in a pavement marker verifying step, whether or not the pavement marker candidate area extracted in the pavement marker candidate area extracting step and the road shape analyzed in the road shape analyzing step are in a correspondence relationship shown in the road shape and pavement marker correspondence table of the correspondence table data storage unit, and based on the determination result, recognizing the pavement marker candidate area as a true pavement marker;

creating, by the computer, a pavement marker recognition dictionary from a plurality of pavement marker patterns by a statistical learning method; and

extracting, by the computer, an image area having the highest consistency with the pavement marker recognition dictionary from the image captured by the car-mounted camera.

9. The pavement marker recognition method, according to claim 8, wherein in the road shape analyzing step and the pavement marker candidate area extracting step, images captured from different angles are used respectively, and further comprising:

a road shape storing step for storing information of the road shape analyzed in the road shape analyzing step in the storage section is performed, and

in the pavement marker verifying step, the information of the road shape is read from the storage section.

10. The pavement marker recognition method, according to claim 8, further comprising:

before the road shape analyzing step, a travel amount measuring step for measuring a travel amount of the vehicle provided with the car-mounted camera, wherein

in the road shape analyzing step, the road shape information previously stored in the storage section is modified based on the measured travel amount, and information of the road shape analyzed from the car-mounted camera is additionally written in the storage section.

11. The pavement marker recognition method, according to claim 10, wherein

in the pavement marker candidate area extracting step, the pavement marker candidate area is extracted from a captured image by a car-mounted camera facing downward of back of the vehicle.

12. The pavement marker recognition method, according to claim 11, wherein

in the road shape analyzing step, the road shape is analyzed from a captured image of a car-mounted camera of at least one of a car-mounted camera facing front of the vehicle, a car-mounted camera facing a side of the vehicle, and an all-round car-mounted camera.

13. The pavement marker recognition method, according to claim 8, further comprising:

after the road shape analyzing step, a pavement marker recognizing step in place of the pavement marker candidate area extracting step and the pavement marker verifying step, wherein

in the pavement marker recognizing step, a type of a pavement marker corresponding to the road shape analyzed in the road shape analyzing step is identified based on the road shape and pavement marker correspondence table, and a pavement marker of the identified type is recognized from the image captured by the car-mounted camera.

14. A non-transitory computer readable recording medium storing a pavement marker recognition program for controlling recognition of a plurality of types of pavement markers of a road based on an image obtained by capturing the surface of the road ahead of the vehicle, the pavement marker recognition program causing a computer to perform:

a road shape analyzing function for analyzing the road shape from an image of the shape of the road captured by a car-mounted camera, wherein the car-mounted camera captures simultaneously a road shape and a surface of the road on which the vehicle is actually traveling;

a pavement marker candidate area extracting function for extracting, in addition to an image area having a feature coincides with a feature of a pavement marker, an image area having a feature similar to a feature of the pavement marker although it is worn off from the image of the road shape captured by the car-mounted camera as a pavement marker candidate area; and

a pavement marker verifying function for determining whether or not the pavement marker candidate area extracted by the pavement marker candidate area extracting function and the road shape analyzed by the road shape analyzing function are in a correspondence relationship shown in a road shape and pavement marker correspondence table of the correspondence table data storage unit, and based on the determination result, recognizing the pavement marker candidate area as a true pavement marker, wherein the road shape and pavement marker correspondence table is stored in advance in a correspondence table data storage unit and shows correspondence relationships between the plurality of types of pavement markers and road shapes, wherein

the pavement marker candidate area extracting function further creates a pavement marker recognition dictionary from a plurality of pavement marker patterns by a statistical learning method and extracts an image area having the highest consistency with the pavement marker recognition dictionary from the image captured by the car-mounted camera.

15. The non-transitory computer readable recording medium storing the pavement marker recognition program, according to claim 14, wherein

in the road shape analyzing function and the pavement marker candidate area extracting function, images captured from different angles are used respectively, and

in addition to a road shape storing function for storing information of the road shape analyzed by the road shape analyzing function in the storage section,

the program causes the computer to perform the pavement marker verifying function as a function of reading the information of the road shape is from the storage section, determining whether or not a road shape indicated by this information and the pavement marker candidate area extracted by a pavement marker candidate extraction unit is in a correspondence relationship shown in the road shape and pavement marker correspondence table, and based on a determination result, recognizing the pavement marker candidate area as a true pavement marker.

16. The non-transitory computer readable recording medium storing the pavement marker recognition program, according to claim 15, wherein

in addition to a travel amount measuring function for measuring a travel amount of the vehicle,

the program causes the computer to perform the road shape analyzing function as a function for modifying the road shape information previously stored in the storage section based on the measured travel amount, and additionally writing information of the road shape analyzed from an image captured by the car-mounted camera in the storage section.

17. The non-transitory computer readable recording medium storing the pavement marker recognition program, according to claim 16, wherein

the pavement marker candidate area extracting function is a function of extracting the pavement marker candidate area from a captured image by a car-mounted camera facing downward of back of the vehicle.

18. The non-transitory computer readable recording medium storing the pavement marker recognition program, according to claim 17, wherein

the road shape analyzing function is a function of analyzing the road shape from a captured image of a car-mounted camera of at least one of a car-mounted camera facing front of the vehicle, a car-mounted camera facing a side of the vehicle, and an all-round car-mounted camera.

19. The non-transitory computer readable recording medium storing the pavement marker recognition program, according to claim 14, wherein

in place of the pavement marker candidate area extracting function and the pavement marker verifying function, the program causes the computer to perform a pavement marker recognizing function of reading, from the storage section, and referring to the road shape and pavement marker correspondence table, identifying a type of a pavement marker corresponding to the road shape analyzed by the road shape analyzing function, and recognizing a pavement marker of the identified type from the image captured by the car-mounted camera.

20. The non-transitory computer readable recording medium storing the pavement marker recognition program, according to claim 14, wherein the road shape and pavement marker correspondence table is information indicating at least one of road shapes of places where the pavement markers are present, road shapes of places where the pavement markers are absent, and both.