Systems and methods for identifying traffic control devices and testing the retroreflectivity of the same

• US 9,767,371 B2
• Filed: 03/27/2015
• Issued: 09/19/2017
• Est. Priority Date: 03/27/2014
• Status: Active Grant

• Current Assignee: Georgia Tech Research Corporation (University System of Georgia)
• Original Assignee: Georgia Tech Research Corporation (University System of Georgia)
• Inventors: Ai, Chengbo, Tsai, Yichang, Wang, Zhaohua, Yezzi, Anthony Joseph
• Primary Examiner(s): Seth, Manav
• Application Number:
  US15/129,655
• Publication Number:
  US 20170193312A1
• Time in Patent Office: 907 Days
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G06F 13/16 for access to memory bus G06F13/28 takes precedence
• CPC Class Codes: G06F 13/20 for access to input/output bus
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/26 Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/582 of traffic signs
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning
• CPC Class Codes: G08G 1/00 Traffic control systems for road vehicles arrangement of road signs or traffic signals E01F9/00 ; automatic vehicle control B62D
• CPC Class Codes: G09F 2009/3055 for traffic signs

Abstract

Systems and methods for identifying traffic control devices from images, and systems and methods for assessing the retro reflectivity of traffic control devices. The identification of traffic control devices can be accomplished using a lighting-dependent statistical color model. The identification of traffic control devices can be accomplished using an active contour or active polygon method.

Claims

1. A method for identifying a traffic sign comprising:

classifying an image as having a lighting condition;

segmenting the image into a color used for traffic signs using a statistical color model specific to the lighting condition; and

detecting a shape in the image corresponding to the traffic sign.

2. The method of claim 1, wherein the color used for traffic signs is a MUTCD standard color.

3. The method of claim 1, wherein classifying the image further comprises classifying the image as underexposed where a mean pixel brightness value of the image is below an under-saturation threshold.

4. The method of claim 1, wherein classifying the image further comprises classifying the image as overexposed where a mean pixel brightness value of the image is above an over-saturation threshold.

5. The method of claim 1, wherein classifying the image further comprises classifying the image as adverse lighting if the difference between a mean pixel brightness of the image and a median pixel brightness of the image is over an adverse lighting threshold.

6. The method of claim 1, wherein classifying an image further comprises classifying the image as normal if:

a mean pixel brightness value of the image is above an under-saturation threshold;

a mean pixel brightness value of the image is below an over-saturation threshold; and

a difference between a mean pixel brightness of the image and median pixel brightness of the image is below an adverse lighting threshold.

7. The method of claim 5, wherein images having a lighting condition of adverse lighting are divided into a region having an over-exposed condition, and a region having an under-exposed condition, by:

generating a threshold surface;

comparing the threshold surface to the image to create a thresholded image;

identifying candidate regions of the image; and

applying a morphological open and close operation to the candidate regions of the image.

8. The method of claim 7, wherein generating a threshold surface is accomplished using an anti-geometric heat equation.

9. The method of claim 1, wherein segmenting an image further comprises:

calculating, for a plurality of pixels, a local pixel-level homogeneity value for one of a hue, saturation, and value;

normalizing the local pixel-level homogenity value; and

generating a probability distribution by applying an artificial neural network specific to a lighting condition, having input values of hue, saturation, value, and one or more of hue homogeneity, saturation homogeneity, and value homogeneity.

10. The method of claim 9, wherein the artificial neural network is a functional link network.

11. The method of claim 1, wherein the detecting step is performed by an differential equation based shape detection algorithm.

12. The method of claim 11, wherein the differential equation based shape detection algorithm comprises a region-based energy function.

13. The method of claim 11, wherein the differential equation based shape detection algorithm comprises an active contour algorithm.

14. The method of claim 13, wherein the active contour function comprises a probability distribution function sub-energy component that represents the probability of a sign image occurring in each pixel.

15. The method of claim 13, wherein the active contour function comprises a statistical color model sub-energy component represents the probability of a traffic sign color occurring in each pixel of the image.

16. The method of claim 13, wherein the active contour function comprises a global contour length sub-energy component with a maximum contour length.

17. The method of claim 16, wherein the maximum contour length is calculated as a function of a total perimeter of the image.

18. The method of claim 11, wherein the differential equation based shape detection algorithm comprises an active polygon algorithm.

19. The method of claim 18, wherein the active polygon contour algorithm comprises a generalized Hough transform.

20. The method of claim 19, wherein the generalized Hough transform comprises:

calculating an R-table corresponding to the shape of a traffic sign;

detecting the center where the maximum similarity is obtained compared to the R-table; and

solving the region-based energy function for the optimal value.

34. A method of evaluating a traffic sign comprising:

identifying the traffic sign by the method of claim 1; and

assessing a retroreflectivity of the traffic sign by the method of claim 21.

21. A method of assessing the retroreflectivity condition of a traffic sign comprising:

receiving, at a processor and from a LiDAR sensor, a plurality of LiDAR data points, each LiDAR data point in the plurality of LiDAR data points relating to a location on the face of the traffic sign, each LiDAR data point comprising 3D position information and a set of retro-intensity data, wherein each set of retro-intensity data comprises;

a retro-intensity value;

a distance value; and

an angle value;

determining, for each LiDAR data point, an incidence angle value;

receiving a plurality of image data points, wherein each image data point represents a portion of a traffic sign image, each image data point comprising;

color data; and

2D location data representing a location on the face of the traffic sign;

associating each of a plurality of LiDAR data points with a corresponding image data point, wherein 2D location data of a particular image data point corresponds to a location on the face of the traffic sign from which a particular LiDAR data point associated with the particular image data point relates;

grouping each LiDAR data point into one or more color clusters based on the associated color data,

normalizing, for each color cluster of LiDAR data points, each retro-intensity value based on the corresponding distance value and incidence angle value; and

determining, for each color cluster of LiDAR data points, whether the normalized retro-intensity values indicate a retroreflectivity above a predetermined threshold.

22. The method of claim 21, wherein the 3D position information comprises latitude data, longitude data and elevation data.

23. The method of claim 21, wherein each retro-intensity value represents a ratio of energy redirected from the traffic sign to the energy emitted from the LiDAR sensor.

24. The method of claim 21, wherein the distance value is a value that is representative of the distance between the traffic sign and the LiDAR sensor at the time of the measurement of the LiDAR data point.

25. The method of claim 21, wherein the angle value represents a LiDAR beam angle with respect to the level of the LiDAR sensor.

26. The method of claim 21, wherein the portion of the traffic sign image comprises a pixel.

27. The method of claim 21, wherein the color data represents the color of the portion of the traffic sign image.

28. The method of claim 21, wherein the 2D location data represents the location of the portion of the traffic sign image on a face of the traffic sign.

29. The method of claim 21, wherein determining whether the normalized retro-intensity values indicate a retroreflectivity above a predetermined threshold based on the color comprises:

determining a median value the normalized retro-intensity values for a color cluster of LiDAR data points; and

comparing the median value to a predetermined threshold associated with the color of the color cluster of the median value.

30. The method of claim 21, wherein the predetermined threshold based on the color represents the minimum acceptable level of retroreflectivity for a particular color.

31. The method of claim 30, wherein the particular color is a MUTCD color.

32. The method of claim 21 further comprising determining whether the condition of the traffic sign meets a minimum standard of retroreflectivity.

33. The method of claim 21, wherein the incidence angle value is determined from the direction of a LiDAR beam from the LiDAR sensor relative to the normal direction of a face of the traffic sign.

35. A system for identifying a traffic sign comprising:

at least one memory operatively coupled to at least one processor and configured for storing data and instructions that, when executed by the at least one processor, cause the system to;

classify an image as having a lighting condition;

segment the image into a color used for traffic signs using a statistical color model specific to the lighting condition; and

detect a shape in the image corresponding to the traffic sign.

36. The system of claim 35, wherein the color used for traffic signs is a MUTCD standard color;

wherein classifying the image further comprises classifying the image as underexposed where a mean pixel brightness value of the image is below an under-saturation threshold;

wherein classifying the image further comprises classifying the image as overexposed where a mean pixel brightness value of the image is above an over-saturation threshold;

wherein classifying the image further comprises classifying the image as adverse lighting if the difference between a mean pixel brightness of the image and a median pixel brightness of the image is over an adverse lighting threshold;

wherein classifying an image further comprises classifying the image as normal if;

a mean pixel brightness value of the image is above an under-saturation threshold;

a mean pixel brightness value of the image is below an over-saturation threshold; and

a difference between a mean pixel brightness of the image and median pixel brightness of the image is below an adverse lighting threshold;

wherein images having a lighting condition of adverse lighting are divided into a region having an over-exposed condition, and a region having an under-exposed condition, by;

generating a threshold surface;

comparing the threshold surface to the image to create a thresholded image;

identifying candidate regions of the image; and

applying a morphological open and close operation to the candidate regions of the image;

wherein generating a threshold surface is accomplished using an anti-geometric heat equation;

wherein segmenting an image further comprises;

calculating, for a plurality of pixels, a local pixel-level homogeneity value for one of a hue, saturation, and value;

normalizing the local pixel-level homogenity value; and

generating a probability distribution by applying an artificial neural network specific to a lighting condition, having input values of hue, saturation, value, and one or more of hue homogeneity, saturation homogeneity, and value homogeneity;

wherein the artificial neural network is a functional link network; and

wherein the detecting step is performed by an differential equation based shape detection algorithm.

37. The system of claim 36, wherein the differential equation based shape detection algorithm comprises one or more of:

a region-based energy function;

an active contour algorithm; and

an active polygon algorithm.

38. The system of claim 36, wherein the differential equation based shape detection algorithm comprises an active polygon algorithm;

and

wherein the active polygon contour algorithm comprises a generalized Hough transform.

39. The system of claim 38, wherein the generalized Hough transform comprises:

calculating an R-table corresponding to the shape of a traffic sign;

detecting the center where the maximum similarity is obtained compared to the R-table; and

solving the region-based energy function for the optimal value.

45. A system for evaluating a traffic sign comprising:

identifying the traffic sign by the system of claim 35; and

assessing a retroreflectivity of the traffic sign by the system of claim 40.

40. A system for assessing the retroreflectivity condition of a traffic sign comprising:

at least one memory operatively coupled to at least one processor and configured for storing data and instructions that, when executed by the at least one process, cause the system to;

receive a plurality of LiDAR data points obtained from a LiDAR sensor, each LiDAR data point in the plurality of LiDAR data points relating to a location on the face of the traffic sign, each LiDAR data point comprising 3D position information and a set of retro-intensity data, wherein each set of retro-intensity data comprises;

a retro-intensity value;

a distance value; and

an angle value;

determine, for each LiDAR data point, an incidence angle value;

receive a plurality of image data points, wherein each image data point represents a portion of a traffic sign image, each image data point comprising;

color data; and

2D location data representing a location on the face of the traffic sign;

associate each LiDAR data point with an image data point corresponding to a 2D location on the traffic sign, wherein each 2D location represents the location on the face of the traffic sign from which each respective LiDAR point was obtained;

group each of a plurality of LiDAR data points with a corresponding image data point, wherein 2D location data of a particular image data point corresponds to a location on the face of the traffic sign from which a particular LiDAR data point associated with the particular image data point relates;

normalize for each color cluster of LiDAR data points, each retro-intensity value based on the corresponding distance value and incidence angle value; and

determine, for each color cluster of LiDAR data points, whether the normalized retro-intensity values indicate a retroreflectivity above a predetermined threshold.

41. The system of claim 40, wherein the 3D position information comprises latitude data, longitude data and elevation data.

42. The system of claim 40, wherein each retro-intensity value represents a ratio of energy redirected from the traffic sign to the energy emitted from the LiDAR sensor;

wherein the distance value is a value that is representative of the distance between the traffic sign and the LiDAR sensor at the time of the measurement of the LiDAR data point;

where the angle value represents a LiDAR beam angle with respect to the level of the LiDAR sensor;

wherein the portion of the traffic sign image comprises a pixel;

wherein the color data represents the color of the portion of the traffic sign image;

wherein the 2D location data represents the location of the portion of the traffic sign image on a face of the traffic sign;

wherein determining whether the normalized retro-intensity values indicate a retroreflectivity above a predetermined threshold based on the color comprises;

determining a median value the normalized retro-intensity values for a color cluster of LiDAR data points; and

comparing the median value to a predetermined threshold associated with the color of the color cluster of the median value; and

wherein the predetermined threshold based on the color represents the minimum acceptable level of retroreflectivity for a particular color.

43. The system of claim 40 further comprising determining whether the condition of the traffic sign meets a minimum standard of retroreflectivity.

44. The system of claim 40, wherein the incidence angle value is determined from the direction of a LiDAR beam from the LiDAR sensor relative to the normal direction of a face of the traffic sign.