Device and method for recognizing traffic signs

• US 6,801,638 B1
• Filed: 05/14/2001
• Issued: 10/05/2004
• Est. Priority Date: 11/14/1998
• Status: Expired due to Fees

• Current Assignee: Daimler AG (Mercedes-Benz Group AG)
• Original Assignee: Daimler Chrysler Company LLC (Stellantis NV)
• Inventors: Lindner, Frank, Janssen, Reinhard, Ulmer, Berthold
• Primary Examiner(s): Patel, Jayanti K.
• Assistant Examiner(s): Tabatabai, Abolfazl
• Application Number:
  US09/831,803
• Time in Patent Office: 1,240 Days
• Field of Search: 382/100, 382/104, 382/107, 382/154, 382/199, 340/901, 340/902, 340/903, 340/904, 340/905, 340/906, 340/907, 340/938, 340/988, 340/438, 340/439, 340/990, 701/29, 701/36, 701/40, 701/41, 701/44, 701/70, 701/93, 701/96, 701/98, 701/116, 701/117, 701/225, 348/113, 348/149
• US Class Current: 382/104
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60W 2050/0088 Adaptive recalibration
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 40/06 Road conditions
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/582 of traffic signs
• CPC Class Codes: G06V 30/248 involving plural approaches, e.g. verification by template match; Resolving confusion among similar patterns, e.g. "O" versus "Q" G06V30/242 takes precedence

Abstract

The invention concerns a process and a device, which make it possible to reliably, with the least amount of computing power, recognize traffic signs and to display these as memory aids for an observer. Image data of an image sensor are analyzed and classified in an information processing unit, and on the basis of the results classification image segments and/or symbolic image representing data stored in a memory unit are recorded in a memory unit and are displayed by means of a display unit. In the further processing of the image data, extracted objects are separated and classified into traffic sign specific upper classes and lower classes. Subsequently, a synthetic image of a traffic sign is generated, stored in a memory unit, and displayed by means of a display unit.

Claims

1. A process for recognizing and displaying traffic signs, comprising:

analyzing and classifying image data from an image sensor in an information processing unit,

on the basis of the results of the classification, storing image segments and/or a symbolic image representing image data in a memory unit, and

displaying this by means of a display unit,

wherein in the framework of the analysis in the information processing unit it is determined whether the image data from the image sensor contains one or more objects which, with sufficient confidence, are traffic signs, and subsequently supplies these objects to a further processing and classification comprising;

separating the image data of the objects, in the data processing, into traffic sign specific upper classes and lower classes,

extracting, during the separation, appropriate class,specific characterizing data,

supplying the class-specific characterizing data to separate classifications,

replacing the class-specific characterizing data, if they have been correctly recognized with high confidence by the classification system, by appropriate image data representing the symbolic representation stored in a memory unit, or by the appropriate original image data originating from the image sensor, and

synthesizing the image data resulting from this replacement into a synthetic image of a traffic sign, storing this image in the memory unit, and bringing it to display in the display unit.

2. A process according to claim 1, wherein the classification occurs hierarchically, in multiple classification steps.

3. A process according to claim 1, wherein the classification steps are essentially set up as follows:

a) first, only the characterizing data associated with the upper class of the object is supplied to a classifier,

aa) upon successful classification, wherein the class is recognized correctly with a high degree of confidence, replacing the characterizing data with the appropriate symbolic representation of the upper class stored in the memory unit,

ab) upon unsuccessful classification, wherein the class could not be correctly recognized with a high degree of confidence, replacing the characterizing data for the upper class and the charactering data for the lower class with the corresponding original image data originating from the image sensor,

b) if the classification of the upper class was successful, then subsequently the characterizing data associated with the lower class of the object are supplied to a classifier,

ba) upon successful classification, wherein the class is recognized with a high degree of confidence, the characterizing data are replaced by the appropriate symbolic representation of the lower class stored in the memory unit,

bb) upon unsuccessful classification, wherein the class could not be correctly recognized with a high degree of confidence, the characterizing data for the lower class are substituted by the original image data from the image sensor.

4. A process according to claim 1, wherein the classifiers employed in the classification are trained using multiple learning samples.

5. A process according to claim 1, wherein the classifiers used in the classification make decisions on the basis of a comparison of the distance of the characterizing data supplied to them from the class-typical characterizing data sets stored in the memory unit.

6. A process according to claim 1, wherein, in the case that in the framework of the analysis of the image data from the image sensor a supplemental sign is recognized attached to the traffic sign, and when during the classification at least the upper class of the traffic sign is recognized with high confidence, this supplemental sign is displayed on the display unit.

7. A process according to claim 1, wherein in the case that in the framework of the classification at least the upper class of a traffic sign has been recognized, the respective image data which originated from the area of the environment of the traffic sign in which a supplemental sign is typically located is displayed by means of the display unit.

8. A process according to claim 7, wherein the display of the image data is subjected to a pre-classification, which rejects obviously useless image information and does not forward this to the display.

9. A process according to claim 1, wherein in the framework of the analysis in the information processing unit, during which it is determined whether an image contains one or more objects which with sufficiently high confidence are traffic signs, this is divided into two process steps, wherein

a) in a first process step image areas are determined which contain traffic signs with an above-average degree of confidence, and

b) in a second process step objects possessing a predetermined shape are determined within these image areas by means of geometric analysis.

10. A process according to claim 9, wherein said first process step is carried out on the basis of color values and/or spatial positions, and wherein said second process step is carried out by means of a correlation processes.

11. Process according to claim 9, wherein said shape is round or triangular.

12. A process according to claim 1, wherein said system is carried onboard a vehicle,

wherein a sequence of images is recorded of a recognized object as a vehicle drives once past the traffic sign, from which sequence respectively the best image is selected,

wherein the selection is carried out on the basis of defined quality criteria, and only the synthesized traffic sign image generated on the basis of this best image is stored in the memory unit and displayed on the display unit.

13. A process according to claim 12, wherein said defined quality criteria include size, contrast, and brightness of the object recognized to be a traffic sign.

14. A process according to claim 1, wherein during the combination of the image data into a synthesized image of a traffic sign, the respective image data, which corresponds to the image data originally originating from the image sensor, are processed for improvement with respect to a better recognizability, in such a manner that they are standardized with respect to at least one of:

a) size,

b) contrast,

c) intensity, and

d) color shade.

15. A process according to claim 1, further comprising determining the position of one of the objects recognized to be a traffic sign by means of a position determining unit.

16. A process according to claim 1, wherein the display unit is so controlled by the information processing unit via a program for using a decision logic stored in the memory unit, that it displays the synthesized image of a traffic sign only until a new object is recognized as a traffic sign, at which time the simplified representation of the new traffic sign replaces the previous image.

17. A process according to claim 1, wherein the maximal duration of the display of the synthesized image of a traffic sign is adjusted by means of an adjustment unit included in the display unit.

18. A process according to claim 1, wherein the system is carried onboard a vehicle, wherein the traveled distance between the actual vehicle position and the position of the last recognized traffic sign of a particular class of traffic signs is determined via a program in the memory unit and with the aid of a device or sensor coupled to the information unit.

19. A process according to claim 18, wherein the determination of the distance occurs by means of a vehicle tachometer coupled to the information unit or a wheel sensor for determining the traveled distance and a clock signal producer.

20. A process according to claim 18, wherein the data transmission of the devices coupled to the information unit and the sensors for determining the traveled distance occur via a vehicle on-board data bus.

21. A process according to claim 18, wherein, via a program stored in the memory unit, a decision logic is controlled via which the information processing unit controls the display unit in such a manner that it displays the synthesized image of the traffic sign so long, until a predetermined traveled path stored in the memory unit is traveled, which is preferably class-specific.

22. A device according to claim 18, wherein in the memory unit a program is stored for application of a decision logic by means of which the information processing unit so controls the display unit, that the synthesized image of a traffic sign is display so long, until a travel distance preset in the memory unit has been traveled, which is preferably class specific.

23. A process according to claim 1, further comprising controlling a decision logic via a program stored in the memory unit by means of which the information processing unit directs the display unit in such a manner, that it displays the synthesized image of a traffic sign so long, until a new object is recognized as a traffic sign, optionally including recognizing whether the a traffic sign is of the same or a corresponding class, and replaces the previous one with the simplified representation of the new one.

24. A process according to claim 1, further comprising emitting an acoustic signal via an acoustic signal emitter within the display unit as soon as an object is recognized as a traffic sign, wherein said acoustic signal is optionally class-specific.

25. A process according to claim 1, wherein said system is carried onboard a vehicle, wherein by a suitable arrangement of one or more digital cameras within the digital image sensor, all relevant traffic signs for a vehicle along a traveled path are detected independent of the horizontal and/or vertical tilting of the vehicle path.

26. A process according to claim 25, wherein said digital cameras are monochromic.

27. A process according to claim 1, wherein the system for recognizing and displaying traffic signs utilizes information which originated from at least one map-based navigation system and/or a traffic information system.

28. A process according to claim 27, wherein the system for recognizing and displaying traffic signs communicates with the map-based navigation system and/or traffic information system via an on-board vehicle data bus.

29. A process according to claim 27, wherein the system for traffic sign recognition begins to operate at a higher level of performance when on the basis of map-based information it is recognized that the vehicle is passing through an area in which the recognition of traffic signs may be problematic.

30. A process according to claim 29, wherein within said problematic areas supplemental subdivided recognition processes are carried out, in such a manner that

a) in the area of community entrances and community exits there is a specific search for signs indicating entry of a community and departing of a community,

b) in the area of traffic influencing facilities, changing signs or traffic lights, separately attention is paid to a change in the type and manner of the representation of the traffic sign,

c) in an area, in which poor visibility due to fog or rain is present and a higher probability of contrast-poor images is likely, an increase in the contrast enhancement of the image data is effected.

31. A process according to claim 27, wherein in a data memory unit integrated in the overall system, in addition to or complimentary to possibly already existing information, at least one of the type, the position, the condition, the visibility and the size of a recognized traffic sign is stored.

32. A process according to claim 31, wherein by means of a signal producer contained in the display unit, an acoustic or optical signal is emitted when a change with respect to the position or a change as to the presence or absence of a traffic sign occurs along a previously traveled road segment.

33. A process according to claim 27, wherein information from the map-based navigation system is drawn upon to determine the length of a road segment traveled by the vehicle.

34. A process according to claim 1, wherein a data memory unit is integrated in the overall system, in which, additionally or complimentary to possibly already existing information, at least one of the type, the position, the condition, the visibility and the size of a recognized traffic sign is recorded.

35. A process according to claim 34, wherein the display unit includes a signal emitter which emits an acoustic or optical signal when, while traveling along a path previously traveled at least once, a change with respect to the position or the presence or absence of a traffic sign occurs.

36. A process according to claim 1, wherein said system is carried onboard a vehicle, and wherein the information from the map-based navigation system is utilized in order to determine the distance of a road segment traveled by the vehicle.

37. A device for recognizing and displaying a traffic sign, comprising

a) an image sensor,

b) an information processing unit, comprised of an analyzer, a supplemental processing unit and a classifier for processing image data supplied by the image sensor,

c) a memory unit, which contains symbolic images representing image data, and in which image segments and/or symbolic images representing image data are stored by the information processing unit on the basis of the results of information supplied by the classifier, and

d) a display unit, upon which the image data stored in the memory unit can be brought to display,

wherein the analyzer is programmed to determine whether an image contains one or more objects which with high probability are traffic signs, and as a result thereof to supply this to the classifier,

wherein that the supplemental processing unit includes a separator, which separates the image data supplied to the analyzer into traffic sign specific upper classes and lower classes and extracts class specific characterizing data, which are subsequently supplied to a classifier,

wherein the classifier included in the information processing unit is constructed in such a manner, that the class-specific characterizing data, depending upon whether it has been recognized correctly by the classifier with high degree of confidence, is replaced by image data of representative symbolic images stored in the memory unit or the appropriate original image data originating from the image sensor, and

wherein the information processing unit includes an element for combination of image data, which combines the image data resulting from this replacement into a synthetic image of a traffic sign and stores this image in the memory unit, and by means of the display unit causes this to be displayed.

38. A device according to claim 37, wherein the classifier contained in the information processing unit is organized hierarchically, in multiple steps.

39. A device according to claim 38, wherein the hierarchical classifier contains a first classification step, which is carried out hierarchically, for identifying the upper class of the traffic sign, and contains a second classification step, which is organized hierarchically, for identifying the lower class of the traffic sign.

40. A device according to claim 37, wherein the classifier contained in the information unit is so designed, that it can be trained by multiple learning samples.

41. A device according to claim 37, wherein the classifier contained in the information unit is so designed, that it makes a decision on the basis of a comparison of distance measurements of (a) the characterizing data supplied to it and (b) the class typical characterizing data sets stored in the memory unit.

42. A device according to claim 37, wherein the display unit is so designed, that in the framework of the analysis of the image data from the image sensor a supplemental sign associated with the traffic sign can be displayed, when during the classification at least the upper class of the traffic sign is recognized with high probability.

43. A device according to claim 37, wherein the display unit is so designed, that in the case that in the framework of the classification at least the upper class of the traffic sign is recognized, the respective image data is displayed which originated from that area of the environment of the traffic sign in which supplemental signs are typically located.

44. A device according to claim 43, wherein in the display unit a classifier is provided, which rejects obviously useless image information and prevents the display thereof.

45. A device according to claim 37, wherein said device is carried onboard a vehicle, wherein the device additionally includes a comparator programmed to select the best image from a sequence of images taken as the vehicle moves past an object recognized as a traffic sign, wherein the comparator is programmed to carry out the selection on the basis of defined quality criteria, and causes the information processing unit to display the generated synthetic image of a traffic sign based only upon this best image, to store this in the memory unit, and to display this in the display unit.

46. A device as in claim 45, wherein said quality criteria is size and/or contrast and/or intensity of the object recognized as a traffic sign.

47. A device according to claim 37, wherein the unit for combining image data during the generation of a synthetic image of a traffic sign is programmed to process the respective image data which corresponds to the image data originally originating from the image sensor, processing these with respect to an improved recognizability, in such a manner that they are standardized in at least one of:

a) size,

b) intensity,

c) brightness, and

d) color tone.

48. A device according to claim 37, wherein the device additionally includes a position determining unit for determining the position of an object recognized as a traffic sign.

49. A device according to claim 37, wherein a program is stored in the memory unit for use as a decision logic by means of which the information processing unit controls the display unit in such a manner that the synthetic image of a traffic sign is displayed so long until a new object can be recognized as a traffic sign, and the simplified representation of the new object replaces the previous one.

50. A device according to claim 37, further including an adjustment unit associated with the display device, by means of which the maximal duration of the display of the synthetic image of the traffic sign can be adjusted.

51. A device according to claim 37, wherein said device is carried onboard a vehicle, wherein a program is stored in the memory unit, which with the aid of devices or sensors coupled to the information unit determines the distance covered between the actual position of the vehicle and the position of the last recognized traffic sign of a particular class of traffic signs.

52. A device according to claim 51, wherein for determining the traveled distance a vehicle tachometer or a wheel sensor for determining the covered distance and a time clock are coupled to the information unit.

53. A device according to claim 51, wherein the devices and sensors coupled to the information unit for determining the covered distance are coupled via an onboard vehicle data bus.

54. A device according to claim 37, wherein a program is stored in the memory unit for application of a decision logic by means of which the information processing unit controls the display unit in such manner, that it displays the synthetic image of the traffic sign so long, until a new object can be recognized as a traffic sign, optionally including determining whether the traffic sign is of the same or a corresponding class, and the simplified representation thereof replaces the preceding one.

55. A device according to claim 37, wherein the display unit additionally includes an acoustic signal emitter, which, as soon as an object has been recognized as a traffic sign, emits an acoustic signal, which may optionally be a class specific acoustic signal.

56. A device according to claim 37, wherein said device is carried onboard a vehicle, wherein the digital image sensor includes one or more digital cameras, which are arranged in such a manner, that their composite or total image field suffices to detect all of the traffic signs relevant to the traveled path of the independent of the horizontal and/or vertical tilting of the path.

57. A device according to claim 56, wherein said digital cameras are monochrome.

58. A device according to claim 37, wherein the system for recognizing and displaying traffic signs is connected to at least one of a map-based navigation system and a traffic information system, in order to utilize information originating therefrom.

59. A device according to claim 58, wherein a connection exists, via an onboard vehicle data bus, between the systems for recognizing and displaying of traffic signs the a map-based navigation system or traffic information system.

60. A device according to claim 58, wherein the system for traffic sign recognition is capability of operating at an increased performance when it is recognized, on the basis of map-based information, that the vehicle is passing through an area in which the recognition of traffic signs may be problematic.

61. A method for enhancing the safety of a vehicle in operation, said method comprising providing to the operator of said vehicle a memory aid, said memory aid, obtained by operating an onboard system designed to

analyze and classify image data from an image sensor in an information processing unit,

on the basis of the results of the classification, store image segments and/or a symbolic image representing image data in a memory unit, and

display this by means of a display unit,

wherein in the framework of the analysis in the information processing unit it is determined whether the image data from the image sensor contains one or more objects which, with sufficient confidence, are traffic signs, and subsequently supplies these objects to a further processing and classification comprising:

separating the image data of the objects, in the data processing, into traffic sign specific upper classes and lower classes,

extracting, during the separation, appropriate class-specific characterizing data,

supplying the class-specific characterizing data to separate classifications,

replacing the class-specific characterizing data, if they have been correctly recognized with high confidence by the classification system, by appropriate image data representing the symbolic representation stored in a memory unit, or by the appropriate original image, data originating from the image sensor, and

synthesizing the image data resulting from this replacement into a synthetic image of a traffic sign, storing this image in the memory unit, and bringing it to display in the display unit.

62. A method as in claim 61, wherein said method involves operating said system for inspection of regulated traffic routes.