Method and device for determining a roadway state by means of a vehicle camera system

• US 10,289,920 B2
• Filed: 10/28/2014
• Issued: 05/14/2019
• Est. Priority Date: 11/15/2013
• Status: Active Grant

• Current Assignee: Continental Teves AG & Co. OHG (Continental AG)
• Original Assignee: Continental Teves AG & Co. OHG (Continental AG)
• Inventors: Hartmann, Bernd
• Primary Examiner(s): Retallick, Kaitlin A
• Application Number:
  US14/903,444
• Publication Number:
  US 20160379065A1
• Time in Patent Office: 1,659 Days
• Field of Search: None
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road

Abstract

A method and a device are for determining the presence and type of a roadway coating by a vehicle camera system. The vehicle camera system records at least one image of the surroundings of the subject vehicle. The at least one image is evaluated to detect indications of the presence of a roadway coating including a characteristic feature of the roadway coating that results as the roadway is traveled over by the subject vehicle or by another vehicle. The detected indications and characteristic feature are taken into consideration in the determination of the presence and type of a roadway coating. The result of the determination of the roadway coating, or a friction coefficient estimation derived therefrom, is output to a driver assistance function, a vehicle control function or as information to the driver.

Claims

1. A method of determining a roadway coating on a roadway with a vehicle camera system of a subject vehicle, wherein the roadway coating is formed by extraneous media or objects lying on the roadway, wherein plural different types of roadway coatings respectively exhibit different characteristic features when driven over by the subject vehicle or another vehicle, wherein the plural different types of roadway coatings comprise snow, water, slush, ice, leaves, and sand, and wherein the method comprises:

with the vehicle camera system, recording camera images of surroundings of the subject vehicle including at least one camera image of the roadway coating on the roadway, which exhibits an actual characteristic feature resulting from the roadway coating being driven over by the subject vehicle or the another vehicle;

evaluating at least a partial region of the at least one camera image by image processing to detect, from only the at least one camera image, indications of a presence of the roadway coating including the actual characteristic feature of the roadway coating as the roadway is driven over by the subject vehicle or by the another vehicle;

determining the presence of the roadway coating on the roadway and making a conclusion about a type of the roadway coating among the plural different types of roadway coatings in consideration of and dependent on only the indications including the actual characteristic feature that were detected from only the at least one camera image, and

outputting a result of the determining of the presence and the conclusion about the type of the roadway coating.

2. The method according to claim 1, wherein the extraneous media which forms the roadway coating comprises precipitation lying on the roadway, and wherein the indications of the presence of the roadway coating detected from only the at least one camera image further comprise effects of the precipitation on the roadway, on the subject vehicle, on the another vehicle, or on a vehicle windshield of the subject vehicle or of the another vehicle.

3. The method according to claim 1, wherein the actual characteristic feature comprises effects on the roadway coating as it is traveled over by at least one tire of the subject vehicle or of the another vehicle, which effects are detected in the evaluating of the at least one camera image.

4. The method according to claim 1, further comprising determining at least one selected partial region in the at least one camera image, and limiting the evaluating of the at least one camera image to only the at least one selected partial region thereof so as to detect the indications of the presence of the roadway coating only from the at least one selected partial region of the camera image.

5. The method according to claim 1, wherein the making of the conclusion is performed by a classifier which maps the detected actual characteristic feature onto a plurality of classes characterized by the different characteristic features which are respectively assigned to the plural different types of roadway coatings.

6. The method according to claim 1, wherein the vehicle camera system has a rear and/or lateral detection area of the surroundings located behind and/or beside the subject vehicle so that the indications of the presence of the roadway coating including the actual characteristic feature exhibited by the roadway coating as the roadway is traveled over by the subject vehicle are detected in the evaluating of the at least one camera image.

7. The method according to claim 1, wherein the vehicle camera system has a forward detection area of the surroundings located in front of the subject vehicle so that the indications of the presence of the roadway coating including the actual characteristic feature exhibited by the roadway coating as the roadway is traveled over by the another vehicle are detected in the evaluating of the at least one camera image, wherein the another vehicle comprises a leading vehicle traveling in front of the subject vehicle, an oncoming vehicle traveling opposite the subject vehicle, or a crossing vehicle traveling on a roadway path that intersects a roadway path of the roadway on which the subject vehicle is traveling.

8. The method according to claim 1, wherein the actual characteristic feature of the roadway coating comprises whirled-up components of the roadway coating which are detectable at least in the partial region of the at least one camera image in the evaluating thereof.

9. The method according to claim 1, wherein the indications of the presence of the roadway coating detected from only the at least one camera image further comprise obstructed views in a field of view of the vehicle camera system because the roadway is being traveled over by the another vehicle, which comprises a leading vehicle traveling in front of the subject vehicle, an oncoming vehicle traveling opposite the subject vehicle, or a crossing vehicle traveling on a roadway path that intersects a roadway path of the roadway on which the subject vehicle is traveling.

10. The method according to claim 1, wherein the actual characteristic feature of the roadway coating comprises tire ruts in the roadway coating which are detectable at least in the partial region of the at least one camera image in the evaluating thereof.

11. The method according to claim 5, wherein the classifier comprises a neural network that has been trained to be able to assign the detected actual characteristic feature to a specific matching one of the different characteristic features of the plural different types of roadway coatings.

12. The method according to claim 1, further comprising performing a roadway state detection or classification of the roadway, combining the result of the determining of the presence of the roadway coating with a result of the roadway state detection or classification, and estimating a friction coefficient of the roadway from the combined results.

13. A device for determining a roadway coating on a roadway, wherein the roadway coating is formed by extraneous media or objects lying on the roadway, wherein plural different types of roadway coatings respectively exhibit different characteristic features when driven over by a subject vehicle or another vehicle, wherein the plural different types of roadway coatings comprise snow, water, slush, ice, leaves, and sand, and wherein the device comprises:

a vehicle camera system on the subject vehicle, which vehicle camera system is configured to record camera images of surroundings of the subject vehicle including at least one camera image of the roadway coating on the roadway, which exhibits an actual characteristic feature resulting from the roadway coating being driven over by the subject vehicle or the another vehicle,

an evaluating unit which is configured to evaluate at least a partial region of the at least one camera image by image processing to detect, from only the at least one camera image, indications of a presence of the roadway coating including the actual characteristic feature of the roadway coating as the roadway is driven over by the subject vehicle or by the another vehicle,

a determining unit which is configured to determine the presence of the roadway coating on the roadway and to make a conclusion about a type of the roadway coating among the plural different types of roadway coatings in consideration of and dependent on only the indications including the actual characteristic feature that were detected from only the at least one camera image, and

an output unit which is configured to output a result of the determining of the presence and the conclusion about the type of the roadway coating.

14. A method for a driver assistance system of a subject vehicle, comprising the following steps performed by the driver assistance system:

a) with a camera system of the subject vehicle recording surroundings of the subject vehicle and producing corresponding camera image data including at least one camera image of a roadway coating on a roadway, wherein the roadway coating is formed by extraneous media or objects lying on the roadway, and wherein the roadway coating exhibits a characteristic feature resulting from being driven over by the subject vehicle or another vehicle;

b) evaluating at least a partial region of the at least one camera image by image processing to detect, from only the at least one camera image, indications of a presence of the roadway coating including the characteristic feature of the roadway coating;

c) classifying the characteristic feature, which has been detected from only the at least one camera image, among pre-determined plural different characteristic features that are respectively exhibited by plural different types of roadway coatings as a result of being driven over, wherein the plural different types of roadway coatings comprise snow, water, slush, ice, leaves, and sand;

d) based on a result of the classifying, making a conclusion about a type of the roadway coating on the roadway among the plural different types of roadway coatings; and

e) outputting the conclusion about the type of the roadway coating.

15. The method according to claim 14, wherein the classifying of the characteristic feature and the making of the conclusion are based on only the camera image data, without consideration of temperature sensor data and without consideration of ultrasonic sensor data.

16. The method according to claim 14, wherein the plural different characteristic features comprise whirled-up or sprayed components of the extraneous media or objects that form the roadway coating, which whirled-up or sprayed components are whirled-up or sprayed into the air above the roadway when the roadway coating is driven over by the vehicle, wherein the whirled-up or sprayed components are selected from the group consisting of water spray, slush spray, snow spray, spray mist, whirled-up water, whirled-up snow, sprayed snow, whirled-up leaves, sprayed leaves, whirled-up sand, sprayed sand, whirled-up dust, and sprayed dust.

17. The method according to claim 14, wherein the camera system comprises a camera with a detection zone of the camera directed to a rear area behind the subject vehicle or a lateral area to a side of the subject vehicle, wherein the characteristic feature exhibited by the roadway coating is a characteristic feature resulting from being driven over by the subject vehicle, and wherein the plural different characteristic features respectively exhibited by the plural different roadway coatings are characteristic features resulting from being driven over by the subject vehicle.