Method for determining a state of a pavement from surroundings sensor data

• US 9,676,331 B2
• Filed: 12/09/2013
• Issued: 06/13/2017
• Est. Priority Date: 12/20/2012
• Status: Active Grant

• Current Assignee: Continental Teves AG & Co. OHG (Continental AG)
• Original Assignee: Continental Teves AG & Co. OHG (Continental AG)
• Inventors: Hartmann, Bernd
• Primary Examiner(s): GEROLEO, FRANCIS
• Application Number:
  US14/424,148
• Publication Number:
  US 20150224925A1
• Time in Patent Office: 1,282 Days
• Field of Search: 348148
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2520/28 Wheel speed
• CPC Class Codes: B60W 40/068 Road friction coefficient
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

For determining a state of a pavement from surroundings sensor data, local data received from at least one device that measures a local pavement state or coefficient of friction is merged with camera image data received from a camera for imaging a pavement extending in front of the vehicle. When analyzing the camera image data, the local data representing the measured pavement state or coefficient of friction is assigned to individual image sectors of a camera image while taking odometric and time information into account to achieve proper correspondence, and the local data is taken into account for supporting and/or plausibility-checking of an anticipatory estimation of the local coefficient of friction or pavement state based on the camera image data.

Claims

1. A method of determining a state of a pavement on which a vehicle is driving, comprising

merging locally measured data received from at least one locally measuring device that measures a local pavement state or coefficient of friction of the pavement, with camera data of a camera image received from a camera, and

performing an image analysis comprising analyzing the camera data, which involves:

sub-dividing the camera image into a two-dimensional grid of grid cells representing image sectors of the camera image in a plane of the pavement,

assigning the locally measured data representing the local pavement state or coefficient of friction respectively to individual ones of the image sectors of the camera image in the camera data while taking odometric information of the vehicle and time information into account, and

taking the locally measured data representing the local pavement state or coefficient of friction into account for support and/or plausibilization of an anticipatory and locally resolved coefficient-of-friction estimation or state-of-pavement determination based on the camera data.

2. The method according to claim 1, wherein the image analysis includes assigning the locally measured data representing the local pavement state or coefficient of friction to at least one pavement segment respectively represented in at least one of the image sectors in the camera image when the odometric information and the time information reveal that the locally measured data has been measured respectively at the at least one pavement segment.

3. The method according to claim 1, wherein the image analysis provides a classification of the individual image sectors in the camera image based on particular features of pavement segments depicted in the image sectors.

4. The method according to claim 1, wherein a total number of the grid cells of the grid is determined dependent on a homogeneity of the pavement.

5. The method according to claim 1, wherein a total number of the grid cells of the grid is determined dependent on a current driving situation and/or a criticality thereof.

6. The method according to claim 1, wherein a total number of the grid cells of the grid is determined dependent on an available computing power for performing the method.

7. The method according to claim 1, wherein a result of the image analysis of the camera data is predictively applied afterwards, whilst taking the local pavement state or coefficient of friction assigned to the camera image into account, to a subsequently acquired camera image.

8. The method according to claim 1, further comprising calculating a vehicle corridor from a predicted movement trajectory of the vehicle, by which vehicle corridor, positions of the at least one locally measuring device comprising individual wheels of the vehicle and/or at least one locally measuring sensor are predictively assigned to pavement segments respectively represented in the image sectors of the camera image, said pavement segments extending in front of the vehicle.

9. The method according to claim 1, further comprising assigning probability values to a respective one of the image sectors or a respective pavement segment represented in the respective image sector, said probability values indicating with what probability the respective image sector or the respective pavement segment is to be assigned to a first class and to at least a second class.

10. The method according to claim 1, wherein a mono camera is used as the camera.

11. The method according to claim 1, wherein a stereo camera is used as the camera.

12. The method according to claim 1, wherein a radar sensor, an ultrasonic sensor or an optical sensor is used as the locally measuring device, which is configured for locally determining a three-dimensional shape of a pavement surface of the pavement.

13. The method according to claim 1, wherein at least one measuring device that measures and/or derives a local coefficient of friction from speed signals of a vehicle wheel is used as the locally measuring device.

14. A device for determining a state of a pavement on which a vehicle is driving, comprising a camera that provides camera data of a camera image, at least one locally measuring device configured to measure a local pavement state or coefficient of friction of the pavement and to provide corresponding locally measured data, a merger device configured to merge the locally measured data with the camera data, and an image analysis device configured to perform an image analysis comprising analyzing the camera data, wherein, the image analysis device is further configured to sub-divide the camera image into a two-dimensional grid of grid cells representing image sectors of the camera image in a plane of the pavement, and to assign the locally measured data representing the local pavement state or coefficient of friction respectively to individual ones of the image sectors of the camera image in the camera data while taking odometric information of the vehicle and time information into account, and to take the locally measured data representing the local pavement state or coefficient of friction into account for support and/or plausibilization of an anticipatory and locally resolved coefficient-of-friction estimation or state-of-pavement determination based on the camera data.

15. A method of determining a surface condition of a driving surface on which a vehicle is driving, comprising steps:

a) with a camera on said vehicle, producing camera data including a camera image of a selected surface area of said driving surface ahead in front of said vehicle;

b) performing an image analysis and classification of said selected surface area in said camera image to determine, among predetermined classes, an estimated class of a surface condition comprising an estimated pavement state or an estimated coefficient of friction of said selected surface area;

c) driving said vehicle forward whereby said selected surface area comes into a sensing range of a locally measuring sensor on said vehicle, and using time information and odometric information of said vehicle to achieve sensing registration of said locally measuring sensor with said selected surface area;

d) with said locally measuring sensor, sensing locally measured data representing an actual surface condition comprising an actual pavement state or an actual coefficient of friction of said selected surface area;

e) merging said locally measured data representing said actual surface condition with said estimated class of said surface condition for said selected surface area, and thereby assigning a new value of said actual surface condition to said estimated class of said surface condition, or validating, plausibilizing or correcting a previously assigned value of said actual surface condition that had previously been assigned to said estimated class of said surface condition; and

f) repeating said steps a) to e), with regard to a subsequent selected surface area of said driving surface ahead in front of said vehicle.