Road vertical contour detection using a stabilized coordinate frame

• US 10,115,024 B2
• Filed: 02/26/2016
• Issued: 10/30/2018
• Est. Priority Date: 02/26/2015
• Status: Active Grant

• Current Assignee: MobilEye Vision Technologies Ltd. (Intel Corporation)
• Original Assignee: MobilEye Vision Technologies Ltd. (Intel Corporation)
• Inventors: Stein, Gideon, Blumenthal, Itay, Shaag, Nadav, Kingsley, Yaakov
• Primary Examiner(s): Elahi, Shan E
• Application Number:
  US15/055,322
• Publication Number:
  US 20160253566A1
• Time in Patent Office: 977 Days
• Field of Search: 348148
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/40 Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

In some embodiments, a first homography, created from two images of a roadway, is decomposed to determine an ego-motion, and the ego-motion is used to adjust a previous estimate of a road plane. The adjusted previous estimate of the road plane is combined with the current estimate of the plane to create a second homography, and the second homography is used to determine residual motion and vertical deviation in the surface of the roadway. In some embodiments, multiple road profiles each corresponding to a common portion of a roadway are adjusted in slope and offset by optimizing a function having a data term, a smoothness term and a regularization term; and the adjusted road profiles are combined into a multi-frame road profile. In some embodiments, road profile information for a predetermined number of data points is transmitted in periodic data bursts, with more than one data point per data burst.

Claims

1. A method for detecting a road surface deviation, the method performed by a driver assistance system mounted in a vehicle,

wherein the driver assistance system comprises:

a processor,

a camera operatively connectible to the processor, and

a memory storing instructions that can be executed by the processor,

the method comprising;

capturing multiple images via the camera, the multiple images comprising a first image of the road surface captured at a first time and a second image of the road surface captured at a second time, the second time being subsequent to the first time,

determining, by the processor, based on at least the first image and the second image, a first current estimate of a plane normal of the road surface,

creating, by the processor, a model of the road surface based on a combination of;

(i) a previous estimate of the plane normal of the road surface updated based on an ego-motion of the vehicle; and

(ii) the current estimate of the plane normal of the road surface based on the first image and the second image;

using, by the processor, the model of the road surface to determine a residual motion along a projected path of the vehicle,

computing, by the processor, a vertical deviation of the road surface based on the residual motion, and

transmitting, by the processor, vertical deviation data to a vehicle control system for controlling a motion of the vehicle.

2. The method of claim 1, wherein the model of the road surface is a homography.

3. The method of claim 1, wherein determining the first current estimate of a plane normal of the road surface comprises:

computing an original homography based on at least the first image of the road and the second image of the road; and

decomposing the original homography to determine the ego motion of the vehicle and the current estimate of the plane normal of the road surface.

4. The method of claim 3, wherein computing the original homography comprises tracking points between the first image and the second image, wherein the points are distributed in the images to correspond to locations on the roadway that are evenly spaced from one another.

5. The method of claim 3, wherein computing the original homography comprises tracking points between the first image and the second image, wherein variable weight is assigned to the points in accordance with their location in the image.

6. The method of claim 1, wherein the vehicle control system is configured to adjust a suspension system of the vehicle in response to the computer vertical deviation in the road surface.

7. The method of claim 1, wherein the model of the road surface is based on at least a historical average of the plane normal.

8. The method of claim 7, wherein the updated previous first estimate of the plane normal is weighted more heavily in the model of the road surface than the historical average of the plane normal.

9. The method of claim 1, wherein the first updated previous estimate of the plane normal is weighted more heavily in the model of the road surface than the second current estimate of the plane normal.

10. The method of claim 1, comprising:

computing a vehicle path in world coordinates; and

projecting the vehicle path onto the second image using stabilized plane parameters determined in accordance with the model of the road surface.

11. The method of claim 1, comprising:

correcting coordinates of points in the multiple images for at least one of radial distortion and rolling shutter.

12. The method of claim 11, wherein the correction for at least one of radial distortion and rolling shutter is performed after computing the model of the road surface.

13. A driver assistance system mounted in a vehicle, the system comprising:

a processor;

a camera operatively connectible to the processor; and

a memory storing instructions that, when executed by the processor, cause the system to;

capture multiple images via the camera, the multiple images comprising a first image of the road surface captured at a first time and a second image of the road surface captured at a second time, the second time being subsequent to the first time,

determine, by the processor, based on at least the first image and the second image, a current estimate of a plane normal of the road,

create, by the processor, a model of the road surface based on a combination of;

(i) a previous estimate of the plane normal of the road surface updated based on an ego-motion of the vehicle; and

(ii) the current estimate of the plane normal of the road surface based on the first image and the second image;

use, by the processor, the model of the road surface to determine a residual motion along a projected path of the vehicle,

compute, by the processor, a vertical deviation of the road surface based on the residual motion, and

transmit, by the processor, vertical deviation data to a vehicle control system for controlling a motion of the vehicle.

14. A non-transitory computer readable storage medium storing instructions that, when executed by a processor operatively coupled to a driver assistance system mounted in a host vehicle, cause the system to:

capture multiple images via a camera operatively connectible to the processor, the multiple images comprising a first image of the road surface captured at a first time and a second image of the road surface captured at a second time, the second time being subsequent to the first time;

determine, by the processor, based on at least the first image and the second image, a current estimate of a plane normal of the road;

create, by the processor, a model of the road surface based on a combination of;

(i) a previous estimate of the plane normal of the road surface updated based on an ego-motion of the vehicle; and

(ii) the current estimate of the plane normal of the road surface based on the first image and the second image;

use, by the processor, the model of the road surface to determine a residual motion along a projected path of the vehicle;

compute, by the processor, a vertical deviation of the road surface based on the residual motion; and

transmit, by the processor, vertical deviation data to a vehicle control system for controlling a motion of the vehicle.

15. A method for detecting a road surface deviation, the method performed by a driver assistance system mounted in a vehicle,

wherein the driver assistance system comprises:

a processor,

a camera operatively connectible to the processor, and

a memory storing instructions that can be executed by the processor,

the method comprising;

selecting, by the processor, at least two road profiles, wherein each of the at least two road profiles comprises a common portion of a road and wherein each of the at least two road profiles comprises one or more height values associated with the road;

optimizing, by the processor, a function comprising a data term, a smoothness term, and a regularization term to align a first profile from the at least two profiles; and

combining, by the processor, the at least two road profiles into a multi-frame profile.

16. The method of claim 15, wherein aligning the first profile from the at least two profiles comprises:

determining, based on the optimization, an optimized slope value and an optimized offset value;

using the optimized slope value and the optimized offset value to modify the first profile from the at least two road profiles.

17. The method of claim 15, wherein the at least two road profiles are generated using a stabilized world coordinate frame.

18. The method of claim 15, wherein combining the at least two road profiles comprises computing a median of the profiles.

19. A driver assistance system mounted in a vehicle, the system comprising:

a processor;

a camera operatively connectible to the processor; and

a memory storing instructions that, when executed by the processor, cause the system to;

select, by the processor, at least two road profiles, wherein each of the at least two road profiles comprises a common portion of a road and wherein each of the at least two road profiles comprises one or more height values associated with the road,

optimize, by the processor, a function comprising a data term, a smoothness term, and a regularization term to align a first profile from the at least two profiles, and

combine, by the processor, the at least two road profiles into a multi-frame profile.

20. A non-transitory computer readable storage medium storing instructions that, when executed by a processor operatively coupled to a driver assistance system mounted in a host vehicle, cause the system to:

select, by the processor, at least two road profiles, wherein each of the at least two road profiles comprises a common portion of a road, and wherein each of the at least two road profiles comprises one or more height values associated with the road;

optimize, by the processor, a function comprising a data term, a smoothness term, and a regularization term to align a first profile from the at least two profiles; and

combine, by the processor, the at least two road profiles into a multi-frame profile.