Dense structure from motion

• US 9,959,595 B2
• Filed: 02/18/2014
• Issued: 05/01/2018
• Est. Priority Date: 09/21/2010
• Status: Active Grant

• Current Assignee: MobilEye Vision Technologies Ltd. (Intel Corporation)
• Original Assignee: MobilEye Vision Technologies Ltd. (Intel Corporation)
• Inventors: Livyatan, Harel, Berberian, Oded, Cohen, Barak, Stein, Gideon
• Primary Examiner(s): KALAPODAS, DRAMOS
• Application Number:
  US14/182,715
• Publication Number:
  US 20140161323A1
• Time in Patent Office: 1,533 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road

Abstract

Determining three-dimensional structure in a road environment using a system mountable in a host vehicle including a camera connectable to a processor. Multiple image frames are captured in the field of view of the camera. In the image frames, a line is selected below which the road is imaged. The line separates between upper images essentially excluding images of the road and lower images essentially including images of the road. One or more of the lower images is warped, according to a road homography to produce at least one warped lower image. The three-dimensional structure may be provided from motion of a matching feature within the upper images or from motion of a matching feature within at least one of the lower images and at least one warped lower image.

Claims

1. A method for determining a three-dimensional structure in an environment, the method performed by a system mountable in a host vehicle, wherein the system includes a camera operatively connectable to a processor, the method comprising:

capturing a plurality of image frames in a field of view of the camera, wherein the plurality of image frames includes a first image frame and a second image frame;

identifying a road of the environment in the first image frame;

in the first image frame, selecting a line, wherein the line separates a first portion of the first image frame determined to include the road of the environment and a second portion of the first image frame determined to exclude the road of the environment;

detecting a feature in the first portion of the first image frame;

matching the feature in the first portion of the first image frame with image data in the second image frame to produce a first matching;

warping the first portion of the first image frame according to a road homography and ego-motion of the host vehicle to produce a warped portion of the image frame, wherein the ego-motion is based on at least one of speed, inertia, and steering angle of the host vehicle;

detecting the feature in the warped portion of the first image frame;

matching the feature in the warped portion of the first image frame with the image data in the second image frame to produce a second matching; and

selecting from among the first matching and the second matching to produce a selected matching.

2. The method of claim 1, further comprising:

providing a three-dimensional structure based on a selection of the image data in the second image frame.

3. The method of claim 2, wherein providing the three-dimensional structure based on the selection of the image data in the second image frame comprises:

computing image motion disparity between the feature and the image data in the second image frame; and

providing the three-dimensional structure from the image motion disparity.

4. The method of claim 1, further comprising:

providing a disparity map based on the warped portion of the first image frame and the image data in the second image frame;

from the disparity map, determining which image points are images of object points positioned on the road; and

selecting an image disparity for said images of object points from matching features of the warped portion of the first image frame.

5. The method of claim 1, wherein warping the first portion of the first image frame comprises:

pre-warping the first portion of the first image frame according to a homography corresponding to a vertical plane perpendicular to the direction of travel at a distance Z from the vehicle, wherein the distance Z is selectably either a fixed value or dependent on vehicle motion.

6. The method of claim 1, wherein warping the first portion of the first image frame comprises:

pre-warping the first portion of the first image frame according to respective homographies corresponding to two vertical planes parallel to the direction of travel at a fixed distance to the left and right of the camera.

7. The method of claim 1, wherein capturing a plurality of image frames in the field of view of the camera comprises:

capturing at least three image frames in the field of view of the camera of the road environment, the at least three image frames comprising;

a first image frame captured at a time A,

a second image frame captured at a time B, and

a third image captured frame at a time C,

wherein a time interval (C-A) is significantly greater than a time interval (B-A), wherein an epipole is a stationary point on image transformation between the at least three image frames;

for an image feature near the epipole, computing image motion between the third image frame captured at the time C and the first image frame captured at the time A;

for an image feature far from the epipole, computing image motion from the second image frame captured at the time B and the first image frame captured at the time A.

8. The method of claim 1, further comprising:

reducing image resolution of the image frames by performing a local histogram equalization around a matching feature within the image frames;

providing the three-dimensional structure from motion of said matching feature within the image frames of reduced resolution.

9. The method of claim 8, further comprising:

performing a rank transform on the image frames, wherein the rank transform replaces a gray-scale of a picture element with a count of pixels in a local neighborhood of the picture element which a lower gray-scale value than the picture element.

14. The method of claim 1, further comprising:

estimating the road homography responsive to time varying values of speed and yaw rate of the host vehicle.

15. The method of claim 1, wherein matching the feature in the first portion of the first image frame with image data in the second image frame to produce a first matching comprises:

matching a plurality of features at substantially different distances in image space from an epipole.

16. The method of claim 1, wherein matching the feature in the warped portion of the first image frame with the image data in the second image frame to produce a second matching comprises:

finding a correspondence between the image data in the second image frame and the warped portion of the first image frame using a patch of N pixels by N pixels.

18. The method of claim 1, wherein the feature is a point, line, curve, closed shape, open shape, or any combination thereof.

10. A system for determining three-dimensional structure in a road environment, the system, mountable in a host vehicle, including a camera operatively connectible to a processor, the processor configured to:

capture a plurality of image frames in a field of view of the camera, wherein the plurality of image frames includes a first image frame and a second image frame;

identify a road of the environment in the first image frame;

in the first image frame, select a line, wherein the line separates a first portion of the first image frame determined to include the road of the environment and a second portion of the first image frame determined to exclude the road of the environment;

detect a feature in the first portion of the first image frame;

match the feature in the first portion of the first image frame with image data in the second image frame to produce a first matching;

warp the first portion of the first image frame according to a road homography and ego-motion of the host vehicle to produce a warped portion of the image frame, wherein the ego-motion is based on at least one of speed, inertia, and steering angle of the host vehicle;

detect the feature in the warped portion of the first image frame;

match the feature in the warped portion of the first image frame with the image data in the second image frame to produce a second matching; and

select from among the first matching and the second matching to produce a selected matching.

11. The system of claim 10, wherein the processor is further configured to:

provide a three-dimensional structure based on a selection of the image data in the second image frame.

12. The system of claim 11, wherein providing the three-dimensional structure based on the selected matching comprises:

computing image motion disparity between the feature and the image data in the second image frame; and

;

providing the three-dimensional structure from the image motion disparity.

13. The system of claim 10, wherein the road homography is estimated responsive to time varying values of speed and yaw rate of the host vehicle and default values of camera height and road plane normal.

17. The system of claim 10, wherein matching the feature in the warped portion of the first image frame with the image data in the second image frame to produce a second matching comprises:

finding a correspondence between the image data in the second image frame and the warped portion of the first image frame using a patch of N pixels by N pixels.

19. The system of claim 10, wherein the feature is a point, line, curve, closed shape, open shape, or any combination thereof.

20. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device, cause the electronic device to:

receive a previous image of a road environment and a current image of the road environment;

identify a road of the road environment in the previous image;

identify a first portion of the previous image including the road of the road environment and a second portion of the previous image excluding the road of the road environment;

warping the first portion of the previous image according to a homography of the road and ego-motion of a host vehicle to provide a warped portion of the previous image, wherein the ego-motion is based on at least one of speed, inertia, and steering angle of the host vehicle;

matching a feature between the first portion of the previous image and a first portion of the current image to provide a first matching;

matching the feature between the warped portion of the previous image and the first portion of the current image to provide a second matching; and

providing a three-dimensional structure based on at least one of the first matching or the second matching.