IMAGE PROCESSING METHOD FOR DETECTING OBJECTS USING RELATIVE MOTION

• US 20140169627A1
• Filed: 01/20/2012
• Published: 06/19/2014
• Est. Priority Date: 04/25/2011
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna International Incorporated
• Inventors: Gupta, Nikhil
• Granted Patent: US 9,547,795 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

An image based obstacle detection method. A camera mounted on a vehicle provides a set of image frames whilst the vehicle is in motion. The image frames define an image plane having a vertical aspect and a horizontal aspect. The relevancy of an object is determined by (i) selecting first and second feature points from the object that are spaced apart vertically in a first image frame; (ii) tracking the positions of the first and second feature points over at least a second image frame; and (iii) deciding the object to be relevant if the first and second feature points move dissimilar distances in physical space, within a tolerance, and deciding the object to be irrelevant otherwise. The motion of relevant objects is then estimated to determine if any relevant object is likely to become an obstacle to the vehicle.

Claims

1. :

An obstacle detection method, comprising;

(a) provisioning a vehicle with a camera having an external field of view;

(b) acquiring a set of image frames provided by the camera whilst the vehicle is in motion, the image frames defining an image plane having a vertical aspect and a horizontal aspect;

(c) determining one or more objects present in the set of images frames by edge detection;

(d) determining the relevancy of an object by

(i) selecting first and second feature points from the object contour that are spaced apart vertically in a first image frame,

(ii) tracking the positions of the first and second feature points over at least a second image frame, and

(iii) deciding the object to be relevant if the first and second feature points move dissimilar distances in physical space, within a tolerance, and deciding the object to be irrelevant otherwise;

(e) estimating the motion of relevant objects to determine if any relevant object is likely to become an obstacle to the vehicle.

2. :

A method according to claim 1, wherein step (d)(iii) is determined by reference to physical space and includes;

determining first and second physical locations along a selected physical horizontal plane corresponding to pixel positions of the first and second feature points in the first image frame;

determining third and fourth physical locations along the selected real horizontal plane corresponding to pixel positions of the first and second feature points in the second image frame;

determining a first distance Δ

d1 between the third and first physical locations and determining a second distance Δ

d2 between the fourth and second physical locations; and

deciding the object to be relevant if the difference between the first distance Δ

d1 and the second distance Δ

d2 exceeds a threshold, and otherwise deciding the object to be irrelevant.

3. :

A method according to claim 2, including unwarping a given pixel position to account for lens distortion prior to determining a physical location corresponding to the given pixel position.

4. :

A method according to claim 3, wherein the first and second feature points are portions of top and bottom edge contours of the object.

5. :

A method according to claim 1, wherein step (d)(iii) is determined by reference to the image plane and, where the first feature point is vertically lower than the second feature point in the method, includes;

estimating, from the first image frame, a physical distance D₁ of the first feature point O_1b to the camera in a selected physical horizontal plane;

estimating, from the first image frame, a physical location D_T of the second feature point O_1T assuming that the second feature point physically lies at a height T above the selected physical horizontal plane;

estimating, from the second image frame, a physical distance D₂ of the first feature point O_2b to the camera in the selected physical horizontal plane, and determining the distance AD between D₂ and D₁;

estimating a physical location D*_T which is equivalent to D_T+Δ

D, with D*_T being located at a height T above the selected physical horizontal plane;

computing a nominal feature point coordinate O*_2t in the image plane corresponding to D*_T;

determining the object to be relevant if the absolute distance between the nominal feature point coordinate O*_2t and the second feature point coordinate in the first image frame O_1T is less than the absolute distance between the second feature point coordinate in second image frame O_2T and the second feature point coordinate in the first image frame O_1T.

6. :

A method according to claim 5, wherein the selected physical horizontal plane is the ground plane.

7. :

A method according to claim 6, including unwarping a given pixel position to account for lens distortion prior to determining a physical location corresponding to the given pixel position.

8. :

A method according to claim 7, wherein the first and second feature points are portions of top and bottom edge contours of the object.

9. :

An obstacle detection method, comprising;

(a) provisioning a vehicle with a camera having an external field of view;

(b) acquiring a set of image frames in a video stream provided by the camera whilst the vehicle is in motion, the image frames defining an image plane having a vertical aspect and a horizontal aspect;

(c) determining one or more objects present in the set of images frames by edge detection;

(d) determining the relevancy of an object by

(i) selecting, in a first image frame, first and second feature points j and k from the object contour that are spaced apart vertically,

(ii) determining first and second physical locations along a selected real horizontal plane corresponding to the pixel positions of the first and second feature points in the first image frame,

(iii) tracking the pixel positions of the first and second feature points over at least a second image frame,

(iv) determining third and fourth physical locations along the selected real horizontal plane corresponding to the pixel positions of the first and second feature points in the second image frame,

(v) determining a first distance Δ

d1 between the third and first physical locations and determining a second distance Δ

d2 between the fourth and second physical locations, and

(vi) deciding the object to be relevant if the difference between the first distance Δ

d1 and the second distance Δ

d2 exceeds a threshold, and otherwise deciding the object to be irrelevant; and

(e) estimating the motion of relevant objects to determine if any relevant object is likely to become an obstacle to the vehicle.

10. :

An obstacle detection system for a vehicle, comprising;

a camera mounted to the vehicle, the camera providing a set of image frames from a video stream obtained whilst the vehicle is in motion, the image frames defining an image plane having a vertical aspect and a horizontal aspect;

a controller coupled to the camera, the controller configured to detect one or more objects in the video stream and determine the relevancy of an object by

(i) selecting first and second feature points on the object that are spaced apart vertically in a first image frame,

(ii) tracking the positions of the first and second feature points over at least a second image frame, and

(iii) deciding the object to be relevant if the first and second feature points move dissimilar distances in physical space, within a tolerance, and deciding the object to be irrelevant otherwise; and

the controller estimating the relative motion of relevant objects to determine if any relevant object is likely to become an obstacle to the vehicle; and

the controller providing an indication to the driver of detected obstacles.

11. :

An obstacle detection system according to claim 10, wherein the process of deciding objects to be relevant is determined by reference to physical space and the controller is configured to;

determine first and second physical locations along a selected physical horizontal plane corresponding to pixel positions of the first and second feature points in the first image frame;

determine third and fourth physical locations along the selected real horizontal plane corresponding to pixel positions of the first and second feature points in the second image frame;

determine a first distance Δ

d1 between the third and first physical locations and determining a second distance Δ

d2 between the fourth and second physical locations; and

decide the object to be relevant if the difference between the first distance Δ

d1 and the second distance Δ

d2 exceeds a threshold, and otherwise deciding the object to be irrelevant.

12. :

An obstacle detection system according to claim 11, wherein the controller de-warps a given pixel position to account for lens distortion prior to determining a physical location corresponding to the given pixel position.

13. :

An obstacle detection system according to claim 12, wherein the first and second feature points are portions of top and bottom edge contours of the object.

14. :

An obstacle detection system according to claim 11, wherein the process of deciding objects to be relevant is determined by reference to the image plane and, where the first feature point is vertically lower than the second feature point, the controller;

estimates, from the first image frame, a physical distance D₁ of the first feature point O_1b to the camera in a selected physical horizontal plane;

estimates, from the first image frame, a physical location D_T of the second feature point O_1T assuming that the second feature point physically lies at a height T above the selected physical horizontal plane;

estimates, from the second image frame, a physical distance D₂ of the first feature point O_2b to the camera in the selected physical horizontal plane, and determines the distance Δ

D between D₂ and D₁;

estimates a physical location D*_T which is equivalent to D_T+Δ

D, with D*_T being located at a height T above the selected physical horizontal plane;

computes a nominal feature point coordinate O*_2t in the image plane corresponding to D*_T;

determines the object to be relevant if the absolute distance between the nominal feature point coordinate O*_2t and the second feature point coordinate in the first image frame O_1T is less than the absolute distance between the second feature point coordinate in second image frame O_2T and the second feature point coordinate in the first image frame O_1T.

15. :

An obstacle detection system according to claim 14, wherein the selected physical horizontal plane is the ground plane.

16. :

An obstacle detection system according to claim 15, wherein the controller de-warps a given pixel position to account for lens distortion prior to determining a physical location corresponding to the given pixel position.

17. :

An obstacle detection system according to claim 16, wherein the first and second feature points are portions of top and bottom edge contours of the object.

18. :

An obstacle detection system for a vehicle, comprising;

a camera mounted to the vehicle, the camera providing a set of image frames from a video stream obtained whilst the vehicle is in motion, the image frames defining an image plane having a vertical aspect and a horizontal aspect;

a controller connected to the camera, the controller is operable to determine one or more objects present in the set of images frames by edge detection;

the controller determining the relevancy of an object by

(i) selecting, in a first image frame, first and second feature points j and k from the object contour that are spaced apart vertically,

(ii) determining first and second physical locations along a selected real horizontal plane corresponding to the pixel positions of the first and second feature points in the first image frame,

(iii) tracking the pixel positions of the first and second feature points over at least a second image frame,

(iv) determining third and fourth physical locations along the selected real horizontal plane corresponding to the pixel positions of the first and second feature points in the second image frame,

(v) determining a first distance Δ

d1 between the third and first physical locations and determining a second distance Δ

d2 between the fourth and second physical locations, and

(vi) deciding the object to be relevant if the difference between the first distance Δ

d1 and the second distance Δ

d2 exceeds a threshold, and otherwise deciding the object to be irrelevant;

the controller estimating the motion of relevant objects to determine if any relevant object is likely to become an obstacle to the vehicle; and

the controller providing an indication to the driver of detected obstacles.

19. :

An obstacle detection system according to claim 18, wherein the controller de-warps a given pixel position to account for lens distortion prior to determining a physical location corresponding to the given pixel position.

20. :

An obstacle detection system according to claim 18, wherein the first and second feature points are portions of top and bottom edge contours of the object.