Motor vehicle obstacle monitoring system using optical flow processing

• US 5,521,633 A
• Filed: 09/24/1993
• Issued: 05/28/1996
• Est. Priority Date: 09/25/1992
• Status: Expired due to Term

• Current Assignee: Yazaki Corporation
• Original Assignee: Yazaki Corporation
• Inventors: Fujishiro, Takahiro, Oikawa, Takahiro, Kitamura, Norio, Sasaki, Kazuyuki, Nakajima, Masato, Ishii, Kouji
• Primary Examiner(s): Chin, Tommy
• Assistant Examiner(s): LIN, BRYAN T
• Application Number:
  US08/126,298
• Time in Patent Office: 977 Days
• Field of Search: 348/113, 348/114, 348/116, 348/118, 348/119, 348/148, 348/143, 348/149, 348/169, 348/172, 348/171, 348/170, 364/424.01, 364/424.02
• US Class Current: 348/118
• CPC Class Codes: B60G 2401/142 Visual Display Camera, e.g. LCD
• CPC Class Codes: B60Q 9/008 for anti-collision purposes
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2554/801 Lateral distance
• CPC Class Codes: G01S 11/12 using electromagnetic waves other than radio waves
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 2207/30248 Vehicle exterior or interior
• CPC Class Codes: G06T 7/223 using block-matching
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

A monitoring method for moving vehicles allows the presence of danger to be determined from the magnitude of an optical flow vector. The optical flow vector is detected as a movement of a single point on an object taken over two images, with one image taken at a preceding time and the other image taken at succeeding time in a series of images. A long and narrow window set in a radial direction from a focus of expansion (FOE) of the earlier image is moved in the same direction on the latter image. An optical flow vector of a target point is defined by a vector connecting the midpoint of a location of a subsequent window and the midpoint of a location at which the long and narrow window is set in the earlier image. The location of the subsequent window is determined by minimizing the sum of absolute values of differences in luminance between the long and narrow window and an area of the latter image overlapping such long and narrow window. The optical flow calculation area is limited to areas in which a difference in luminance between the earlier and latter images exceeds a predetermined threshold, and any influence from scenes outside the road boundaries and from lane-dividing lines or other symbols drawn on the road surface is removed in calculating the optical flow.

Claims

1. A method for detecting obstacles in the path of a moving vehicle, comprising the steps of:

imaging a view from the moving vehicle as at least two image frames, a first frame of the two image frames being taken at a first time and a second frame of the two image frames being taken at a succeeding time;

targeting a point in the first frame;

defining a location in the first frame for a long and narrow window encompassing the targeted point in a predetermined relationship and extending in a radial direction relative to a focus of expansion in the first frame;

establishing a location in the second frame for the long and narrow window by (i) stepwise moving the window in the radial direction while calculating a sum of absolute values of differences in luminance between the window in the first frame and the window in the second frame, and (ii) determining where the calculated sum attains a minimum value;

ascertaining a point in the second frame corresponding to the targeted point based on the location of the window in the second frame by utilizing the predetermined relationship;

defining an optical flow vector as a displacement measured from the targeted point to the corresponding point; and

utilizing a magnitude and a positioning of the optical flow vector relative to the focus of expansion to determine a degree of danger for the moving vehicle.

2. A method for detecting obstacles in the path of a moving vehicle, comprising the steps of:

using a video camera to capture a view from the moving vehicle as at least two image frames, a first frame of the two image frames being taken at a first time and a second frame of the two image frames being taken at a succeeding time;

targeting points in the first frame;

detecting displacements of the targeted points in the second frame as respective optical flow vectors;

removing optical flow vectors resulting from a lane-dividing line or a character drawn on a road surface by;

(i) determining a height measured from the line or the character to the video camera, and

(ii) deleting those optical flow vectors for which the determined height coincides with a height of the video camera to the road surface; and

utilizing magnitudes and positionings relative to a focus of expansion of remaining optical flow vectors to determine a degree of danger for the moving vehicle.

3. A method for detecting obstacles in the path of a moving vehicle, comprising the steps of:

using a video camera to capture a view from the moving vehicle as at least two image frames, a first frame of the two image frames being taken at a first time and a second frame of the two image frames being taken at a succeeding time;

targeting points in a processing area of the first frame;

detecting displacements of the targeted points in the second frame as respective optical flow vectors;

dividing the processing area into a plurality of zones;

for each of the zones, weighting a sum of magnitudes of the optical flow vectors present in each of the zones on a zone basis; and

determining a degree of danger for the moving vehicle from the weighted sum calculated for each of the zones.

4. The method according to claim 3, wherein said determining step comprises:

setting a respective, predetermined threshold for each of the zones; and

if the weighted sum in at least one of the zones exceeds the predetermined threshold for that one zone, judging that the moving vehicle is in a dangerous situation.

5. The method according to claim 3, wherein said determining step comprises:

for each of the zones, setting plural, predetermined thresholds of differing levels; and

for each of the zones, determining the degree of danger based on which of the thresholds is exceeded by the weighted sum for that zone.

6. The method according to claim 5, further comprising the step of:

outputting an alarm in accordance with the determined degree of danger.

7. A method for detecting obstacles in a vicinity of a moving vehicle, comprising the steps of:

imaging a view from the moving vehicle as at least two image frames, a first frame of the two image frames being taken at a first time and a second frame of the two image frames being taken at a succeeding time;

targeting a point in the first frame;

defining a location in the first frame for a long and narrow window encompassing the targeted point in a predetermined relationship and extending in a radial direction relative to a focus of expansion in the first frame;

establishing a location in the second frame for the long and narrow window by (i) stepwise moving the window in the radial direction while calculating a sum of absolute values of differences in luminance between the window in the first frame and the window in the second frame, and (ii) determining where the calculated sum attains a minimum value;

ascertaining a point in the second frame corresponding to the targeted point based on the location of the window in the second frame by utilizing the predetermined relationship;

defining an optical flow vector as a displacement measured from the targeted point to the corresponding point;

selecting the optical flow vector only if the optical flow vector defines a displacement away from the focus of expansion; and

if said selecting step yields a selected optical flow vector, utilizing a magnitude and a positioning of the selected optical flow vector relative to the focus of expansion to determine a degree of danger for the moving vehicle.

8. The method according to claim 7, wherein the view imaged from the moving vehicle is a rear view.

9. The method according to claim 7, wherein the view imaged from the moving vehicle is a side view.