Vision-based highway overhead structure detection system

• US 20030002713A1
• Filed: 06/20/2001
• Published: 01/02/2003
• Est. Priority Date: 06/20/2001
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Chen, Yang
• Granted Patent: US 7,027,615 B2
• Time in Patent Office: Days
• US Class Current: 382/104
• CPC Class Codes: B60T 2201/08 Lane monitoring; Lane Keeping Systems
• CPC Class Codes: B60T 2201/089 using optical detection
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G08G 1/04 using optical or ultrasonic detectors
• CPC Class Codes: G08G 1/164 Centralised systems, e.g. external to vehicles

Abstract

An image sequence is inputted 200 from the camera and vertical motion is estimated 202. A windowed horizontal edge projection 204 is extracted from the inputted image sequence 200 and the horizontal edges are projected 206. The horizontal edge projection 206 and the vertical motion estimation 202 are combined in a horizontal segmentation and tracking element 208, and forwarded to an object parameter estimation element 210 where the object'"'"'s distance and height are estimated. This data is combined in a fusion with radar detection element 212. By correctly matching the overhead objects sensed by the radar and video camera, the proximity and relative speed can be ascertained. Once overhead objects have been identified they can be isolated and not considered for collision avoidance purposes.

Claims

1. A vision-based identification apparatus comprising:

a host vehicle;

wherein said host vehicle is equipped with a plurality of elements including;

i) an image capture device element;

operatively interconnected with;

ii) an image signal processor element;

which is operatively interconnected with;

iii) a matching processor element;

which is also operatively interconnected with;

iv) a radar transceiver element;

wherein the image capture device is configured to provide a time based sequence of data frames to the image signal processor and the image signal processor provides a processed image signal to the matching processor, and

wherein the data frames include a two dimensional array of pixel elements; and

wherein the radar transceiver is configured to provide a radar signal to the matching processor, and

wherein the matching processor combines the processed image signal and the radar signal, whereby the combined signals complement each other and allow the apparatus to effectively identify objects likely to be misidentified as collision threats.

2. The vision-based identification apparatus of claim 1, wherein the image capture device is a video camera responsive to electromagnetic radiation substantially in at least one of the regions selected from the group consisting of the visible region, and the infrared region.

3. The vision-based identification apparatus of claim 1, wherein the time based sequence of data frames include a plurality of data elements including at least one horizontal edge;

and

wherein each horizontal edge is identified based on a plurality of pixels having a substantially similar electromagnetic radiation response across a plurality of substantially horizontally aligned pixels; and

wherein the image signal processor extracts horizontal edges, from the time based sequence of data frames, in the form of edge pixels, and

wherein the edge pixels are projected in each row of the data frames, to get a horizontal edge projection in the sequence of data frames; and

wherein each horizontal edge projection may be tracked in time based on the horizontal edge projection'"'"'s sequential appearance in the data frames.

4. The vision-based identification apparatus of claim 3, wherein horizontal edge projections may be tracked in time by recording sequential data frames and matching horizontal edge projections in the sequential data frames while allowing, in the sequential data frames, for minor variations from, in at least one of the following categories:

i. the relative position of the horizontal edge projection in the frame;

ii. the relative orientation of the horizontal edge projection in the frame; and

iii. the relative length of the horizontal edge projection; and

wherein at least one new tracking sequence can exist for horizontal edge projections that have predefined characteristics but were not present in prior data frames.

5. The vision-based identification apparatus of claim 4, wherein allowed, minor variations in the sequential data frames provide data that allows for the determination and recordation of distance traveled since the beginning of each horizontal edge projection tracking sequence, and

wherein horizontal edge projection tracks are sorted based on at least one of the following;

i) the duration of image inputs that the horizontal edge projection track records,

ii) the average length of the horizontal edge projection, and

iii) whether the horizontal edge projection track at current image frame is in an updating mode, a non-updating mode, or is a new horizontal edge projection.

6. The vision-based identification apparatus of claim 4 wherein the horizontal edge projection tracks are compared with predetermined parameters to determine if tracking possibilities exist;

if tracking possibilities exist then the vision-based identification apparatus is utilized to track substantially horizontal edges through successive image inputs.

7. The vision-based identification apparatus of claim 6 wherein the predetermined parameters include at least one of the following;

i. the number of successive image inputs having horizontal edge projection tracks, and

ii. the magnitude of the horizontal edge projection tracks.

8. The vision-based identification apparatus of claim 4 wherein if no tracking possibilities exist, tracking may still occur for a pre-specified number of image inputs without tracking possibilities before the tracked horizontal edge projection track is discarded.

9. The vision-based identification apparatus of claim 1 wherein tracking of horizontal edge projection vectors is assisted by using at least one of the following:

i) vertical motion compensation;

ii) forward motion compensation;

wherein vertical motion compensation helps predict where tracked horizontal edge projection vectors will be located on successive image inputs by compensating for vertical motion, and

wherein the forward motion compensation helps predict where tracked horizontal edge projection vectors will be located on successive image inputs by compensating for forward motion.

10. The vision-based identification apparatus of claim 9 wherein forward motion compensation is achieved by using at least two previous points from the image input on the tracked horizontal edge projection tracks, immediately before the image input that requires forward motion compensation for tracked horizontal edge projection vectors location prediction.

11. The vision-based identification apparatus of claim 9 wherein vertical motion is determined by:

i. extracting vertical slices of the image input; and

ii. extracting one dimensional optical flow profiles of relative intensity of the vertical slices; and

iii. averaging the optical flow of all vertical slices.

12. The vision-based identification apparatus of claim 11 wherein the vertical slices are extracted from each image input, and

the relative extreme negative intensities relate to a change from light to dark of an apparent horizon.

13. The vision-based identification apparatus of claim 11 wherein the extreme negative intensities'"'"' average variation between rows on successive image inputs is an indicia of how an image collection device is moving vertically relative to at least one of the following:

i. an apparent horizon; and

ii. a distinct feature in the distance.

14. The vision-based identification apparatus of claim 1 wherein the signal input from the image capture device to the image signal processor is a single horizontally centered window of the image signal input from the image capture device.

15. The vision-based identification apparatus of claim 14 wherein the single horizontally centered window is set at a predetermined width and a predetermined height and wherein the single horizontally centered window can be adjusted either left or right based on steering wheel position, or lane information.

16. The vision-based identification apparatus of claim 14 wherein successive horizontal edge projection tracks have a length in excess of a predetermined length.

17. The vision-based identification apparatus of claim 14 wherein if no tracking possibilities exist, a tracking protocol will be allowed to remain in operation for a pre-specified number of image inputs without tracking possibilities before the tracked horizontal edge projection track is discarded.

19. A vision-based identification method as set forth in claim 18, wherein the image capture device element is a video camera responsive to electromagnetic radiation substantially in at least one of the regions selected from the group consisting of the visible region, and the infrared region.

20. A vision-based identification method as set forth in claim 18, wherein the time based sequence of data frames include a plurality of data elements including at least one horizontal edge;

and

wherein each horizontal edge is identified based on a plurality of pixels having a substantially similar electromagnetic radiation response across a plurality of substantially horizontally aligned pixels; and

wherein the image signal processor extracts horizontal edges, from the time based sequence of data frames, in the form of edge pixels; and

wherein the edge pixels are projected in each row of the data frames, to get a horizontal edge projection in the sequence of data frames; and

wherein each horizontal edge projection may be tracked in time based on the horizontal edge projection'"'"'s sequential appearance in the data frames.

21. A vision-based identification method as set forth in claim 20, wherein horizontal edge projections may be tracked in time by recording sequential data frames and matching horizontal edge projections in the sequential data frames while allowing, in the sequential data frames, for minor variations from, in at least one of the following categories:

i. the relative position of the horizontal edge projection in the frame;

ii. the relative orientation of the horizontal edge projection in the frame; and

iii. the relative length of the horizontal edge projection; and

wherein at least one new tracking sequence can exist for horizontal edge projections that have predefined characteristics but were not present in prior data frames.

22. A vision-based identification method as set forth claim 21, wherein allowed, minor variations in the sequential data frames provide data that allows for the determination and recordation of distance traveled since the beginning of each horizontal edge projection tracking sequence, and

wherein horizontal edge projection tracks are sorted based on at least one of the following;

i. the duration of image inputs that the horizontal edge projection track records;

ii. the average length of the horizontal edge projection; and

iii. whether the horizontal edge projection track at current image frame is in an updating mode, a non-updating mode, or is a new horizontal edge projection.

23. A vision-based identification method as set forth in claim 21 wherein the horizontal edge projection tracks are compared with predetermined parameters to determine if tracking possibilities exist;

if tracking possibilities exist then the vision-based identification apparatus is utilized to track substantially horizontal edges through successive image inputs.

24. A vision-based identification method as set forth in claim 23 wherein the predetermined parameters include at least one of the following;

i. the number of successive image inputs having horizontal edge projection tracks; and

ii. the magnitude of the horizontal edge projection tracks.

25. A vision-based identification method as set forth in claim 21 wherein if no tracking possibilities exist, tracking may still occur for a pre-specified number of image inputs without tracking possibilities before the tracked horizontal edge projection track is discarded.

26. The vision-based identification method of claim 18 wherein tracking of horizontal edge projection vectors is assisted by using at least one of the following:

i) vertical motion compensation; and

ii) forward motion compensation;

wherein vertical motion compensation helps predict where tracked horizontal edge projection vectors will be located on successive image inputs by compensating for vertical motion, and

wherein the forward motion compensation helps predict where tracked horizontal edge projection vectors will be located on successive image inputs by compensating for forward motion.

27. The vision-based identification method of claim 26, wherein forward motion compensation is achieved by using at least two previous points from at the image input on the tracked horizontal edge projection tracks, immediately before the image input that requires forward motion compensation for tracked horizontal edge projection vectors location prediction.

28. A vision-based identification method as set forth in claim 26, wherein vertical motion is determined by:

i. extracting vertical slices of the image input; and

ii. extracting one dimensional optical flow profiles of relative intensity of the vertical slices; and

iii. averaging the optical flow of all vertical slices.

29. A vision-based identification method as set forth in claim 28, wherein the vertical slices are extracted from each image input, and

the relative extreme negative intensities relate to a change from an electromagnetic radiation emission region having a greater intensity to an electromagnetic radiation emission region of lower intensity of an apparent horizon.

30. A vision-based identification method as set forth in claim 28, wherein the extreme negative intensities'"'"' average variation between rows on successive image inputs is an indicia of how an image collection device is moving vertically relative to at least one of the following:

i. an apparent horizon; and

ii. a distinct feature in the distance.

31. A vision-based identification method as set forth in claim 18, wherein the signal input from the image capture device to the image signal processor is a single horizontally centered window of the image signal input from the image capture device.

32. A vision-based identification method as set forth in claim 31, wherein the single horizontally centered window is set at a predetermined width and a predetermined height;

and

wherein the single horizontally centered window can be adjusted either left or right based on steering wheel position, or lane information.

33. A vision-based identification method as set forth in claim 31, wherein successive horizontal edge projection tracks have a length in excess of a predetermined length.

34. A vision-based identification method as set forth in claim 31 wherein if no tracking possibilities exist, a tracking protocol will be allowed to remain in operation for a pre-specified number of image inputs without tracking possibilities before the tracked horizontal edge projection track is discarded.

18. A vision-based identification method comprising the steps of:

providing a host vehicle; and

equipping the host vehicle with a plurality of elements including;

i) an image capture device element;

operatively interconnected with;

ii) an image signal processor element;

which is operatively interconnected with;

iii) a matching processor element;

which is also operatively interconnected with;

iv) a radar transceiver element;

wherein the image capture device is configured to provide a time based sequence of data frames to the image signal processor and the image signal processor provides a processed image signal to the matching processor; and

wherein the data frames include a two dimensional array of pixel elements; and

wherein the radar transceiver is configured to provide a radar signal to the matching processor; and

wherein the matching processor combines the processed image signal and the radar signal, whereby the combined signals complement each other and allow for the effective identification of objects likely to be misidentified as collision threats.