Fusion of far infrared and visible images in enhanced obstacle detection in automotive applications

• US 9,443,154 B2
• Filed: 09/25/2015
• Issued: 09/13/2016
• Est. Priority Date: 05/31/2006
• Status: Active Grant

• Current Assignee: MobilEye Vision Technologies Ltd. (Intel Corporation)
• Original Assignee: MobilEye Vision Technologies Ltd. (Intel Corporation)
• Inventors: Stein, Gideon, Shashua, Amnon, Gdalyahu, Yoram
• Primary Examiner(s): TRIEU, VAN THANH
• Application Number:
  US14/866,362
• Publication Number:
  US 20160104048A1
• Time in Patent Office: 354 Days
• US Class Current: 1/1
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/106 using night vision cameras
• CPC Class Codes: B60R 2300/304 using merged images, e.g. merging camera image with stored images
• CPC Class Codes: B60R 2300/8033 for pedestrian protection
• CPC Class Codes: B60R 2300/804 for lane monitoring
• CPC Class Codes: B60R 2300/8053 for bad weather conditions or night vision
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G08G 1/167 Driving aids for lane monitoring, lane changing, e.g. blind spot detection

Abstract

A computerized system mountable on a vehicle operable to detect an object by processing first image frames from a first camera and second image frames from a second camera. A first range is determined to said detected object using the first image frames. An image location is projected of the detected object in the first image frames onto an image location in the second image frames. A second range is determined to the detected object based on both the first and second image frames. The detected object is tracked in both the first and second image frames When the detected object leaves a field of view of the first camera, a third range is determined responsive to the second range and the second image frames.

Claims

1. A method performable by a computerized system mounted to travel with a vehicle, the system comprising a first camera having a first field of view and a second camera having a second field of view, the method comprising:

acquiring multiple first image frames via the first camera,

acquiring multiple second image frames via the second camera,

processing the first image frames to detect an object,

processing the first image frames to determine a first distance between the vehicle and the detected object,

projecting an image location of the detected object in at least one of the first image frames onto a location in at least one of the second image frames,

processing the first image frames and the second image frames to determine a second distance between the vehicle and the detected object,

tracking the detected object in the first image frames and the second image frames, and

processing the second image frames and the second distance to determine a third distance between the vehicle and the detected object,

whereby estimated distances between the vehicle and the detected object are provided when the detected object is inside the first field of view and when the detected object is outside the first field of view.

2. The method of claim 1, wherein projecting an image location of the detected object in at least one of the first image frames onto a location in at least one of the second image frames comprises aligning a patch of at least one first image and a patch of at least one second image.

3. The method of claim 2, wherein aligning a patch of at least one first image and a patch of at least one second image comprises at least one of gray scale value histogram computation, sub-patch correlation and Hausdorf distance computation.

4. The method of claim 1, wherein processing the first image frames and the second image frames to determine a second distance comprises stereo analysis.

5. The method of claim 1, wherein the first camera continues tracking the detected object when the second camera terminates tracking the detected object and wherein the second camera continues tracking the detected object when the first camera terminates tracking the detected object.

6. The method of claim 1, comprising adjusting a gain or exposure of the second camera responsive to detecting the object in first camera image frame.

7. The method of claim 1, wherein the locating the detected object in the second camera image frame includes searching for the image of the detected object in the second camera image frame around an epipolar line corresponding to the detected object in the first camera image frame.

8. A computerized system mountable on a vehicle, the system comprising:

a first camera and a second camera connectable to a processor,

the processor operable to;

acquire multiple first image frames from the first camera,

acquire multiple second image frames from the second camera,

detect an object by processing the first image frames to produce thereby a detected object,

determine a first distance to the detected object using the first image frames,

project an image location of the detected object in at least one of the first image frames onto an image location in at least one of the second image frames,

determine a second distance from the vehicle to the detected object based on both the first image frames and the second image frames,

track the detected object in both the first image frames and second image frames and when the detected object leaves a field of view of the first camera,

determine a third distance to the detected object responsive to the second distance and the second image frames.

9. The system of claim 8, wherein the first camera has a field of view less than thirty five degrees.

10. The system of claim 8, wherein the second camera has a field of view greater than twenty five degrees.

11. The system of claim 8, wherein the vehicle has a travel direction and the first and second cameras are positioned to acquire images in the vehicle travel direction.

12. The system of claim 8, wherein the vehicle has a travel direction and the first and second cameras are positioned to acquire images in a direction other than the vehicle travel direction.

13. The system of claim 8, wherein the first and second cameras are relatively displaced along a vertical axis.

14. The system of claim 8, wherein the vehicle has a travel direction and the first and second cameras are relatively displaced along an axis perpendicular to the vehicle travel direction.

15. The system of claim 8, wherein the first camera comprises a filter that blocks visible light and passes near infrared light.