Fusion of far infrared and visible images in enhanced obstacle detection in automotive applications

• US 8,981,966 B2
• Filed: 01/23/2013
• Issued: 03/17/2015
• Est. Priority Date: 05/31/2006
• Status: Active Grant

• Current Assignee: MobilEye Vision Technologies Ltd. (Intel Corporation)
• Original Assignee: MobilEye Vision Technologies Ltd. (Intel Corporation)
• Inventors: Stein, Gideon, Shashua, Amnon, Gdalyahu, Yoram, Livyatan, Harel
• Primary Examiner(s): TRIEU, VAN THANH
• Application Number:
  US13/747,657
• Publication Number:
  US 20130135444A1
• Time in Patent Office: 783 Days
• Field of Search: 340/425.5, 340/435, 340/436, 340/437, 340/903, 340/904, 340/907, 340/937, 348/148, 348/188, 348/208.99, 382/104, 180/169, 180/170, 701/1, 701/300
• US Class Current: 340/937
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/106 using night vision cameras
• CPC Class Codes: B60R 2300/304 using merged images, e.g. merging camera image with stored images
• CPC Class Codes: B60R 2300/8033 for pedestrian protection
• CPC Class Codes: B60R 2300/804 for lane monitoring
• CPC Class Codes: B60R 2300/8053 for bad weather conditions or night vision
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G08G 1/167 Driving aids for lane monitoring, lane changing, e.g. blind spot detection

Abstract

A computerized system mountable on a vehicle operable to detect an object by processing first image frames from a first camera and second image frames from a second camera. A first range is determined to said detected object using the first image frames. An image location is projected of the detected object in the first image frames onto an image location in the second image frames. A second range is determined to the detected object based on both the first and second image frames. The detected object is tracked in both the first and second image frames When the detected object leaves a field of view of the first camera, a third range is determined responsive to the second range and the second image frames.

Claims

1. A method performable by a computerized system mountable on a vehicle, wherein the system includes a first camera and a second camera connectible to a processor, the method comprising:

first capturing a plurality of first image frames from the first camera;

second capturing a plurality of second image frames from the second camera;

detecting said object by processing said first image frames thereby producing a detected object;

first determining a first range to said detected object using said first image frames;

projecting an image location of said detected object in at least one of said first image frames onto an image location in at least one of said second image frames;

second determining a second range from the vehicle to said detected object based on both said first image frames and said second image frames; and

tracking said detected object in both said first image frames and second image frames and when said detected object leaves a field of view of the first camera, third determining a third range to said detected object responsive to said second range and said second image frames.

2. The method of claim 1, wherein said projecting is performed by aligning a patch of said at least one second image with a patch of said at least one first image, thereby producing an aligned object, wherein said aligning is bounded by said first range.

3. The method of claim 2, wherein said aligning is performed using at least one mechanism selected from the group consisting of histograms of gray scale values, sub-patch correlation and Hausdorf distance computational techniques.

4. The method of claim 1, wherein said second determining second second range is performed by stereo analysis.

5. The method of claim 1, wherein said tracking of said detected object by either of said first camera and said second camera is terminated, the camera that has not terminated said tracking proceeds with said tracking.

6. The method of claim 1, further comprising adjusting a gain or exposure of the second camera responsive to detecting the object in first camera image frame.

7. The method of claim 1, wherein said locating the detected object in the second camera image frame includes searching for the image of the detected object in the second camera image frame around an epipolar line corresponding to the detected object in the first camera image frame.

8. A computerized system mountable on a vehicle, wherein the system includes a first camera and a second camera connectible to a processor, the processor operable to:

first capture a plurality of first image frames from the first camera;

second capture a plurality of second image frames from the second camera;

detect said object by processing said first image frames to produce thereby a detected object;

determine a first range to said detected object using said first image frames;

project an image location of said detected object in at least one of said first image frames onto an image location in at least one of said second image frames;

determine a second range from the vehicle to said detected object based on both said first image frames and said second image frames;

track said detected object in both said first image frames and second image frames and when said detected object leaves a field of view of the first camera, determine a third range to said detected object responsive to said second range and said second image frames.

9. The system of claim 8, wherein the first camera has a field of view (FOV) of less than thirty five degrees.

10. The system of claim 8, wherein the second camera has a field of view (FOV) of more than twenty five degrees.

11. The system of claim 8, wherein the first and second cameras are directed forwards to acquire images in the direction of travel of the vehicle.

12. The system of claim 8, wherein the first and second cameras are directed backwards to acquire images in the direction opposite the direction of travel of the vehicle.

13. The system of claim 8, wherein the first and second cameras are mounted on the vehicle at substantially different heights above road surface.

14. The system of claim 8, wherein the first and second cameras are mounted on the vehicle at substantially different lateral locations in a dimension perpendicular to the forward direction of movement of the vehicle.

15. The system of claim 8, wherein the first camera includes a visible light blocking and near infrared (NIR) passing filter, and thereby images only NIR spectrum light.