FUSION OF FAR INFRARED AND VISIBLE IMAGES IN ENHANCED OBSTACLE DETECTION IN AUTOMOTIVE APPLICATIONS

• US 20080036576A1
• Filed: 04/05/2007
• Published: 02/14/2008
• Est. Priority Date: 05/31/2006
• Status: Active Grant

• Current Assignee: MobilEye Vision Technologies Ltd. (Intel Corporation)
• Original Assignee: Mobileye Technologies Limited (Intel Corporation)
• Inventors: Liyatan, Harel, Stein, Gideon S., Shashua, Amnon, Gdalyahu, Yoram
• Granted Patent: US 7,786,898 B2
• Time in Patent Office: Days
• US Class Current: 340/425.5
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/106 using night vision cameras
• CPC Class Codes: B60R 2300/304 using merged images, e.g. merging camera image with stored images
• CPC Class Codes: B60R 2300/8033 for pedestrian protection
• CPC Class Codes: B60R 2300/804 for lane monitoring
• CPC Class Codes: B60R 2300/8053 for bad weather conditions or night vision
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G08G 1/167 Driving aids for lane monitoring, lane changing, e.g. blind spot detection

Abstract

A method in computerized system mounted on a vehicle including a cabin and an engine. The system including a visible (VIS) camera sensitive to visible light, the VIS camera mounted inside the cabin, wherein the VIS camera acquires consecutively in real time multiple image frames including VIS images of an object within a field of view of the VIS camera and in the environment of the vehicle. The system also including a FIR camera mounted on the vehicle in front of the engine, wherein the FIR camera acquires consecutively in real time multiple FIR image frames including FIR images of the object within a field of view of the FIR camera and in the environment of the vehicle. The FIR images and VIS images are processed simultaneously, thereby producing a detected object when the object is present in the environment.

Claims

1. In a computerized system mounted on a vehicle including a cabin and an engine, the system including a visible (VIS/NIR) camera sensitive to visible/near infrared VIS/NIR) light, the VIS/NIR camera mounted inside the cabin, wherein the VIS/NIR camera acquires consecutively in real time a plurality of image frames including respectively a plurality of VIS/NIR images of an object within a field of view of the VIS/NIR camera, a method for detecting the object, the method comprising the steps of:

(a) providing a FIR camera mounted on the vehicle in front of the engine, wherein said FIR camera acquires consecutively in real time a plurality of FIR image frames including respectively a plurality of FIR images of an object within a field of view of said FIR camera, wherein at least one of said FIR images is of the environment of the vehicle; and

(b) simultaneous processing at least one of said FIR images and at least one of the VIS/NIR images, thereby producing a detected object when the object is present in said environment.

2. The method of claim 1, wherein the VIS/NIR camera includes a visible light blocking and near infrared passing filter, and thereby images only NIR spectrum light.

3. The method of claim 1, further comprising the step of:

(c) determining range from the vehicle front to said detected object upon detecting the object.

4. The method of claim 3, wherein said determining of the range to said detected object is performed from images of said VIS/NIR camera, having a field of view (FOV) of more than twenty five degrees.

5. The method of claim 4, wherein said detecting of said detected object and said determining of said range to said detected object is performed from images of said VIS/NIR camera, the method further comprising the steps of:

(d) projecting the image location of said detected object in said VIS/NIR image onto a location in the at least one FIR image; and

(e) identifying said detected object in a corresponding said at least one FIR image.

6. The method of claim 5, wherein said detected object is a vehicle and wherein said identifying of said detected object in said corresponding at least one FIR image includes locating bright spot features selected from the group of features consisting of the target vehicle wheels (being hot) and the exhaust pipe.

7. The method of claim 4, wherein said VIS/NIR camera is operatively connected to a processor, wherein said processor performs collision avoidance by triggering braking.

8. The method of claim 3, wherein said determining of the range to said detected object is performed from images of said FIR camera, having a field of view (FOV) of less than thirty five degrees, said detected object typically situated more than 10 meters from the vehicle in the vehicle environment.

9. The method of claim 8, wherein said detecting of said detected object and said determining of said range are performed in said at least one FIR image, the method further comprising the steps of;

(d) upon said detecting said detected object, projecting an image location of said detected object in said FIR image onto a location in the VIS/NIR image; and

(e) identifying said detected object in a corresponding said at least one VIS/NIR image,

10. The method of claim 9, wherein upon said detecting, identifying said detected object in said corresponding said at least one VIS/NIR image, includes alignment of patches of said at least one FIR image with patches of said at least one VIS/NIR image, thereby producing al aligned object.

11. The method of claim 10, wherein said alignment of at least one patch of said at least one FIR image and said at least one VIS/NIR image, said alignment using at least one mechanism selected from the group consisting of histograms of gray scale values, sub-patch correlation and Hausdorf distance computational techniques.

12. The method of claim 10, wherein said alignment of said at least one patch is bounded by an estimate of said range, said estimate computed from said detected object in said at least two FIR image.

13. The method of claim 10, further comprising the steps of:

(f) tracking said aligned object by both said VIS/NIR camera and said FIR camera; and

(g) if said tracking of said aligned object by either of said VIS/NIR camera and said FIR camera is terminated, the camera that has not terminated said tracking proceeds with said tracking.

14. The method of claim 10, wherein said detected object in said at least one FIR image is a pedestrian and wherein said image brightness in said FIR image is used to verify that said pedestrian temperature matches that of a human.

15. The method of claim 8, further comprising the steps of:

(d) said processing optimizes said VIS/NIR camera gain and exposure for said projected image locations of at least one patch of said at least one FIR image location, wherein said VIS image includes at least a portion of said detected object;

(e) Identifying said detected object in a corresponding said at least one VIS/NIR image.

16. The method of claim 15, wherein said identifying said detected object in a corresponding said at least one FIR image, includes aligning patches of said at least one FIR image with patches of said at least one VIS/NIR image, thereby producing an aligned object.

17. The method of claim 16, wherein said alignment of at least a patch of said at least one FIR image and said at least one VIS/NIR image, said alignment using at least one algorithm/mechanism/calculation selected from the group consisting of histograms of the gray level values, sub-patch correlation and/or Hausdorf distance computational techniques.

18. The method of claim 16, wherein said alignment of at least one patch is bounded by said estimate of said range, said estimate computed from said detected object in said at least one FIR image.

19. The method of claim 8, wherein said determining of said range is performed in at least two FIR images, the method further comprising the step of:

(d) determining the scale change ratio between dimensions of said detected object in the images and using said scale change ratio and the vehicle speed to refine said range.

20. The method of claim 3, wherein said determining of the range to an object is performed from said simultaneous processing of respective images of said VIS/NIR camera and said FIR camera.

21. The method of claim 20, wherein said simultaneous processing includes a stereo analysis of at least a pair of corresponding images acquired by said VIS/NIR camera and said FIR camera, providing refined range estimation.

22. The method of claim 21, wherein said VIS/NIR camera is operatively connected to a processor, wherein said processor said simultaneously performs collision avoidance by triggering braking.

23. The method of claim 3, wherein said determining of said range to said detected object further comprises the step of:

(d) determining an accurate lateral distance to said detected object and determining if the object is in the vehicle path and in danger of collision with the moving vehicle.

25. The system of claim 24, wherein said processor performs the method of claim 1.

24. A system mounted on a vehicle including a cabin and an engine, the system including a visible (VIS/NIR) camera sensitive to visible/near infrared (VIS/NIR) light, the VIS/NIR camera mounted inside the cabin, wherein the VIS/NIR camera acquires consecutively in real time a plurality of image frames including respectively a plurality of VIS/NIR images of an object within a field of view of the VIS/NIR camera, the system comprising:

(a) a FIR camera mounted on the vehicle in front of the engine, wherein said FIR camera acquires consecutively in real time a plurality of FIR image frames including respectively a plurality of FIR images of an object within a field of view of said FIR camera, wherein at least one of said FIR images is of the environment of the vehicle; and

(b) A processor simultaneous processing at least one of said FIR images and at least one of the VIS/NIR images, thereby producing a detected object when the object is present in said environment.