Fusion of far infrared and visible images in enhanced obstacle detection in automotive applications

• US 7,786,898 B2
• Filed: 04/05/2007
• Issued: 08/31/2010
• Est. Priority Date: 05/31/2006
• Status: Expired due to Fees

• Current Assignee: MobilEye Vision Technologies Ltd. (Intel Corporation)
• Original Assignee: Mobileye Technologies Limited (Intel Corporation)
• Inventors: Shashua, Amnon, Gdalyahu, Yoram, Liyatan, Harel, Stein, Gideon S.
• Primary Examiner(s): TRIEU, VAN THANH
• Application Number:
  US11/696,731
• Publication Number:
  US 20080036576A1
• Time in Patent Office: 1,244 Days
• Field of Search: 340/903, 340/904, 340/907, 340/937, 340/435, 340/436, 340/437, 340/425.5, 348/148, 382/104, 250/338.1, 250/336.1, 250/330, 180/169, 180/170
• US Class Current: 340/937
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/106 using night vision cameras
• CPC Class Codes: B60R 2300/304 using merged images, e.g. merging camera image with stored images
• CPC Class Codes: B60R 2300/8033 for pedestrian protection
• CPC Class Codes: B60R 2300/804 for lane monitoring
• CPC Class Codes: B60R 2300/8053 for bad weather conditions or night vision
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G08G 1/167 Driving aids for lane monitoring, lane changing, e.g. blind spot detection

Abstract

A method in computerized system mounted on a vehicle including a cabin and an engine. The system including a visible (VIS) camera sensitive to visible light, the VIS camera mounted inside the cabin, wherein the VIS camera acquires consecutively in real time multiple image frames including VIS images of an object within a field of view of the VIS camera and in the environment of the vehicle. The system also including a FIR camera mounted on the vehicle in front of the engine, wherein the FIR camera acquires consecutively in real time multiple FIR image frames including FIR images of the object within a field of view of the FIR camera and in the environment of the vehicle. The FIR images and VIS images are processed simultaneously, thereby producing a detected object when the object is present in the environment.

Claims

1. A method of determining a danger of collision between a vehicle and an object, comprising:

acquiring by a first camera mounted on the vehicle, a plurality of first camera image frames acquired at different times;

acquiring by a second camera, mounted on the vehicle, a second camera image frame;

detecting an object, possibly in a course of collision with the vehicle, in one or more of the first camera image frames;

locating the detected object in the second camera image frame;

determining a distance between the vehicle and the object responsive to the location of the detected object in both the first camera and second camera image frames; and

determining if there is a danger of collision between the vehicle and the object at least partially responsive to the determined distance.

2. The method of claim 1, wherein the first camera includes a visible light blocking and near infrared (NIR) passing filter, and thereby images only NIR spectrum light.

3. The method of claim 1, wherein locating the detected object in the second camera image frame comprises determining a range from the vehicle front to said detected object upon detecting the object, based on information from the one or more first camera image frames, and locating the object in the second image frame responsive thereto.

4. The method of claim 3, wherein said determining of said range is performed in at least two first camera image frames, by determining the scale change ratio between dimensions of said detected object in the at least two images.

5. The method of claim 1, wherein said detected object is a vehicle.

6. The method of claim 1, comprising performing collision avoidance by triggering braking by a processor if a danger of collision was determined.

7. The method of claim 1, wherein locating the detected object in the second camera image frame comprises aligning at least one patches of said at least one first camera image frame with patches of said at least one second camera image frame, thereby producing an aligned image.

8. The method of claim 7, wherein said alignment of at least one patch of said at least one first camera image and said at least one second camera image, is performed using at least one mechanism selected from the group consisting of histograms of gray scale values, sub-patch correlation and Hausdorf distance computational techniques.

9. The method of claim 1, further comprising tracking said detected object by both said first camera and said second camera;

and if said tracking of said detected object by either of said first camera and said second camera is terminated, the camera that has not terminated said tracking proceeds with said tracking.

10. The method of claim 1, wherein said detected object is a pedestrian.

11. The method of claim 1, further comprising adjusting a gain or exposure of the second camera responsive to detecting the object in the one or more first camera image frames.

12. The method of claim 11, wherein adjusting a gain or exposure of the second camera comprises adjusting responsive to the location of the object in the second camera image frame.

13. The method of claim 1, wherein determining the distance responsive to the location of the detected object in both the first camera and second camera image frames includes a stereo analysis of at least a pair of corresponding first camera and second camera image frames.

14. The method of claim 1, wherein locating the detected object in the second camera image frame comprises searching for the object in the second image around an epipolar line corresponding to the detected object in the first image.

15. According to claim 14, wherein searching for the object in the second camera image frame around an epipolar line corresponding to the determined line in the one or more first camera image frames comprises determining an estimate of the distance of the object from the vehicle based on the first camera image frame and searching in the second camera image frame at an area along the epipolar line corresponding to the estimated distance.

16. A method according to claim 1, wherein the first camera comprises a visible light camera.

17. A method according to claim 16, wherein the second camera comprises a far infrared camera.

18. A method according to claim 1, wherein the first camera is mounted inside a cabin of the vehicle.

19. A method according to claim 1, wherein the second camera is mounted on the vehicle in front of an engine of the vehicle.

20. A method according to claim 1, wherein the first camera is mounted on the vehicle in front of an engine of the vehicle.

21. A method according to claim 1, wherein the first and second cameras are at different longitudinal positions along the length of the vehicle.

22. The method of claim 1, wherein locating the detected object in the second image frame, comprises using information from the one or more first camera image frames.

23. The method of claim 1, comprising applying active safety measures if a danger of collision is identified.

24. A method according to claim 1, wherein the second camera comprises a visible light camera.

25. A method according to claim 24, wherein the first camera comprises a far infrared camera.

26. A vehicle warning system mounted on a vehicle including a cabin and an engine, the system including:

a first camera mounted on the vehicle, which acquires consecutively in real time a plurality of first camera image frames;

a second camera mounted on the vehicle, which acquires second camera image frames; and

a processor configured to detect an object in one or more first camera image frames, to locate the detected object in the second camera image frame, to determine the distance between the vehicle and the object responsive to the location of the detected object in both the first and second camera image frame and to determine if there is a danger of a collision between the vehicle and the object responsive to the determined distance.

27. The system of claim 26, wherein the first camera has a field of view (FOV) of less than thirty five degrees.

28. The system of claim 26, wherein the second camera has a field of view (FOV) of more than twenty five degrees.

29. The system of claim 26, wherein the first and second cameras are directed forwards to acquire images in the direction of travel of the vehicle.

30. The system of claim 26, wherein the first and second cameras are directed backwards to acquire images in the direction opposite the direction of travel of the vehicle.

31. The system of claim 26, wherein the processor is adapted to locate the detected object in the second camera image frame using information from the one or more first camera image frames.