Fusion of far infrared and visible images in enhanced obstacle detection in automotive applications

• US 9,323,992 B2
• Filed: 02/09/2015
• Issued: 04/26/2016
• Est. Priority Date: 05/31/2006
• Status: Active Grant

• Current Assignee: MobilEye Vision Technologies Ltd. (Intel Corporation)
• Original Assignee: MobilEye Vision Technologies Ltd. (Intel Corporation)
• Inventors: Stein, Gideon, Shashua, Amnon, Gdalyahu, Yoram
• Primary Examiner(s): TRIEU, VAN THANH
• Application Number:
  US14/617,559
• Publication Number:
  US 20160086041A1
• Time in Patent Office: 442 Days
• US Class Current: 1/1
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/106 using night vision cameras
• CPC Class Codes: B60R 2300/304 using merged images, e.g. merging camera image with stored images
• CPC Class Codes: B60R 2300/8033 for pedestrian protection
• CPC Class Codes: B60R 2300/804 for lane monitoring
• CPC Class Codes: B60R 2300/8053 for bad weather conditions or night vision
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G08G 1/167 Driving aids for lane monitoring, lane changing, e.g. blind spot detection

Abstract

A computerized system mountable on a vehicle operable to detect an object by processing first image frames from a first camera and second image frames from a second camera. A first range is determined to said detected object using the first image frames. An image location is projected of the detected object in the first image frames onto an image location in the second image frames. A second range is determined to the detected object based on both the first and second image frames. The detected object is tracked in both the first and second image frames When the detected object leaves a field of view of the first camera, a third range is determined responsive to the second range and the second image frames.

Claims

1. A system comprising:

a first camera that has a first field of view, the first camera configured to travel with a vehicle,

a second camera that has a second field of view, the second camera configured to travel with a vehicle, the first field of view being narrower than the second field of view, and

a processor that is configured to detect an object at a first time using a first frame of first image frames acquired via the first camera,

the processor is configured to track the object using the first image frames,

the processor is configured to project, at a second time, a location of the object in a second frame of the first image frames onto a location in a first frame of second image frames acquired via the second camera, and detect the object in the first frame of the second image frames using the projected location of the object.

2. The system of claim 1, wherein

the object and the vehicle are mutually spaced by a first distance when the object is detected in the first frame of the first image frames,

the object and the vehicle are mutually spaces by a second distance when the object is detected in the first frame of the second image frames, and

the first distance is greater than the second distance.

3. The system of claim 1, wherein the processor is configured to initiate a search for the object in frames of the second image frames following detection of the object in the first frame of the first image frames, the search for the object in frames of the second image frames using a projection of a location of the object in one or more frames of the first image frames onto a location in one or more frames of the second image frames.

4. The system of claim 3, wherein the processor is configured to use an up to date location of the object obtained from the first images frames to search for the object in the second image frames.

5. The system of claim 4, wherein the processor is configured to search for the object in at least one frame of the second image frames which precedes the first frame of the second images frames before successfully detecting the object in the first frame of the second images frames.

6. The system of claim 5, wherein the processor is configured to perform stereo analysis of the object when the object is tracked using first and second image frames.

7. The system of claim 5, wherein at the third time, the vehicle is within the field of view of the second camera and outside the field of view of the first camera.

8. The system of claim 1, wherein the processor is configured to track the object using the second image frames.

9. The system of claim 8, wherein the processor is configured to use stereo analysis to determine a size of the object or a distance to the object at a given time.

10. The system of claim 8, wherein following the third time, a distance to the object is estimated by processing the second image frames and using information obtained from the stereo analysis.

11. The system of claim 1, wherein at a third time, the object is outside the field of view of the first camera, such that at least a portion of the object is outside a third frame of the first frames.

12. The system of claim 1, wherein at the first time, the vehicle is at a distance relative to the vehicle which is within the effective object detection range of the first camera and outside the effective object detection range of the second camera.

13. The system of claim 1, wherein at the second time the object enters an illumination area of the vehicle.

14. The system of claim 1, wherein the second camera is sensitive to electromagnetic radiation in the visible spectrum.

15. The system of claim 1, wherein the first camera is sensitive to electromagnetic radiation outside the visible spectrum.

16. The system of claim 15, wherein at least one control measure of the vehicle which is controllable by the processor is controlled by activation criteria that are met at a time subsequent to the third time, and are evaluated based at least on the second image frames.

17. The system of claim 1, wherein the first camera is sensitive to IR radiation and wherein the processor is configured to use an IR signature of the object in the first image frame to classify the object.

18. The system of claim 1, wherein the processor is configured to control a motion of the vehicle according to a distance between the vehicle and the object.

19. The system of claim 1, wherein the first camera and the second camera are relatively displaced along at least one of three mutually perpendicular axes.

20. A method comprising:

obtaining first image frames from a first camera that has a first field of view and that is configured to travel with a vehicle,

obtaining second image frames from a second camera that has a second field of view and that is configured to travel with a vehicle, the first field of view being narrower than the second field of view,

detecting an object at a first time using a first frame of the first image frames,

tracking the object using the first image frames,

projecting, at a second time, a location of the object in a second frame of the first image frames onto a location in a first frame of the second image frames, and

detecting the object in the first frame of the second image frames using the projected location of the object.

21. The method of claim 20, performing stereo analysis of the object when the object is tracked using first and second image frames, and wherein at a third time, the object is outside the field of view of the first camera, such that at least a portion of the object is outside a third frame of the first frames, and at the third time, estimating a distance to the object by processing the second image frames and using information obtained from the stereo analysis.

22. The method of claim 20, further comprising initiating a search for the object in frames of the second image frames following detection of the object in the first frame of the first image frames, the search for the object in frames of the second image frames uses a projection of a location of the object in one or more frames of the first image frames onto a location in one or more frames of the second image frames.

23. A non-transitory computer readable medium storing a program causing a computer to execute a processing method for use in a system, the system comprising a first camera configured to travel with a vehicle and a second camera configured to travel with a vehicle, the first camera having a first field of view and the second camera having a second field of view, the first field of view being narrower than the second field of view, the method comprising:

detecting an object at a first time using a first frame of first image frames acquired via the first camera,

tracking the object using the first image frames,

projecting, at a second time, a location of the object in a second frame of the first image frames onto a location in a first frame of second image frames acquired via the second camera, and

detecting the object in the first frame of the second image frames using the projected location of the object.