Headlight, taillight and streetlight detection

• US 7,566,851 B2
• Filed: 03/22/2007
• Issued: 07/28/2009
• Est. Priority Date: 03/24/2006
• Status: Active Grant

• Current Assignee: MobilEye Vision Technologies Ltd. (Intel Corporation)
• Original Assignee: Mobileye Technologies Limited (Intel Corporation)
• Inventors: Stein, Gideon P., Wolfovitz, Uri, Hadassi, Ofer, Ben Haim, Noam
• Primary Examiner(s): Luu; Thanh X
• APPLICATION: US 20070221822A1
• Time in patent office: 859 Days
• Field of Search: 250/208.1, 340435-437, 382/104, 701/23, 701/28, 701/36, 701/117, 701300-302, 356/213, 356/218, 356/222, 356/229
• US Class Current: 250/205
• CPC Class Codes: B60Q 1/143 combined with another condition, e.g. using vehicle recognition from camera images or activation of wipers
• CPC Class Codes: B60Q 2300/3321 Detection of streetlights
• CPC Class Codes: B60Q 2300/41 preceding vehicle
• CPC Class Codes: B60Q 2300/42 oncoming vehicle
• CPC Class Codes: B60Q 2300/45 Special conditions, e.g. pedestrians, road signs or potential dangers
• CPC Class Codes: G06K 9/00818 Recognising traffic signs
• CPC Class Codes: G06K 9/00825 Recognition of vehicle or traffic lights
• CPC Class Codes: G08G 1/165 for passive traffic, e.g. including static obstacles, trees
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: G08G 1/167 Driving aids for lane monitoring, lane changing, e.g. blind spot detection

Abstract

A method in a computerized system including an image sensor mounted in a moving vehicle. The image sensor captures image frames consecutively in real time. In one of the image flames, a spot is detected of measurable brightness; the spot is matched in subsequent image frames. The image frames are available for sharing between the computerized system and another vehicle control system. The spot and the corresponding spot are images of the same object. The object is typically one or more of headlights from an oncoming vehicle, taillights of a leading vehicle, streetlights, street signs and/or traffic signs. Data is acquired from the spot and from the corresponding spot. By processing the data, the object (or spot) is classified. producing an object classification. The vehicle control system controls preferably headlights of the moving vehicle based on the object classification. The other vehicle control system using the image frames is one or more of: lane departure warning system, collision warning system and/or ego-motion estimation system.

Claims

1. In a computerized vehicle control system including an image sensor mounted on a moving vehicle, wherein the image sensor captures consecutively in real time a plurality of image frames, a method comprising the steps of:

(a) in at least one of the image frames, detecting a spot of measurable brightness;

(b) matching in at least one subsequent image frame of the image frames, a corresponding spot, wherein said spot and said corresponding spot are images of a light source;

(c) acquiring data respectively from said spot and from said corresponding spot,

wherein said data includes at least one of position characteristics, intensity characteristics, size characteristics, and shape characteristics, but not color characteristics, of said light source; and

(d) processing said data, thereby classifying the light source based on said data, and producing a light source classification, wherein substantially all the image frames are available to the computerized vehicle control system and at least one other vehicle control system;

wherein the vehicle control system controls headlights of the moving vehicle based on said light source classification and said at least one other vehicle control system is selected from the group consisting of a lane departure warning system, a collision warning system, and an ego-motion estimation system.

2. The method according to claim 1, wherein said light source is selected from the group consisting of:

headlights from an oncoming vehicle, taillights of a leading vehicle, streetlights, street signs and traffic signs.

3. The method according to claim 1, wherein said light source classification is provided to and used by said at least one other vehicle control system.

4. The method according to claim 1, further comprising the step of:

(e) tracking motion of said spot by comparing respective image frame positions of said spot and said corresponding spot.

5. The method according to claim 4, said classifying said light source not as a street sign when said motion is outward and upward.

6. The method according to claim 4, further comprising the steps of:

(f) deactivating high beams upon said classifying said light source as being a portion of a vehicle selected from the group consisting of a passing vehicle, a preceding vehicle and an oncoming vehicle; and

(g) reactivating high beams based on said tracking motion.

7. The method according to claim 1, wherein said data is related to a shape of said spot, wherein said shape is indicative of said spot splitting into a plurality of spots in said at least one subsequent image frame.

8. The method according to claim 1, further comprising the steps of:

(e) in said at least one image frame, detecting a second spot of measurable brightness; and

(f) tracking motion of said spot and a second motion of said second spot between said at least one image frame and said at least one subsequent image frame.

9. The method according to claim 8, pairing said spot and said second spot based on comparing said motion to said second motion.

10. The method according to claim 8, said classifying said light source as a taillight when said motion and said second motion are inward.

11. The method, according to claim 8, further comprising the step of, prior to said tracking motion:

(g) compensating for yaw motion of the vehicle.

12. In a computerized system including an image sensor mounted on a moving vehicle, wherein the image sensor captures in real time an image frame, a method comprising the steps of:

(a) detecting in the image frame a plurality of spots of measurable brightness, wherein said spots are respective images of a plurality of light sources;

(b) acquiring data from said spots; and

(c) processing said data, thereby classifying said light sources based on said data;

wherein said classifying is performed by previously training with a plurality of known images, and

wherein at least one edge of at least one of said spots is detected and an intensity profile across said at least one edge is used for said classifying.

13. The method according to claim 12, wherein the image sensor includes a filter said filter having a spatial profile including at least one portion preferentially transmitting red light, further comprising the step of:

(d) correlating said spatial profile with at least one of said spots.

14. The method according to claim 12, wherein said known images include images from light sources selected from the group consisting of:

taillights, headlights of an oncoming vehicle, streetlights, and traffic signs.

15. The method according to claim 12, further comprising the steps of:

(d) upon classifying said light sources as taillights of a leading vehicle, determining distance to said leading vehicle; and

(e) adaptively controlling headlights of the moving vehicle based on said distance.

16. The method according to claim 12, further comprising the step of:

(d) upon classifying said light sources as at least three streetlights along a road, determining curvature of said road; and

(e) adaptively controlling headlights of the moving vehicle based on said curvature.

17. The method according to claim 12, wherein said detecting includes determining a quadrant of said spots within said image frame, further comprising the step of:

(d) controlling solely one headlight of the moving vehicle based on said quadrant.

18. The method according to claim 12, wherein said training and said classifying are performed using radial basis functions.

19. The method according to claim 12, wherein said classifying uses pairs of said spots to identify vehicle taillights.

20. The method according to claim 12, wherein said classifying includes spatial alignment of at least a portion of said spots to detect streetlights.

21. The method according to claim 12, wherein texture characteristics of said spots are used for said classifying.

22. The method according to claim 12, wherein for each of said spots, said classifying uses said data from an area centered around each said spot including N by N picture elements of the image frame.

23. The method according to claim 12, further comprising the steps of:

(g) computing a function of image gradients for a patch of intensity of at least one of said spots;

(h) for said at least one spot computing a ratio of the number of pixels above a threshold to the size of said patch; and

(i) if said ratio is below a threshold, said classifying said sign based on said intensity profile across said at least one edge, otherwise if said ratio is below said threshold, delaying said classifying until further image frames from the image sensor are available and processed.

24. In a computerized system including an image sensor mounted on a moving vehicle, wherein the image sensor captures in real time an image frame, a method comprising the steps of:

(a) detecting in the image frame a spot of measurable brightness, wherein said spot is an image of at least one light source, said spot including a cluster of patches of intensity;

(b) acquiring data from said spot; and

(c) processing said data, thereby classifying said at least one light source based on said patches,

wherein said classifying is based on a symmetry of said cluster, wherein said at least one light source is classified as a traffic sign based on a lack of said symmetry, and

wherein the image sensor has a high dynamic range with a linear response corresponding to more than eight bits or by having a nonlinear response to light intensity and wherein said acquiring data is performed using said high dynamic range.

25. The method according to claim 24, wherein said symmetry is radial, said at least one light source is classified as a headlight.

26. The method according to claim 24, wherein said cluster includes two upper and two lower said patches of intensity, said at least one light source is classified as two headlights and two reflections thereof.

27. The method according to claim 24, further comprising the step of, prior to said classifying:

(d) training a classifier on a plurality of image moments of a plurality of clusters imaged from known light sources.