Forward looking sensor system

• US 8,184,159 B2
• Filed: 03/26/2007
• Issued: 05/22/2012
• Est. Priority Date: 03/26/2007
• Status: Active Grant

• Current Assignee: TRW Automotive USLLC (Zeppelin - Stiftung der Stadt Friedrichshafen)
• Original Assignee: TRW Automotive USLLC (Zeppelin - Stiftung der Stadt Friedrichshafen)
• Inventors: Luo, Yun
• Primary Examiner(s): Etienne, Ario
• Assistant Examiner(s): KIM, HEE SOO
• Application Number:
  US11/728,547
• Publication Number:
  US 20080239076A1
• Time in Patent Office: 1,884 Days
• Field of Search: 348/148
• US Class Current: 348/148
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/582 of traffic signs
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Systems and methods are provided for controlling a plurality of vehicle systems. A forward looking sensor (12) is configured to obtain a two-dimensional image of a space in front of the vehicle. A region identification element (14) divides the two-dimensional image into a plurality of regions. Each region represents a portion of the space forward of the vehicle. An image transform (16) transforms the two-dimensional image to a three-dimensional coordinate system representing the space in front of the vehicle to calculate depth information for at least a portion of the image. Each of a plurality of expert systems (18-20) have at least one associated region and control a vehicle system (22-24) according to data from their respective at least one associated region and the calculated depth information.

Claims

1. A forward looking vision system for a vehicle, comprising:

a forward looking sensor configured to obtain a two-dimensional image of a space in front of the vehicle;

a region identification element that divides the two-dimensional image into a plurality of regions, each region representing a portion of the space in front of the vehicle, the region identification element comprising;

a sub-region location element that determines a plurality of subregions having homogenous characteristics within the two-dimensional image; and

a clustering element that performs a clustering analysis to combine spatially proximate subregions having similar characteristics to form the plurality of regions;

an image transform element that transforms the two-dimensional image to a three-dimensional coordinate system representing the space in front of the vehicle to calculate depth information for at least a portion of the image; and

a plurality of expert systems, each expert system having at least one associated region and each region having at least one associated expert system, that control respective vehicle functions according to data from their respective associated at least one associated regions and the calculated depth information

wherein the region identification element is operative to process sub-regions above a horizon line of the two-dimensional image separately from sub-regions beneath the horizon line such that the clustering element performs separate clustering analyses for the area above the horizon line to detect a sky region and the area below the horizon line to detect a road region, with any portion of the image not included in the sky region and road region combined into a side region.

2. The system of claim 1, further comprising a road modeling component that determines the location and geometry of a road within the two dimensional image, the region identification element using the determined road location in determining at least one of the plurality of regions.

3. The system of claim 1, the region identification element dividing the image into a road region, representing a roadway in front of the vehicle, a sky region, representing the sky, and a side region, representing objects at the side of the roadway.

4. The system of claim 3, the plurality of expert systems comprising a headway detection system that utilizes data from the road region and side region of the image to determine the distance of objects within the roadway from the vehicle.

5. The system of claim 3, the plurality of expert systems comprising a road type recognition system that is operative to recognize road types and conditions from features extracted from the road region of the image.

6. The system of claim 3, the plurality of expert systems comprising a pedestrian recognition system that is operative to evaluate image data from the side region to notify a driver of pedestrians near the path of travel.

7. The system of claim 3, the plurality of expert systems comprising a fog lamp detection system that utilizes image data from the road region and the sky region to detect low visibility conditions due to fog and adjusts fog lamps on the vehicle accordingly.

8. The system of claim 3, the plurality of expert systems comprising a sign recognition system that can evaluate image data from the side region to notify a driver of signs along the path of travel.

9. The system of claim 3, the plurality of expert systems comprising a dynamic curve speed warning system that is operative to adaptively calculate the curvatures and limited lateral acceleration of the road from image data from the road region and data from at least one other sensor.

10. The system of claim 1, the plurality of expert systems comprising a head light status classifier that evaluates the two-dimensional image to determine if it is appropriate to activate head lights on the vehicle.

11. The system of claim 1, the plurality of expert systems comprising a high beam status classifier that evaluates the two-dimensional image to detect at least one of headlights from oncoming vehicles, taillights from preceding vehicles, and other lights within the two-dimensional image to determine if it is appropriate to activate high beams on the vehicle.

12. A method for controlling vehicle systems, comprising:

imaging the space in front of a vehicle to obtain a two-dimensional image;

dividing the two-dimensional image into sky, road, and side regions, each region representing a portion of the space in front of the vehicle, wherein dividing the two-dimensional image comprises;

locating a horizon line within the two-dimensional image;

determining a plurality of subregions having homogenous characteristics within the two-dimensional image;

performing a first clustering analysis on subregions above the horizon line to combine spatially proximate subregions having similar characteristics to form the sky region; and

performing a second clustering analysis on subregions below the horizon line to combine spatially proximate subregions having similar characteristics to form the road and side regions;

producing an image transform of the two-dimensional image into a three-dimensional coordinate system representing the space forward of the vehicle to calculate depth information for at least a portion of the image;

controlling a first vehicle function with a first expert system that utilizes image data and calculated depth information from a first region of the plurality of regions; and

controlling a second vehicle function with a second expert system that utilizes image data and calculated depth information from a second region of the plurality of regions.

13. The method of claim 12, wherein producing an image transform of the two-dimensional image comprises determining depth information along a ground plane of an image, and controlling a first expert system comprises:

locating an object within the first region of the two-dimensional image;

determining an intersection of the object with the ground plane; and

determining a distance for the object according to the vertical position of the determined intersection within the image.

14. The method of claim 12, wherein dividing the two-dimensional image further comprises:

locating a road boundary within the two-dimensional image;

and

determining at least one region from the located road boundary and the located horizon line.

15. A non-transitory computer readable medium, containing computer executable instructions for processing two-dimensional image data from a forward looking sensor within a vehicle, comprising:

a region identification element that divides the two-dimensional image data into a plurality of regions, with each region representing an area in front of the vehicle, the region identification element comprising;

a texture classification element that determines a plurality of subregions having homogenous characteristics within the two-dimensional image data; and

a clustering element that performs a first clustering analysis to combine spatially proximate subregions having similar characteristics above a horizon line to form a sky region and a second clustering analysis to combine spatially proximate subregions having similar characteristics below the horizon line to form a road and side regions;

an image transform that transforms the two-dimensional image to a three-dimensional coordinate system representing the space forward of the vehicle to calculate depth information for at least a determined ground plane of the image; and

a plurality of expert systems, each expert system having at least one associated region and each region having at least one associated expert system, that control respective vehicle functions according to data from their respective associated at least one associated regions and the calculated depth information.

16. The computer readable medium of claim 15, a first expert system of the plurality of expert systems comprising:

a region growing element that locates an object within a first region of the plurality of regions and determines an intersection of the object with the ground plane; and

a distance calculation element that determines a distance of the object from the vehicle according to the vertical position of the determined intersection within the image.