Driver's aid using image processing

• US 6,727,807 B2
• Filed: 12/14/2001
• Issued: 04/27/2004
• Est. Priority Date: 12/14/2001
• Status: Expired due to Fees

• Current Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Original Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Inventors: Trajkovic, Miroslav, Gutta, Srinivas, Colmenarez, Antonio
• Primary Examiner(s): Hofsass, Jeffrey
• Assistant Examiner(s): Previl, Daniel
• Application Number:
  US10/017,376
• Publication Number:
  US 20030112132A1
• Time in Patent Office: 865 Days
• Field of Search: 340/436, 340/435, 340/438, 340/995.13, 340/426.14, 340/426.15, 340/426.16, 340/471, 340/472, 340/468, 340/988, 340/995.1, 340/576, 340/439, 340/903
• US Class Current: 340/436
• CPC Class Codes: B60K 2360/785 on or in relation to the windshield or windows
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/60 Instruments characterised by their location or relative disposition in or on vehicles arrangements of lighting devices on dashboards B60Q3/10
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 2300/205 using a head-up display
• CPC Class Codes: B60R 2300/30 characterised by the type of image processing
• CPC Class Codes: B60R 2300/302 combining image information with GPS information or vehicle data, e.g. vehicle speed, gyro, steering angle data
• CPC Class Codes: B60R 2300/308 by overlaying the real scene, e.g. through a head-up display on the windscreen

Abstract

A system and method for alerting a driver of an automobile to a traffic condition. The system comprises at least one camera having a field of view and facing in the forward direction of the automobile. The camera captures images of the field of view in front of the automobile. A control unit receives images of the field of view from the camera and identifies objects therein of a predetermined type. The control unit analyzes the object images of at least one predetermined type to determine whether one or more of the identified objects present a condition that requires the driver'"'"'s response. A display receives control signals from the control unit regarding the one or more of the objects that present a condition that requires the driver'"'"'s response. The display displays an image of the object to the driver that is positioned and scaled so that it overlays the actual object as seen by the driver, the displayed image of the object enhancing a feature of the actual object to alert the driver.

Claims

1. A system for alerting a driver of an automobile to a traffic condition, the system comprising:

at least one camera having a field of view and facing in the forward direction of the automobile, said at least one camera capturing images of the field of view in front of the automobile;

a control unit for receiving the images of the field of view captured by the at least one camera, said control unit identifying objects of at least one predetermined type in the captured images, and analyzing the identified objects to determine whether any of the identified objects present a condition requiring a response by the driver; and

a display for receiving control signals from the control unit regarding one or more of said identified objects which have been determined to present a condition requiring a response by the driver, and for displaying an image of said one or more of said identified objects to the driver, said displayed image being positioned and scaled so that said displayed image overlays a corresponding actual object as seen by the driver, the displayed image of the object enhancing a feature of the corresponding actual object to alert the driver,

wherein the control unit identifies features in the objects of at least one predetermined type, said features being borders of a roadway in the image, and the control unit determines whether a movement of the automobile is inconsistent with the borders of the roadway.

2. A system for alerting a driver of an automobile to a traffic condition, the system comprising:

at least one camera having a field of view and facing in the forward direction of the automobile, said at least one camera capturing images of the field of view in front of the automobile;

a control unit for receiving the images of the field of view captured by the at least one camera, said control unit identifying objects of at least one predetermined type in the captured images, and analyzing the identified objects to determine whether any of the identified objects present a condition requiring a response by the driver; and

a display for receiving control signals from the control unit regarding one or more of said identified objects which have been determined to present a condition requiring a response by the driver, and for displaying an image of said one or more of said identified objects to the driver, said displayed image being positioned and scaled so that said displayed image overlays a corresponding actual object as seen by the driver, the displayed image of the object enhancing a feature of the corresponding actual object to alert the driver,

wherein the control unit identifies objects of at least one predetermined type in the captured images are pedestrians in the image, and the control unit determines whether the pedestrian and the automobile are on a collision course.

3. A system for alerting a driver of an automobile to a traffic condition, the system comprising:

at least one camera having a field of view and facing in the forward direction of the automobile, said at least one camera capturing images of the field of view in front of the automobile;

a control unit for receiving the images of the field of view captured by the at least one camera, said control unit identifying objects of at least one predetermined type in the captured images, and analyzing the identified objects to determine whether any of the identified objects present a condition requiring a response by the driver; and

a display for receiving control signals from the control unit regarding one or more of said identified objects which have been determined to present a condition requiring a response by the driver, and for displaying an image of said one or more of said identified objects to the driver, said displayed image being positioned and scaled so that said displayed image overlays a corresponding actual object as seen by the driver, the displayed image of the object enhancing a feature of the corresponding actual object to alert the driver,

wherein the objects of at least one predetermined type in the captured images traffic signs in the image, and the control unit determines whether a movement of the automobile is inconsistent with the traffic signs.

4. A system for alerting a driver of an automobile to a traffic condition, the system comprising:

at least one camera having a field of view and facing in the forward direction of the automobile, said at least one camera capturing images of the field of view in front of the automobile;

a control unit for receiving the images of the field of view captured by the at least one camera, said control unit identifying objects of at least one predetermined type in the captured images, and analyzing the identified objects to determine whether any of the identified objects present a condition requiring a response by the driver; and

a display for receiving control signals from the control unit regarding one or more of said identified objects which have been determined to present a condition requiring a response by the driver, and for displaying an image of said one or more of said identified objects to the driver, said displayed image being positioned and scaled so that said displayed image overlays a corresponding actual object as seen by the driver, the displayed image of the object enhancing a feature of the corresponding actual object to alert the driver,

wherein the objects of at least one predetermined type in the captured images are traffic signals in the image, and the control unit determines whether a movement of the automobile is inconsistent with the traffic signals.

5. A system for alerting a driver of an automobile to a traffic condition, the system comprising:

at least one camera having a field of view and facing in the forward direction of the automobile, said at least one camera capturing images of the field of view in front of the automobile;

a control unit for receiving the images of the field of view captured by the at least one camera, said control unit identifying objects of at least one predetermined type in the captured images, and analyzing the identified objects to determine whether any of the identified objects present a condition requiring a response by the driver; and

a display for receiving control signals from the control unit regarding one or more of said identified objects which have been determined to present a condition requiring a response by the driver, and for displaying an image of said one or more of said identified objects to the driver, said displayed image being positioned and scaled so that said displayed image overlays a corresponding actual object as seen by the driver, the displayed image of the object enhancing a feature of the corresponding actual object to alert the driver,

wherein the system comprises two cameras each facing in the forward direction of the automobile from different positions and each having a different respective field of view, the two cameras each capturing images of their respective field of view in front of the automobile.

6. The system as claimed in claim 5, wherein the control unit receives the images of the field of views from the respective cameras, identifies objects of at least one predetermined type in the images from each of the respective cameras, and analyzes the objects in the images received from each of the respective cameras to determine whether one or more of the identified objects presents a condition that requires a response by the driver.

7. The system as claimed in claim 6, wherein, when the control unit determines that an identified object presents a condition that requires a response by the driver, corresponding images of the identified object in the images received from each of the respective cameras are used by the control unit to determine a distance between a point on the automobile and the identified object.

8. The system as claimed in claim 7, wherein the control unit determines the distance between a point on the automobile and the object using stereo techniques.

9. The system as claimed in claim 8, wherein the point on the automobile is located approximately at a position of the driver'"'"'s head when driving the automobile.

10. The system as claimed in claim 9, wherein the control unit generates control signals for the display unit designating the position and scale of the image of the object using the distance between the driver'"'"'s head and the object.

11. A method for alerting a driver of an automobile to a traffic condition, the method comprising the steps:

capturing images of a field of view in front of the automobile;

identifying objects in the images of at least one predetermined type, and analyzing the objects to determine whether one or more of the identified objects present a condition that requires a response of the driver; and

displaying to the driver an image of at least one of the identified objects requiring the driver'"'"'s response, the displayed image being positioned and scaled so that displayed image of the object overlays a corresponding actual object as seen by the driver, the displayed image of the object enhancing a feature of the corresponding actual object to alert the driver,

wherein the step of identifying objects in the image of at least one predetermined type and analyzing the objects to determine whether one or more of the identified objects present a condition that requires the driver'"'"'s response, includes identifying at least one of pedestrians and other automobiles in the image, and determining whether any identified pedestrians and other automobiles are on a collision course with the automobile.

12. A method for alerting a driver of an automobile to a traffic condition, the method comprising the steps:

capturing images of a field of view in front of the automobile;

identifying objects in the images of at least one predetermined type, and analyzing the objects to determine whether one or more of the identified objects present a condition that requires a response of the driver; and

displaying to the driver an image of at least one of the identified objects requiring the driver'"'"'s response, the displayed image being positioned and scaled so that displayed image of the object overlays a corresponding actual object as seen by the driver, the displayed image of the object enhancing a feature of the corresponding actual object to alert the driver,

wherein the step of identifying objects in the image of at least one predetermined type and analyzing the objects to determine whether one or more of the identified objects present a condition that requires the driver'"'"'s response, includes identifying at least one of traffic signs and traffic signals in the image, and determining whether the movement of the automobile is inconsistent with any identified traffic signs and traffic signals.