Active warning system using the detection of driver awareness of traffic signs

• US 9,626,866 B2
• Filed: 08/12/2014
• Issued: 04/18/2017
• Est. Priority Date: 08/12/2014
• Status: Active Grant

• Current Assignee: Continental Automotive Systems, Inc. (Continental AG)
• Original Assignee: Continental Automotive Systems, Inc. (Continental AG)
• Inventors: Waite, Sheldon G
• Primary Examiner(s): WILSON, BRIAN P
• Application Number:
  US14/457,430
• Publication Number:
  US 20160049076A1
• Time in Patent Office: 980 Days
• Field of Search: 340905, 340435, 340436, 340438, 340439, 340457, 340500, 340501, 340540, 3405731, 340575, 340576, 382104, 382181, 348118, 348120
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: A61B 5/18 for vehicle drivers or machine operators
• CPC Class Codes: A61B 5/746 Alarms related to a physiological condition, e.g. details of setting alarm thresholds or avoiding false alarms alarms per se G08B
• CPC Class Codes: B60K 28/066 actuating a signalling device B60K28/063 takes precedence
• CPC Class Codes: B60W 2040/0863 due to erroneous selection or response of the driver
• CPC Class Codes: B60W 2555/60 Traffic rules, e.g. speed limits or right of way
• CPC Class Codes: B60W 2710/182 Brake pressure, e.g. of fluid or between pad and disc
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: G06V 20/582 of traffic signs
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G08G 1/09623 Systems involving the acquisition of information from passive traffic signs by means mounted on the vehicle G08G1/0967 takes precedence
• CPC Class Codes: G08G 1/096725 where the received information generates an automatic action on the vehicle control
• CPC Class Codes: G08G 1/096783 where the origin of the information is a roadside individual element
• CPC Class Codes: G08G 1/165 for passive traffic, e.g. including static obstacles, trees

Abstract

Traffic sign warnings are provided to a driver by optically scanning an area in front of the moving vehicle to obtain an image of said area in front of the moving vehicle. Road signs are recognized and their importance categorized. A driver'"'"'s eyes are continuously scanned to determine whether the driver'"'"'s eyes'"'"' ever focus on, or are directed to a detected and recognized road sign. Multiple different visual and/or audible warnings or alarms are generated based on the nature of a road sign, its proximity to the moving vehicle, and the driver'"'"'s determined awareness of the road sign.

Claims

1. A method of providing traffic sign warnings to a driver of a moving vehicle, the method comprising:

optically scanning an area in front of the moving vehicle to obtain an image of said area in front of the moving vehicle;

detecting and recognizing a road sign in said image;

determining a location of the recognized road sign relative to the moving vehicle by localizing the recognized road sign in three-dimensions using geographic coordinates associated with the moving vehicle and a distance between the moving vehicle and the recognized road sign obtained from the image;

determining a first vehicle operation parameter that should be changed responsive to the recognized road sign;

optically scanning the driver'"'"'s head and eyes;

calculating at least one focal point of the driver'"'"'s eyes from said optically scanning the driver'"'"'s head and eyes;

determining whether the driver has seen the recognized road sign from the calculated at least one focal point of the driver'"'"'s eyes and the determined location of the recognized road sign relative to the moving vehicle;

determining whether the driver has changed a vehicle operation parameter;

if the driver has not changed the vehicle operation parameter, providing a first warning to the driver responsive to the recognized road sign; and

updating a database of road sign locations with the determined location of the recognized road sign.

2. The method of claim 1, wherein determining whether the driver has seen the recognized road sign comprises:

comparing the calculated focal point of the driver'"'"'s eyes to the determined location of the recognized road sign; and

determining the recognized road sign to have been recognized if;

the calculated focal point of the driver'"'"'s eyes and the determined location of the recognized road sign are substantially the same; and

the driver has changed or is changing the vehicle operation parameter that is consistent with the recognized road sign.

3. The method of claim 1, wherein determining the first vehicle operation parameter that should be changed responsive to the recognized road sign comprises:

ranking the recognized road sign according to at least one of;

importance;

contextual relevance;

temporal relevance; and

urgency.

4. The method of claim 1, wherein providing the first warning comprises providing a visible indicator on a vehicle control panel.

5. The method of claim 1, wherein providing the first warning comprises providing an audible alarm inside a passenger compartment of the moving vehicle.

6. The method of claim 1, further comprising automatically changing the first vehicle operation parameter after providing the first warning to the driver.

7. The method of claim 6, wherein automatically changing the first vehicle operation parameter comprises automatically applying vehicle brakes.

8. An apparatus for providing traffic sign warnings to a driver of a vehicle when the vehicle is moving, the apparatus comprising:

a first camera configured to scan an area in front of the vehicle to obtain an image;

a road sign detector and recognizer coupled to the first camera, the road sign detector and recognizer configured to detect and recognize a road sign in said image;

a road sign location determiner configured to determine a location of the recognized road sign relative to the vehicle by localizing the recognized road sign in three-dimensions using geographic coordinates associated with the vehicle and a distance between the vehicle and the recognized road sign obtained from the image;

a vehicle operation parameter controller configured to determine a first vehicle operation parameter that should be changed responsive to the recognized road sign;

a second camera directed at the driver and configured to optically scan the driver'"'"'s head and eyes and obtain at least one image of the driver'"'"'s eyes;

a driver'"'"'s eye focal point determiner configured to calculate at least one focal point of the driver'"'"'s eyes from said at least one image of the driver'"'"'s eyes;

a recognized road sign determiner, configured to determine whether the driver has seen the recognized road sign from the calculated at least one focal point of the driver'"'"'s eyes and the determined location of the recognized road sign;

an operation parameter determiner configured to determine whether the driver is changing the first vehicle operation parameter;

a controller configured to provide a first warning to the driver if the driver does not change the first vehicle operation parameter responsive to the recognized road sign; and

a database configured to be updated with the determined location of the recognized road sign.

9. The apparatus of claim 8, wherein the road sign detector and recognizer, the road sign location determiner and the vehicle operation parameter controller comprise a computer, operatively coupled to a non-transitory memory device storing program instructions for the computer.

10. The apparatus of claim 8, wherein the driver'"'"'s eye focal point determiner, the recognized road sign determiner and the operation parameter determiner comprise a computer, operatively coupled to a non-transitory memory device storing program instructions for the computer.

11. The apparatus of claim 8, further comprising a display device configured to display a message to the driver responsive to the controller providing the first warning.

12. The apparatus of claim 8, further comprising an audio device configured to provide a predetermined sound inside the vehicle responsive to the controller providing the first warning.