SAFETY SYSTEM FOR AUGMENTING ROADWAY OBJECTS ON A HEADS-UP DISPLAY

• US 20160098861A1
• Filed: 10/06/2014
• Published: 04/07/2016
• Est. Priority Date: 10/06/2014
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Inventors: SISBOT, Emrah Akin, YALLA, Veera Ganesh
• Granted Patent: US 9,472,023 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/582 of traffic signs
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G08G 1/09623 Systems involving the acquisition of information from passive traffic signs by means mounted on the vehicle G08G1/0967 takes precedence
• CPC Class Codes: G08G 1/09626 where the origin of the information is within the own vehicle, e.g. a local storage device, digital map
• CPC Class Codes: G08G 1/096775 where the origin of the information is a central station
• CPC Class Codes: G08G 1/096791 where the origin of the information is another vehicle

Abstract

The disclosure includes a system and method for providing a heads-up display in a vehicle for alerting users to roadway objects. The system includes a processor and a memory storing instructions that, when executed, cause the system to: monitor, with the one or more processors, for a roadway object; detect, with the one or more processors, the roadway object; generate graphical data configured to emphasize the roadway object on a heads-up display; determine whether a user sees the roadway object; responsive to the user failing to see the roadway object, determine whether the roadway object is critical; and responsive to the roadway object being critical, enhance a graphic output on the heads-up display to get the user'"'"'s attention.

Claims

1. A computer-implemented method comprising:

monitoring, with one or more processors, for a roadway object;

detecting, with the one or more processors, the roadway object;

generating graphical data configured to emphasize the roadway object on a heads-up display;

determining whether a user sees the roadway object;

responsive to the user failing to see the roadway object, determining whether the roadway object is critical; and

responsive to the roadway object being critical, enhancing a graphic output on the heads-up display to get the user'"'"'s attention.

2. The method of claim 1, further comprising:

responsive to enhancing the graphic output on the heads-up display to get the user'"'"'s attention, determining whether the user saw the roadway traffic; and

responsive to the user failing to see the roadway object, determining a way to enhance the graphic output on the heads-up display.

3. The method of claim 1, wherein determining whether the user saw the roadway object further comprises:

receiving a first recording of the user while driving and a second recording of the road;

identifying eye movement of the user in the first recording; and

correlating the eye movement in the first recording with the roadway object in the second recording to determine whether the user saw the roadway object.

4. The method of claim 1, wherein determining whether the roadway object is critical further comprises:

determining a distance between a client device and a location referenced by the roadway object; and

identifying the roadway object as being critical when the distance falls below a threshold.

5. The method of claim 1, wherein the graphical data on the heads-up display includes representing the user'"'"'s field of view in a spherical coordinate system, representing the roadway object with an angle of theta, an angle of phi, and a distance of radius.

6. The method of claim 1, wherein the roadway object is a roadway traffic sign indicating a regulation applicable to driving on the roadway.

7. The method of claim 1, further comprising:

analyzing data describing roadway objects that are overlooked by drivers;

determining one or more roadway objects that are overlooked by drivers;

determining clusters of roadway objects overlooked by drivers;

generating a map describing roadway objects overlooked by drivers; and

determining a way to increase roadway safety using the map describing overlooked roadway objects.

8. The method of claim 7, wherein the way to increase roadway safety using the map includes highlighting the overlooked roadway objects on the user'"'"'s heads-up display.

9. The method of claim 1, wherein determining whether the roadway object is critical is further based on an update from a social network application.

10. A computer program product comprising a non-transitory computer-usable medium including a computer-readable program, wherein the computer-readable program when executed on a computer causes the computer to:

monitor for a roadway object;

detect the roadway object;

generate graphical data configured to emphasize the roadway object on a heads-up display;

determine whether a user sees the roadway object;

responsive to the user failing to see the roadway object, determine whether the roadway object is critical; and

responsive to the roadway object being critical, enhance a graphic output on the heads-up display to get the user'"'"'s attention.

11. The computer program product of claim 10, wherein the computer-readable program when executed on a computer is further configured to cause the computer to:

receive a first recording of the user while driving and a second recording of the road;

identify eye movement of the user in the first recording; and

correlate the eye movement in the first recording with the roadway object in the second recording to determine whether the user saw the roadway object.

12. The computer program product of claim 10, wherein the graphical data on the heads-up display includes representing the user'"'"'s field of view in a spherical coordinate system, representing the roadway object with an angle of theta, an angle of phi, and a distance of radius.

13. The computer program product of claim 10, wherein the computer-readable program when executed on a computer is further configured to cause the computer to:

analyze data describing roadway objects that are overlooked by drivers;

determine one or more roadway objects that are overlooked by drivers;

determine clusters of roadway objects overlooked by drivers;

generate a map describing roadway objects overlooked by drivers; and

determine a way to increase roadway safety using the map describing overlooked roadway objects.

14. The computer program product of claim 10, wherein determining whether the roadway object is critical is further based on an update from a social network application.

15. A system comprising:

a processor; and

a memory storing instructions that, when executed, cause the system to;

monitor for a roadway object;

detect the roadway object;

generate graphical data configured to emphasize the roadway object on a heads-up display;

determine whether a user sees the roadway object;

responsive to the user failing to see the roadway object, determine whether the roadway object is critical; and

responsive to the roadway object being critical, enhance a graphic output on the heads-up display to get the user'"'"'s attention.

16. The system of claim 15, wherein the instructions when executed cause the system to also:

receive a first recording of the user while driving and a second recording of the road;

identify eye movement of the user in the first recording; and

correlate the eye movement in the first recording with the roadway object in the second recording to determine whether the user saw the roadway object.

17. The system of claim 15, wherein the graphical data on the heads-up display includes representing the user'"'"'s field of view in a spherical coordinate system, representing the roadway object with an angle of theta, an angle of phi, and a distance of radius.

18. The system of claim 15, wherein the instructions when executed cause the system to also:

analyze data describing roadway objects that are overlooked by drivers;

determine one or more roadway objects that are overlooked by drivers;

determine clusters of roadway objects overlooked by drivers;

generate a map describing roadway objects overlooked by drivers; and

determine a way to increase roadway safety using the map describing overlooked roadway objects.

19. The system of claim 18, wherein the way to increase roadway safety using the map includes highlighting the overlooked roadway objects on the user'"'"'s heads-up display.

20. The system of claim 18, wherein determining whether the roadway object is critical is further based on an update from a social network application.