Safety system for augmenting roadway objects on a heads-up display

• US 9,472,023 B2
• Filed: 10/06/2014
• Issued: 10/18/2016
• Est. Priority Date: 10/06/2014
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Inventors: Sisbot, Emrah Akin, Yalla, Veera Ganesh
• Primary Examiner(s): Tryder, Gregory J
• Assistant Examiner(s): Wang, Yuehan
• Application Number:
  US14/507,047
• Publication Number:
  US 20160098861A1
• Time in Patent Office: 743 Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/582 of traffic signs
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G08G 1/09623 Systems involving the acquisition of information from passive traffic signs by means mounted on the vehicle G08G1/0967 takes precedence
• CPC Class Codes: G08G 1/09626 where the origin of the information is within the own vehicle, e.g. a local storage device, digital map
• CPC Class Codes: G08G 1/096775 where the origin of the information is a central station
• CPC Class Codes: G08G 1/096791 where the origin of the information is another vehicle

Abstract

The disclosure includes a system and method for providing a heads-up display in a vehicle for alerting users to roadway objects. The system includes a processor and a memory storing instructions that, when executed, cause the system to: monitor, with the one or more processors, for a roadway object; detect, with the one or more processors, the roadway object; generate graphical data configured to emphasize the roadway object on a heads-up display; determine whether a user sees the roadway object; responsive to the user failing to see the roadway object, determine whether the roadway object is critical; and responsive to the roadway object being critical, enhance a graphic output on the heads-up display to get the user'"'"'s attention.

Claims

1. A computer-implemented method comprising:

monitoring, with one or more processors of a vehicle, for a roadway object based in part on geographical data describing a present geographic location of the vehicle and an electronic map that describes a plurality of clusters along one or more roadways that are included in the electronic map, wherein a cluster includes a geographic area that includes one or more roadway objects that are determined to be frequently overlooked by other users;

detecting the roadway object;

generating graphical data configured to emphasize the roadway object on a heads-up display of the vehicle;

determining whether a user sees the roadway object;

responsive to the user failing to see the roadway object, determining whether the roadway object is critical, wherein the roadway object is determined to be critical based on a ranked hierarchy of different categories of roadway objects and the highest ranked category is a school zone; and

responsive to the roadway object being critical, enhancing a graphic output on the heads-up display to get the user'"'"'s attention.

2. The method of claim 1, further comprising:

responsive to enhancing the graphic output on the heads-up display to get the user'"'"'s attention, determining whether the user saw the roadway traffic; and

responsive to the user failing to see the roadway object, determining a way to enhance the graphic output on the heads-up display.

3. The method of claim 1, wherein determining whether the user saw the roadway object further comprises:

receiving a first recording of the user while driving and a second recording of the road;

identifying eye movement of the user in the first recording; and

correlating the eye movement in the first recording with the roadway object in the second recording to determine whether the user saw the roadway object.

4. The method of claim 1, wherein determining whether the roadway object is critical further comprises:

determining a distance between a client device and a location referenced by the roadway object; and

identifying the roadway object as being critical when the distance falls below a threshold.

5. The method of claim 1, wherein the graphical data on the heads-up display includes representing the user'"'"'s field of view in a spherical coordinate system, representing the roadway object with an angle of theta, an angle of phi, and a distance of radius.

6. The method of claim 1, wherein the roadway object is a roadway traffic sign indicating a regulation applicable to driving on the roadway.

7. The method of claim 1, wherein the ranked hierarchy is customized for the user.

8. The method of claim 7, wherein the customization is based on a driving history of the user.

9. The method of claim 1, wherein determining whether the roadway object is critical is further based on an update from a social network application.

10. A computer program product comprising a non-transitory computer-usable medium including a computer-readable program, wherein the computer-readable program when executed on a computer causes the computer to:

monitor for a roadway object;

detect the roadway object;

generate graphical data configured to emphasize the roadway object on a heads-up display;

determine whether a user sees the roadway object;

responsive to the user failing to see the roadway object, determine whether the roadway object is critical based on a ranked hierarchy of categories, wherein a school zone is the highest ranked category included in the ranked hierarchy, a pedestrian crosswalk sign is a second highest ranked category included in the ranked hierarchy, a stop sign is a third highest ranked category included in the ranked hierarchy, a sign indicating a roadway condition is a fourth highest ranked category included in the ranked hierarchy and a speed limit sign is a fifth highest ranked category included in the ranked hierarchy; and

responsive to the roadway object being critical, enhance a graphic output on the heads-up display to get the user'"'"'s attention.

11. The computer program product of claim 10, wherein the computer-readable program when executed on a computer is further configured to cause the computer to:

receive a first recording of the user while driving and a second recording of the road;

identify eye movement of the user in the first recording; and

correlate the eye movement in the first recording with the roadway object in the second recording to determine whether the user saw the roadway object.

12. The computer program product of claim 10, wherein the graphical data on the heads-up display includes representing the user'"'"'s field of view in a spherical coordinate system, representing the roadway object with an angle of theta, an angle of phi, and a distance of radius.

13. The computer program product of claim 10, wherein the computer-readable program when executed on a computer is further configured to cause the computer to:

analyze data describing roadway objects that are overlooked by drivers;

determine one or more roadway objects that are overlooked by drivers;

determine clusters of roadway objects overlooked by drivers;

generate a map describing roadway objects overlooked by drivers; and

determine a way to increase roadway safety using the map describing overlooked roadway objects.

14. The computer program product of claim 10, wherein determining whether the roadway object is critical is further based on an update from a social network application.

15. A system comprising:

a processor; and

a memory storing instructions that, when executed, cause the system to;

monitor for a roadway object based on an electronic map describing locations for a plurality of roadway objects that are determined to be frequently overlooked by users;

detect the roadway object;

generate graphical data configured to emphasize the roadway object on a heads-up display;

determine whether a user sees the roadway object;

responsive to the user failing to see the roadway object, determine whether the roadway object is critical based on a ranked hierarchy of categories wherein the highest ranked category is a school zone; and

responsive to the roadway object being critical, enhance a graphic output on the heads-up display to get the user'"'"'s attention.

16. The system of claim 15, wherein the instructions when executed cause the system to also:

receive a first recording of the user while driving and a second recording of the road;

identify eye movement of the user in the first recording; and

correlate the eye movement in the first recording with the roadway object in the second recording to determine whether the user saw the roadway object.

17. The system of claim 15, wherein the graphical data on the heads-up display includes representing the user'"'"'s field of view in a spherical coordinate system, representing the roadway object with an angle of theta, an angle of phi, and a distance of radius.

18. The system of claim 15, wherein a school zone is the highest ranked category included in the ranked hierarchy, a pedestrian crosswalk sign is a second highest ranked category included in the ranked hierarchy, a stop sign is a third highest ranked category included in the ranked hierarchy, a sign indicating a roadway condition is a fourth highest ranked category included in the ranked hierarchy and a speed limit sign is a fifth highest ranked category included in the ranked hierarchy.

19. The system of claim 15, wherein enhancing the graphic output on the heads-up display includes highlighting the overlooked roadway objects on the heads-up display.

20. The system of claim 15, wherein the graphic output of the heads-up display is enhanced contemporaneous to the roadway object being viewable by the user in the heads-up display.

21. A computer-implemented method comprising:

receiving object data from a plurality of vehicles driven by a plurality of users, wherein the object data describes a plurality roadway objects that are overlooked by the plurality of users and the geographic location of the plurality of roadway objects;

analyzing, by the server, the object data to identify a plurality of roadway objects that are overlooked by users of the plurality of vehicles;

determining, by the server, one or more clusters of roadway objects that are overlooked by users of the plurality of vehicles, wherein each cluster includes a geographic area where the object data indicates difficulty for the plurality of users to see roadway objects;

generating, by the server, an electronic map including the one or more clusters along one or more roadways that are included in the electronic map, wherein the clusters are associated with a ranked hierarchy of categories wherein the highest ranked category is a school zone; and

transmitting the electronic map to a vehicle, wherein the electronic map is operable so that the vehicle analyzes the electronic map to proactively identify that a journey of the vehicle includes a cluster included in the electronic map and the vehicle proactively uses a heads-up display unit of the vehicle to emphasize one or more roadway objects included in the geographic area of the cluster based at least in part on whether the one or more roadway objects are critical as indicated by the ranked hierarchy.

22. A computer-implemented method comprising:

monitoring, by an onboard computer of a vehicle, for a roadway object based in part on geographical data describing a present geographic location of the vehicle and an electronic map that describes a plurality of clusters along one or more roadways that are included in the electronic map, wherein a cluster includes a geographic area that includes one or more roadway objects that are determined to be frequently overlooked by other users and the one or more roadway objects are associated with a ranked hierarchy of categories wherein the highest ranked category is a school zone;

identifying a roadway object present in one of the plurality of clusters;

proactively generating graphical data configured to emphasize the roadway object on a heads-up display of the vehicle; and

displaying a graphical element on the heads-up display of the vehicle, wherein the graphic element is generated based on the graphical data based in part on whether the roadway object emphasized by the graphic element is critical as indicated by the ranked hierarchy.