Systems and methods for visualizing potential risks

• US 10,134,279 B1
• Filed: 06/16/2017
• Issued: 11/20/2018
• Est. Priority Date: 05/05/2017
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation), Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Toyoda, Heishiro, Hamada, Hiroto
• Primary Examiner(s): Swarthout, Brent
• Application Number:
  US15/625,595
• Publication Number:
  US 20180322783A1
• Time in Patent Office: 522 Days
• Field of Search: None
• CPC Class Codes: B60K 2360/186 Displaying information according to relevancy
• CPC Class Codes: B60K 2360/334 Projection means
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/29 Instruments characterised by the way in which information is handled, e.g. showing information on plural displays or prioritising information according to driving conditions
• CPC Class Codes: B60W 2554/4046 Behavior, e.g. aggressive or erratic
• CPC Class Codes: B60W 30/08 Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
• CPC Class Codes: B60W 30/0956 the prediction being responsive to traffic or environmental parameters
• CPC Class Codes: B60W 40/04 Traffic conditions
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: B60W 60/00276 for two or more other traffic participants
• CPC Class Codes: B62D 15/029 Steering assistants using warnings or proposing actions to the driver without influencing the steering system
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G07C 5/0825 using optical means
• CPC Class Codes: G07C 5/0841 Registering performance data recording measured values G01D; information storage G11B
• CPC Class Codes: G08G 1/0112 : from the vehicle, e.g. floating car data [FCD]
• CPC Class Codes: G08G 1/0962 : having an indicator mounted inside the vehicle, e.g. giving voice messages
• CPC Class Codes: G08G 1/16 : Anti-collision systems road vehicle drive control systems for predicting or avoiding probable or impending collision otherwise than by control of a particular sub-unit B60W30/08
• CPC Class Codes: G08G 1/166 : for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

System, methods, and other embodiments described herein relate to inducing awareness in a driver about a surrounding environment of a vehicle using an augmented reality (AR) system within the vehicle. In one embodiment, a method includes identifying one or more potential hazards to the vehicle in the surrounding environment from sensor data collected from at least one sensor of the vehicle. The method includes rendering, within the AR system, a display scenario about the one or more potential hazards by displaying one or more graphical elements that correlate with locations of the one or more potential hazards in the surrounding environment.

Claims

1. An engagement system for inducing awareness in a driver about a surrounding environment of a vehicle using an augmented reality (AR) system within the vehicle, comprising:

one or more processors;

a memory communicably coupled to the one or more processors and storing;

a monitoring module including instructions that when executed by the one or more processors cause the one or more processors to identify one or more potential hazards to the vehicle in the surrounding environment from sensor data collected from at least one sensor of the vehicle,

wherein the potential hazards include contextual hazards that are likelihoods of risk relating to combinations of properties of the surrounding environment and positions of objects within the surrounding environment forming circumstances that define the contextual hazards; and

an engagement module including instructions that when executed by the one or more processors cause the one or more processors to render, within the AR system, a display scenario about the one or more potential hazards by displaying one or more graphical elements that correlate with locations of the one or more potential hazards in the surrounding environment,

wherein the engagement module includes instructions to render the display scenario associated with one of the contextual hazards by animating an undetected object according to a likelihood that the undetected object is present in an occluded area associated with one or more of the locations, and wherein animating the undetected object includes displaying the one or more graphical elements according to a type of the undetected object.

2. The engagement system of claim 1, wherein the potential hazards are likelihoods of risk defined according to detected objects and the circumstances in the surrounding environment,

wherein the monitoring module includes instructions to identify the one or more potential hazards by analyzing the sensor data according to a hazard model that identifies the likelihoods of risk associated with hazardous encounters involving the vehicle according to aspects of the detected objects, associations between the detected objects, and contextual information about the surrounding environment, and

wherein the engagement module further includes instructions to render the one or more graphical elements by generating the graphical elements to communicate a control behavior for the vehicle to avoid the one or more potential hazards and why the one or more potential hazards represent a risk to the vehicle.

3. The engagement system of claim 2, wherein the monitoring module includes instructions to identify the one or more potential hazards by analyzing the sensor data using at least the hazard model that accepts the sensor data as an electronic input and identifies the one or more potential hazards according to the hazard model.

4. The engagement system of claim 2, wherein the potential hazards are comprised of the contextual hazards, and discrete hazards, and

wherein the discrete hazards are openly visible hazards that relate to detected objects that are on an intercepting trajectory with the vehicle.

5. The engagement system of claim 1, wherein the monitoring module further includes instructions to collect the sensor data from the at least one sensor by electronically controlling the at least one sensor to at least semi-continuously scan the surrounding environment to detect objects and obstacles in the surrounding environment, and wherein the sensor data includes environmental information about the surrounding environment, contextual information about properties of a current operation of the vehicle, and driver state information indicating a current condition of the driver.

6. The engagement system of claim 1, wherein the engagement module further includes the instructions to render the display scenario by animating the one or more graphical elements to visually depict the one or more potential hazards in graphical form as a portrayal of the one or more potential hazards presently occurring,

and

wherein the engagement module includes instructions to animate the graphical elements as an overlay on the surrounding environment using an AR display of the AR system within the vehicle.

7. The engagement system of claim 6, wherein the engagement module further includes the instructions to render the display scenario by rendering:

animations of objects intersecting with a trajectory of the vehicle from unexpected locations, animations of the objects moving from behind blind corners, animations of the objects within zones proximate to the vehicle, animations of one of the objects moving along a diverging path of movement that is contrary to a current path of movement, and animations that highlight the zones within the surrounding environment associated with the potential hazards.

8. The engagement system of claim 1, wherein the AR system includes at least one display that is integrated with a windshield of the vehicle.

9. A non-transitory computer-readable medium for inducing awareness in a driver about a surrounding environment of a vehicle using an augmented reality (AR) system within the vehicle and storing instructions that when executed by one or more processors cause the one or more processors to:

identify one or more potential hazards to the vehicle in the surrounding environment from sensor data collected from at least one sensor of the vehicle,

wherein the potential hazards include contextual hazards that are likelihoods of risk relating to combinations of properties of the surrounding environment and positions of objects within the surrounding environment that form circumstances defining the contextual hazards; and

render, within the AR system, a display scenario about the one or more potential hazards by displaying one or more graphical elements that correlate with locations of the one or more potential hazards in the surrounding environment,

wherein the instructions to render include instructions to render the display scenario associated with one of the contextual hazards by animating an undetected object according to a likelihood that the undetected object is present in an occluded area associated with one or more of the locations, and wherein animating the undetected object includes displaying the one or more graphical elements according to a type of the undetected object.

10. The non-transitory computer-readable medium of claim 9, wherein the potential hazards are defined according to likelihoods of risk associated with detected objects and the circumstances in the surrounding environment,

wherein the instructions to identify the one or more potential hazards include instructions to analyze the sensor data according to a hazard model that identifies the likelihoods of risk associated with hazardous encounters involving the vehicle according to aspects of the detected objects, associations between the detected objects, and contextual information about the surrounding environment.

11. The non-transitory computer-readable medium of claim 10, wherein the instructions to identify the one or more potential hazards include instructions to analyze the sensor data using the hazard model and a machine learning algorithm that accepts the sensor data as an electronic input and identifies the one or more potential hazards according to the hazard model, and

wherein the instructions to render the one or more graphical elements include instructions to generate the graphical elements to communicate a control behavior for the vehicle to avoid the one or more potential hazards and why the one or more potential hazards represent a risk to the vehicle.

12. The non-transitory computer-readable medium of claim 9, further comprising instructions to collect the sensor data from the at least one sensor by electronically controlling the at least one sensor to at least semi-continuously scan the surrounding environment to detect objects and obstacles in the surrounding environment, wherein the sensor data includes environmental information about the surrounding environment, contextual information about properties of a current operation of the vehicle, and driver state information indicating a current condition of the driver.

13. The non-transitory computer-readable medium of claim 9, wherein the instructions to render the display scenario include instructions to animate the one or more graphical elements to visually depict the one or more potential hazards in graphical form as a portrayal of the one or more potential hazards as presently occurring,

and

wherein the instructions to the animate the graphical elements include instructions to animate the graphical elements as an overlay on the surrounding environment using an AR display of the AR system within the vehicle.

14. A method for inducing awareness in a driver about a surrounding environment of a vehicle using an augmented reality (AR) system within the vehicle, comprising:

identifying one or more potential hazards to the vehicle in the surrounding environment from sensor data collected from at least one sensor of the vehicle,

wherein the potential hazards include contextual hazards that are likelihoods of risk relating to combinations of properties of the surrounding environment and positions of objects within the surrounding environment forming circumstances that define the contextual hazards; and

rendering, within the AR system, a display scenario about the one or more potential hazards by displaying one or more graphical elements that correlate with locations of the one or more potential hazards in the surrounding environment,

wherein rendering the display scenario includes rendering the display scenario associated with one of the contextual hazards by animating an undetected object according to a likelihood that the undetected object is present in an occluded area associated with one or more of the locations, and wherein animating the undetected object includes displaying the one or more graphical elements according to a type of the undetected object.

15. The method of claim 14, wherein the potential hazards are defined according to likelihoods of risk associated with detected objects and the circumstances in the surrounding environment,

wherein identifying the one or more potential hazards includes analyzing the sensor data according to a hazard model that identifies the likelihoods of risk associated with hazardous encounters involving the vehicle according to aspects of the detected objects, associations between the detected objects, and contextual information about the surrounding environment.

16. The method of claim 15, wherein identifying the one or more potential hazards includes analyzing the sensor data using the hazard model and a machine learning algorithm that accepts the sensor data as an electronic input and identifies the one or more potential hazards according to the hazard model, and

wherein rendering the one or more graphical elements includes generating the graphical elements to communicate a control behavior for the vehicle to avoid the one or more potential hazards and why the one or more potential hazards represent a risk to the vehicle.

17. The method of claim 15, wherein the potential hazards are comprised of the contextual hazards, and discrete hazards, and wherein the discrete hazards are openly visible hazards that relate to detected objects that are on intercepting trajectories with the vehicle.

18. The method of claim 14, further comprising:

collecting the sensor data from the at least one sensor by electronically controlling the at least one sensor to at least semi-continuously scan the surrounding environment to detect objects and obstacles in the surrounding environment, wherein the sensor data includes environmental information about the surrounding environment, contextual information about properties of a current operation of the vehicle, and driver state information indicating a current condition of the driver.

19. The method of claim 14, wherein rendering the display scenario includes animating the one or more graphical elements to visually depict the one or more potential hazards in graphical form as a forecasted portrayal of the one or more potential hazards presently occurring,

and

wherein the animating overlays the graphical elements on the surrounding environment using an AR display within the vehicle.

20. The method of claim 19, wherein rendering the display scenario includes rendering animations of objects intersecting with a projected trajectory of the vehicle from unexpected locations, animations of objects projected to move from behind blind corners, animations of objects within zones proximate to the vehicle, and animations that highlight the zones within the surrounding environment associated with the potential hazards.