Traffic situation awareness for an autonomous vehicle

• US 10,083,547 B1
• Filed: 05/23/2017
• Issued: 09/25/2018
• Est. Priority Date: 05/23/2017
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Inventors: Tomatsu, Nobuyuki
• Primary Examiner(s): Merouan, Abderrahim
• Application Number:
  US15/603,101
• Time in Patent Office: 490 Days
• CPC Class Codes: B60K 2360/175 Autonomous driving
• CPC Class Codes: B60K 2360/177 Augmented reality
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/211 producing three-dimensional [3D] effects, e.g. stereoscopic images
• CPC Class Codes: B60K 35/22 Display screens
• CPC Class Codes: B60K 35/23 Head-up displays [HUD] optical aspects of head-up displays G02B27/01
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G08G 1/0112 from the vehicle, e.g. floating car data [FCD]
• CPC Class Codes: G08G 1/0137 for specific applications
• CPC Class Codes: G08G 1/096725 where the received information generates an automatic action on the vehicle control
• CPC Class Codes: G08G 1/09675 where a selection from the received information takes place in the vehicle
• CPC Class Codes: G08G 1/096775 where the origin of the information is a central station

Abstract

The disclosure includes embodiments for providing traffic situation awareness for a driver of an autonomous vehicle. In some embodiments, the autonomous vehicle is a highly autonomous vehicle (HAV). A method implemented by the HAV includes, according to some embodiments, analyzing external sensor data recorded by one or more external sensors of the HAV to identify a traffic situation. The external sensor data describes one or more measurements of a physical environment external to the HAV. The method includes generating graphical data describing visual feedback that visually depicts information describing the traffic situation. The method includes providing the graphical data to an interface device to cause the interface device to display the visual feedback. The interface device is communicatively coupled to the HAV and operable to receive the graphical data from the processor of the HAV. The visual feedback provides the traffic situation awareness to the driver of the HAV.

Claims

1. A method implemented by a highly autonomous vehicle (HAV), the method comprising:

analyzing external sensor data recorded by one or more external sensors of the HAV to identify a traffic situation, wherein the external sensor data describes one or more measurements of a physical environment external to the HAV;

determining, based on internal sensor data, view type data describing a view type for a driver of the HAV, wherein the view type is a forward-looking view type or a non-forward looking view type;

generating graphical data describing visual feedback that visually depicts information describing the traffic situation;

generating position data describing a position where the visual feedback is displayed in an electronic display of an interface device based on the internal sensor data and the view type data; and

providing, by a processor of the HAV, the position data and the graphical data to the interface device to cause the interface device to display the visual feedback at the position described by the position data, wherein the interface device is communicatively coupled to the HAV and operable to receive the graphical data from the processor of the HAV.

2. The method of claim 1, wherein the interface device is an augmented reality display device and the graphical data is operable to cause the augmented reality display device to provide an augmented reality experience.

3. The method of claim 2, wherein the augmented reality display device is selected from a group that includes:

a set of augmented reality goggles; and

a set of augmented reality glasses.

4. The method of claim 2, wherein the augmented reality display device is a three-dimensional heads-up display device which is an element of the HAV.

5. The method of claim 1, wherein the interface device is selected from a group that includes:

a smartphone;

a laptop;

a tablet computer;

an electronic book reader; and

a smartwatch.

6. The method of claim 1, further comprising:

analyzing new internal sensor data;

determining that the view type changed from the non-forward looking view type to the forward-looking view type;

generating new graphical data;

determining new position data based on the new internal sensor data and the forward-looking view type; and

providing the new graphical data and the new position data to a new interface device to cause the new interface device to display a graphic described by the new graphical data at a new position described by the new position data.

7. The method of claim 1, wherein the position data is operable so that the position where the visual feedback is displayed in the electronic display of the interface device does not obstruct an object which is being viewed by the driver of the HAV.

8. A system of a highly autonomous vehicle (HAV), the system comprising:

an onboard vehicle computer system including a non-transitory memory storing computer code which, when executed by the onboard vehicle computer system causes the onboard vehicle computer system to;

analyze external sensor data recorded by one or more external sensors of the HAV to identify a traffic situation, wherein the external sensor data describes one or more measurements of a physical environment external to the HAV;

determine, based on internal sensor data, view type data describing a view type for a driver of the HAV, wherein the view type is a forward-looking view type or a non-forward looking view type;

generate graphical data describing visual feedback that visually depicts information describing the traffic situation;

generate position data describing a position where the visual feedback is displayed in an electronic display of an interface device based on the internal sensor data and the view type data; and

provide, by a processor of the HAV, the position data and the graphical data to the interface device to cause the interface device to display the visual feedback at the position described by the position data, wherein the interface device is communicatively coupled to the HAV and operable to receive the graphical data from the processor of the HAV.

9. The system of claim 8, wherein the interface device is an augmented reality display device and the graphical data is operable to cause the augmented reality display device to provide an augmented reality experience.

10. The system of claim 9, wherein the augmented reality display device is selected from a group that includes:

a set of augmented reality goggles; and

a set of augmented reality glasses.

11. The system of claim 9, wherein the augmented reality display device is a three-dimensional heads-up display device which is an element of the HAV.

12. The system of claim 8, wherein the interface device is selected from a group that includes:

a smartphone;

a laptop;

a tablet computer;

an electronic book reader; and

a smartwatch.

13. The system of claim 8, wherein the non-transitory memory stores additional computer code which, when executed by the onboard vehicle computer system causes the onboard vehicle computer system to:

analyze new internal sensor data;

determine that the view type changed from the non-forward looking view type to the forward-looking view type;

generate new graphical data;

determine new position data based on the new internal sensor data and the forward-looking view type; and

provide the new graphical data and the new position data to a new interface device to cause the new interface device to display a graphic described by the new graphical data at a new position described by the new position data.

14. The system of claim 8, wherein the position data is operable so that the position where the visual feedback is displayed in the electronic display of the interface device does not obstruct an object which is being viewed by the driver of the HAV.

15. A computer program product comprising a non-transitory memory of an onboard vehicle computer system of a highly autonomous vehicle (HAV) storing computer-executable code that, when executed by the onboard vehicle computer system, causes the onboard vehicle computer system to:

analyze external sensor data recorded by one or more external sensors of the HAV to identify a traffic situation, wherein the external sensor data describes one or more measurements of a physical environment external to the HAV;

determine, based on internal sensor data, view type data describing a view type for a driver of the HAV, wherein the view type is a forward-looking view type or a non-forward looking view type;

generate graphical data describing visual feedback that visually depicts information describing the traffic situation;

generate position data describing a position where the visual feedback is displayed in an electronic display of an interface device based on the internal sensor data and the view type data; and

provide, by a processor of the HAV, the position data and the graphical data to the interface device to cause the interface device to display the visual feedback at the position described by the position data, wherein the interface device is communicatively coupled to the HAV and operable to receive the graphical data from the processor of the HAV.

16. The computer program product of claim 15, wherein the interface device is an augmented reality display device and the graphical data is operable to cause the augmented reality display device to provide an augmented reality experience.

17. The computer program product of claim 16, wherein the augmented reality display device is selected from a group that includes:

a set of augmented reality goggles; and

a set of augmented reality glasses.

18. The computer program product of claim 16, wherein the augmented reality display device is a three-dimensional heads-up display device which is an element of the HAV.

19. The computer program product of claim 15, wherein the interface device is selected from a group that includes:

a smartphone;

a laptop;

a tablet computer;

an electronic book reader; and

a smartwatch.

20. The computer program product of claim 15, wherein the non-transitory memory stores additional computer code which, when executed by the onboard vehicle computer system causes the onboard vehicle computer system to:

analyze new internal sensor data;

determine that the view type changed from the non-forward looking view type to the forward-looking view type;

generate new graphical data;

determine new position data based on the new internal sensor data and the forward-looking view type; and

provide the new graphical data and the new position data to a new interface device to cause the new interface device to display a graphic described by the new graphical data at a new position described by the new position data.