Generating spatial information for a heads-up display

• US 9,409,519 B2
• Filed: 08/27/2014
• Issued: 08/09/2016
• Est. Priority Date: 08/27/2014
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Inventors: Sisbot, Emrah Akin, Yalla, Veeraganesh
• Primary Examiner(s): NGUYEN, TAI T
• Application Number:
  US14/470,849
• Publication Number:
  US 20160059783A1
• Time in Patent Office: 713 Days
• Field of Search: 340/901, 340/903, 340/905, 340/907, 340/435, 340/438, 340/995.19, 340/995.24, 340/988, 701/209, 701/210, 701/202, 348/143, 348/148, 348/164
• US Class Current: 1/1
• CPC Class Codes: B60R 1/001 integrated in the windows, e.g. Fresnel lenses
• CPC Class Codes: B60R 2300/205 using a head-up display
• CPC Class Codes: B60R 2300/804 for lane monitoring
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: G02B 2027/0196 having transparent supporting structure for display mounting, e.g. to a window or a windshield
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 3/20 Linear translation of whole images or parts thereof, e.g. panning
• CPC Class Codes: G09G 2380/10 Automotive applications

Abstract

The disclosure includes a system and method for spatial information for a heads-up display. The system includes a processor and a memory storing instructions that, when executed, cause the system to: receive sensor data about an entity, assign the entity to a category, estimate a danger index for the entity based on vehicle data, category data, and a position of the entity, generate entity data that includes the danger index, identify a graphic that is a representation of the entity based on the entity data, determine a display modality for the graphic based on the danger index, and position the graphic to correspond to a user'"'"'s eye frame.

Claims

1. A method comprising:

receiving sensor data describing an entity with a processor-based computing device programmed to perform the receiving;

assigning the entity to a category associated with the entity;

estimating a danger index for the entity based on vehicle data describing a vehicle, category data describing the category, and a position of the entity;

generating entity data that includes the danger index;

identifying a graphic that is a representation of the entity based on the entity data;

determining a display modality for the graphic based on the danger index; and

positioning the graphic to correspond to a user'"'"'s eye frame, wherein positioning the graphic to correspond to the user'"'"'s eye frame further includes;

computing a first transformation from a sensor frame to an eye frame;

multiplying the first transformation by a second transformation from graphics to the sensor frame to obtain a third transformation from the graphics to the eye frame; and

projecting the graphics into a viewport placed at a transformation from graphics to an eye frame pose.

2. The method of claim 1, further comprising displaying the graphic as three-dimensional Cartesian coordinates on a heads-up display.

3. The method of claim 1, wherein the graphic is a simplified representation of the entity.

4. The method of claim 1, wherein the graphic is displayed on a heads-up display unit.

5. The method of claim 1, wherein receiving sensor data about an entity further comprises determining a position of the entity in a sensor frame and a bounding box of the entity.

6. The method of claim 1, wherein the vehicle data includes a current or predicted speed of the vehicle and whether the lights are on.

7. The method of claim 1, wherein the category data includes whether the entity is a mobile object.

8. The method of claim 1, wherein estimating the danger index includes determining whether the danger index exceeds a predetermined threshold probability.

9. The method of claim 1, wherein estimating the danger index includes estimating motion associated with the entity based on historical entity data.

10. The method of claim 1, wherein determining the display modality for the graphic further includes determining a more noticeable modality responsive to an increasing danger index.

11. A computer program product comprising a tangible, non-transitory computer-usable medium including a computer-readable program, wherein the computer-readable program when executed on a computer causes the computer to:

receive sensor data describing an entity;

assign the entity to a category associated with the entity;

estimate a danger index for the entity based on vehicle data describing a vehicle, category data describing the category, and a position of the entity;

generate entity data that includes the danger index;

identify a graphic that is a representation of the entity based on the entity data;

determine a display modality for the graphic based on the danger index; and

position the graphic to correspond to a user'"'"'s eye frame, wherein positioning the graphic to correspond to the user'"'"'s eye frame further includes;

compute a first transformation from a sensor frame to an eye frame;

multiply the first transformation by a second transformation from graphics to the sensor frame to obtain a third transformation from the graphics to the eye frame; and

project the graphics into a viewport placed at a transformation from graphics to an eye frame pose.

12. The computer program product of claim 11, wherein the computer-readable program is further configured to display the graphic as three-dimensional Cartesian coordinates on a heads-up display.

13. The computer program product of claim 11, wherein the graphic is a simplified representation of the entity.

14. The computer program product of claim 11, wherein the graphic is displayed on a heads-up display unit.

15. The computer program product of claim 11, wherein receiving sensor data about an entity further comprises determining a position of the entity in a sensor frame and a bounding box of the entity.

16. A system comprising:

a processor; and

a non-transitory memory storing instructions that, when executed, cause the system to;

receive sensor data describing an entity;

assign the entity to a category;

estimate a danger index for the entity based on vehicle data describing a vehicle, category data describing the category, and a position of the entity;

generate entity data that includes the danger index;

identify a graphic that is a representation of the entity based on the entity data;

determine a display modality for the graphic based on the danger index; and

position the graphic to correspond to a user'"'"'s eye frame wherein positioning the graphic to correspond to the user'"'"'s eye frame further includes causing the system to;

compute a first transformation from a sensor frame to an eye frame;

multiply the first transformation by a second transformation from graphics to the sensor frame to obtain a third transformation from the graphics to the eye frame; and

project the graphics into a viewport placed at a transformation from graphics to an eye frame pose.

17. The system of claim 16, further comprising a heads-up display for displaying the graphic as three-dimensional Cartesian coordinates on a heads-up display.

18. The system of claim 16, wherein the graphic is a simplified representation of the entity.

19. The system of claim 16, wherein the graphic is displayed on a heads-up display unit.

20. The system of claim 16, wherein receiving sensor data about an entity further comprises determining a position of the entity in a sensor frame and a bounding box of the entity.