User interface for displaying internal state of autonomous driving system

• US 10,093,324 B1
• Filed: 01/12/2017
• Issued: 10/09/2018
• Est. Priority Date: 04/28/2010
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Szybalski, Andrew Timothy, Prada Gomez, Luis Ricardo, Nemec, Philip, Urmson, Christopher Paul, Thrun, Sebastian
• Primary Examiner(s): Brown, Drew J
• Application Number:
  US15/404,556
• Time in Patent Office: 635 Days
• CPC Class Codes: B60K 2360/16 Type of output information
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 2552/00 Input parameters relating to infrastructure
• CPC Class Codes: B60W 2554/80 Spatial relation or speed relative to objects
• CPC Class Codes: B60W 2555/60 Traffic rules, e.g. speed limits or right of way
• CPC Class Codes: B60W 2556/50 of positioning data, e.g. GPS [Global Positioning System] data
• CPC Class Codes: B60W 2720/106 Longitudinal acceleration
• CPC Class Codes: B60W 30/12 Lane keeping
• CPC Class Codes: B60W 30/16 Control of distance between vehicles, e.g. keeping a distance to preceding vehicle
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: B60W 60/0016 of the vehicle or its occupants
• CPC Class Codes: G01C 21/367 Details, e.g. road map scale, orientation, zooming, illumination, level of detail, scrolling of road map or positioning of current position marker
• CPC Class Codes: G01C 21/3676 Overview of the route on the road map
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 11/20 Drawing from basic elements, e.g. lines or circles
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle

Abstract

Autonomous vehicles use various computing systems to transport passengers from one location to another. A control computer sends messages to the various systems of the vehicle in order to maneuver the vehicle safely to the destination. The control computer may display information on an electronic display in order to allow the passenger to understand what actions the vehicle may be taking in the immediate future. Various icons and images may be used to provide this information to the passenger.

Claims

1. A method of providing for display an image of a driving environment of a vehicle to an occupant of the vehicle, the method comprising:

determining, by one or more processors, a geographic location of the vehicle;

receiving, by the one or more processors, from a sensor of vehicle information identifying an object in an external environment of the vehicle corresponding a traffic signal light;

determining, by the one or more processors, from the vehicle information a state of the traffic signal light;

generating, by the one or more processors, the image for display including a representation of the vehicle, a representation of the traffic signal light, an indication of the state of the traffic signal light, and a background image corresponding to a driving environment of the geographic location of the vehicle; and

providing, by the one or more processors, the image for display to the occupant of the vehicle.

2. The method of claim 1, further comprising:

determining, based on the state of the traffic signal light, how the vehicle will respond to the traffic signal light; and

wherein generating the image includes generating a representation of how the vehicle will respond to the traffic signal light.

3. The method of claim 2, wherein the representation of how the vehicle will respond to the traffic signal light includes a visual representation that indicates that the vehicle will slow down.

4. The method of claim 3, wherein the visual representation includes an arrow.

5. The method of claim 1, further comprising:

prior to determining the state of the traffic signal light, generating a second image that includes a visual representation that indicates that the one or more processors have not yet determined the state of the traffic signal light; and

providing the second image for display to the occupant, prior to providing the image for display.

6. The method of claim 5, wherein the visual representation that indicates that the one or more processors have not yet determined the state of the traffic signal light includes a question mark.

7. The method of claim 6, wherein the visual representation that indicates that the one or more processors have not yet determined the state of the traffic signal light includes the question mark within a polygon.

8. The method of claim 1, further comprising determining a future path that the vehicle will travel upon, and wherein generating the image includes a representation of the future path, and the representation of the future path includes the indication of the state of the traffic signal light.

9. The method of claim 8, wherein the representation of the path includes a first solid portion and a second striped portion.

10. The method of claim 9, wherein the second striped portion corresponds to the indication of the state of the traffic signal light.

11. The method of claim 8, wherein the geographic location of the vehicle corresponds to the vehicle approaching an intersection, and the method further comprises:

determining, based on the state of the traffic signal light, how the vehicle will respond to the traffic signal light; and

wherein the indication of the state of the traffic signal light includes a representation of how the vehicle will respond to the traffic signal light by waiting at the intersection.

12. A system for providing for display an image of a driving environment of a vehicle to an occupant of the vehicle, the system comprising one or more processors configured to:

determine a geographic location of the vehicle;

receive from a sensor of vehicle information identifying an object in an external environment of the vehicle corresponding a traffic signal light;

determine from the vehicle information a state of the traffic signal light;

generate the image for display including a representation of the vehicle, a representation of the traffic signal light, an indication of the state of the traffic signal light, and a background image corresponding to a driving environment of the geographic location of the vehicle; and

provide the image for display to the occupant of the vehicle.

13. The system of claim 12, wherein the one or more processors are further configured to:

determine, based on the state of the traffic signal light, how the vehicle will respond to the traffic signal light; and

generate the image by generating a representation of how the vehicle will respond to the traffic signal light.

14. The system of claim 13, wherein the representation of how the vehicle will respond to the traffic signal light includes a visual representation that indicates that the vehicle will slow down.

15. The system of claim 14, wherein the visual representation includes an arrow.

16. The system of claim 12, wherein the one or more processors are further configured to:

prior to determining the state of the traffic signal light, generate a second image that includes a visual representation that indicates that the one or more processors have not yet determined the state of the traffic signal light; and

provide the second image for display to the occupant, prior to providing the image for display.

17. The system of claim 16, wherein the visual representation that indicates that the one or more processors have not yet determined the state of the traffic signal light includes a question mark.

18. The system of claim 17, wherein the visual representation that indicates that the one or more processors have not yet determined the state of the traffic signal light includes the question mark within a polygon.

19. The system of claim 12, wherein the one or more processors are further configured to determine a future path that the vehicle will travel upon, and wherein generating the image includes a representation of the future path, and the representation of the future path includes the indication of the state of the traffic signal light.

20. The system of claim 12, further comprising the vehicle.