User interface for displaying internal state of autonomous driving system

• US 9,582,907 B1
• Filed: 08/07/2015
• Issued: 02/28/2017
• Est. Priority Date: 04/28/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Szybalski, Andrew Timothy, Prada Gomez, Luis Ricardo, Nemec, Philip, Urmson, Christopher Paul, Thrun, Sebastian
• Primary Examiner(s): Brown, Drew
• Application Number:
  US14/820,947
• Time in Patent Office: 571 Days
• US Class Current: 1/1
• CPC Class Codes: B60K 2360/16 Type of output information
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 2552/00 Input parameters relating to infrastructure
• CPC Class Codes: B60W 2554/80 Spatial relation or speed relative to objects
• CPC Class Codes: B60W 2555/60 Traffic rules, e.g. speed limits or right of way
• CPC Class Codes: B60W 2556/50 of positioning data, e.g. GPS [Global Positioning System] data
• CPC Class Codes: B60W 2720/106 Longitudinal acceleration
• CPC Class Codes: B60W 30/12 Lane keeping
• CPC Class Codes: B60W 30/16 Control of distance between vehicles, e.g. keeping a distance to preceding vehicle
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: B60W 60/0016 of the vehicle or its occupants
• CPC Class Codes: G01C 21/367 Details, e.g. road map scale, orientation, zooming, illumination, level of detail, scrolling of road map or positioning of current position marker
• CPC Class Codes: G01C 21/3676 Overview of the route on the road map
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 11/20 Drawing from basic elements, e.g. lines or circles
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle

Abstract

Autonomous vehicles use various computing systems to transport passengers from one location to another. A control computer sends messages to the various systems of the vehicle in order to maneuver the vehicle safely to the destination. The control computer may display information on an electronic display in order to allow the passenger to understand what actions the vehicle may be taking in the immediate future. Various icons and images may be used to provide this information to the passenger.

Claims

1. A method of providing for display an image representing a driving environment of a vehicle to an occupant of the vehicle, the method comprising:

determining, by one or more processors, a geographic location of the vehicle;

receiving, by the one or more processors, from a sensor of the vehicle information identifying an object in the vehicle'"'"'s external environment;

identifying, by the one or more processors, one or more features expected to be in the vehicle'"'"'s external environment based on the determined location;

generating, by the one or more processors, the image for display including a representation of the vehicle, a representation of the object, and a background scene corresponding to the identified features, wherein a coloring of the representation of the object indicates a degree of relevance of the object to the vehicle;

providing, by the one or more processors, the image for display to the occupant of the vehicle.

2. The method of claim 1, wherein the representation of the object indicates that the object is below the vehicle.

3. The method of claim 2, wherein the information identifying the object indicates that the object is located below an overpass and the method further comprises determining that the vehicle is located on the overpass based on the determined location.

4. The method of claim 1, wherein the coloring of the representation of the object as compared to a coloring of the representation of the vehicle indicates that the object is below the vehicle.

5. The method of claim 1, wherein a shading of the representation of the object as compared to a shading of the representation of the vehicle indicates that the object is below the vehicle.

6. The method of claim 5, wherein the shading of the representation is lighter than the shading of the representation of the vehicle.

7. The method of claim 1, wherein the information identifying the object indicates that an object type of the object is a vehicle and the representation of the object indicates that an object type of the object representation is a vehicle.

8. The method of claim 1, wherein the representation of the object indicates that the object is on a first roadway of the background scene and the representation of the vehicle indicates that the vehicle is on a second roadway of the background scene different from the first roadway.

9. A system for providing for display an image representing a driving environment of a vehicle to an occupant of the vehicle, the system comprising:

memory storing map information identifying features expected to be detected at various locations;

one or more processors configured to;

determine a geographic location of the vehicle;

receiving from a sensor of the vehicle information identifying an object in the vehicle'"'"'s external environment;

identify one or more features expected to be in the vehicle'"'"'s external environment based on the determined location;

generate the image for display including a representation of the vehicle, a representation of the object, and a background scene corresponding to the identified features, wherein a coloring of the representation of the object indicates a degree of relevance of the object to the vehicle;

provide the image for display to the occupant of the vehicle.

10. The system of claim 9, further comprising:

the vehicle;

a display configured to display the image.

11. The system of claim 10, wherein the representation of the object indicates that the object is below the vehicle.

12. The system of claim 11, wherein the information identifying the object indicates that the object is located below an overpass and the method further comprises determining that the vehicle is located on the overpass based on the determined location.

13. The system of claim 10, wherein the coloring of the representation of the object as compared to a coloring of the representation of the vehicle indicates that the object is below the vehicle.

14. The system of claim 10, wherein a shading of the representation of the object as compared to a shading of the representation of the vehicle indicates that the object is below the vehicle.

15. The system of claim 14, wherein the shading of the representation is lighter than the shading of the representation of the vehicle.

16. The system of claim 10, wherein the information identifying the object indicates that an object type of the object is a vehicle and the representation of the object indicates that an object type of the object representation is a vehicle.

17. The system of claim 10, wherein the representation of the object indicates that the object is on a first roadway of the background scene and the representation of the vehicle indicates that the vehicle is on a second roadway of the background scene different from the first roadway.

18. A non-transitory computer-readable medium on which instructions are stored, the instructions, when executed by one or more processors, cause the one or more processors to perform a method of providing for display an image representing a driving environment of a vehicle to an occupant of the vehicle, the method comprising:

determining a geographic location of the vehicle;

receiving from a sensor of the vehicle information identifying an object in the vehicle'"'"'s external environment;

identifying one or more features expected to be in the vehicle'"'"'s external environment based on the determined location;

generating the image for display including a representation of the vehicle, a representation of the object, and a background scene corresponding to the identified features, wherein a coloring of the representation of the object indicates a degree of relevance of the object to the vehicle;

providing the image for display to the occupant of the vehicle.

19. The medium of claim 18, wherein the information identifying the object indicates that the object is located below an overpass and the method further comprises determining that the vehicle is located on the overpass based on the determined location.

20. The medium of claim 18, wherein a shading of the representation of the object as compared to a shading of the representation of the vehicle indicates that the object is below the vehicle.