User interface for displaying internal state of autonomous driving system

• US 10,293,838 B1
• Filed: 05/23/2017
• Issued: 05/21/2019
• Est. Priority Date: 04/28/2010
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Szybalski, Andrew Timothy, Prada Gomez, Luis Ricardo, Nemec, Philip, Urmson, Christopher Paul, Thrun, Sebastian
• Primary Examiner(s): Brown, Drew J
• Application Number:
  US15/602,446
• Time in Patent Office: 728 Days
• CPC Class Codes: B60K 2360/16 Type of output information
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 2552/00 Input parameters relating to infrastructure
• CPC Class Codes: B60W 2554/80 Spatial relation or speed relative to objects
• CPC Class Codes: B60W 2555/60 Traffic rules, e.g. speed limits or right of way
• CPC Class Codes: B60W 2556/50 of positioning data, e.g. GPS [Global Positioning System] data
• CPC Class Codes: B60W 2720/106 Longitudinal acceleration
• CPC Class Codes: B60W 30/12 Lane keeping
• CPC Class Codes: B60W 30/16 Control of distance between vehicles, e.g. keeping a distance to preceding vehicle
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: B60W 60/0016 of the vehicle or its occupants
• CPC Class Codes: G01C 21/367 Details, e.g. road map scale, orientation, zooming, illumination, level of detail, scrolling of road map or positioning of current position marker
• CPC Class Codes: G01C 21/3676 Overview of the route on the road map
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 11/20 Drawing from basic elements, e.g. lines or circles
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle

Abstract

Autonomous vehicles use various computing systems to transport passengers from one location to another. A control computer sends messages to the various systems of the vehicle in order to maneuver the vehicle safely to the destination. The control computer may display information on an electronic display in order to allow the passenger to understand what actions the vehicle may be taking in the immediate future. Various icons and images may be used to provide this information to the passenger.

Claims

1. A method comprising:

displaying navigation information on a first display screen of a first vehicle;

identifying a second vehicle in an environment of the first vehicle;

determining a path of the first vehicle for some period of time into the future;

generating a visual representation of the environment including a representation of the first vehicle, a representation of the second vehicle, and a representation of the path, wherein the representation of the second vehicle is positioned relative to the representation of the first vehicle, the visual representation providing a perspective that is behind and above the first vehicle; and

displaying the visual representation on a second display screen of the first vehicle, the second display screen being separate from the first display screen.

2. The method of claim 1, wherein the representation of the first vehicle is a first icon and the representation of the second vehicle is a second icon.

3. The method of claim 1, wherein the representation of the first vehicle is a first box and the representation of the second vehicle is a second box.

4. The method of claim 3, further comprising:

determining a size and a shape of the second box based on a size and a shape of the second vehicle, and

displaying the second box in the visual representation according to the determined size and shape of the second box.

5. The method of claim 1, wherein the first display screen also includes entertainment information for the first vehicle.

6. The method of claim 1, wherein the visual representation is generated such that the representation of the second vehicle is in line with the path.

7. The method of claim 1, wherein the visual representation is generated such that the representation of the second vehicle is adjacent to the path.

8. The method of claim 1, wherein the second vehicle is identified from information generated using an object detection component of the first vehicle.

9. The method of claim 1, further comprising identifying a portion of roadway, wherein the visual representation includes a representation of the portion of roadway, and the visual representation is displayed in order to provide a passenger of the first vehicle with information about the first vehicle'"'"'s perception of the environment with respect to the portion of roadway.

10. The method of claim 1, further comprising, changing relative positions of the representation of the first vehicle and the representation of the second vehicles based on a detected change in distance between the first vehicle and the second vehicle.

11. A system comprising:

one or more processors configured to;

display navigation information on a first display screen of the first vehicle;

identify a second vehicle in the environment of the first vehicle;

determine a path of the first vehicle for some period of time into the future;

generate a visual representation of the environment including a representation of the first vehicle, a representation of the second vehicle, and a representation of the path, wherein the representation of the second vehicle is positioned relative to the representation of the vehicle, the visual representation providing a perspective that is behind and above the first vehicle; and

display the visual representation on a second display screen of the first vehicle wherein the first display screen is separate from the second display screen.

12. The system of claim 11, wherein the representation of the first vehicle is a first icon and the representation of the second vehicle is a second icon.

13. The system of claim 11, wherein the representation of the first vehicle is a first box and the representation of the second vehicle is a second box.

14. The system of claim 13, wherein the one or more processors are further configured to determine a size and a shape of the second box based on a size and a shape of the second vehicle and to display the second box in the visual representation according to the determined size and shape of the second box.

15. The system of claim 11, wherein the visual representation is generated such that the representation of the second vehicle is in line with the path.

16. The system of claim 11, wherein the visual representation is generated such that the representation of the second vehicle is adjacent to the path.

17. The system of claim 11, further comprising the second display screen and the first display screen.

18. The system of claim 11, wherein the one or more processors are further configured to identify a portion of roadway, the visual representation includes a representation of the portion of roadway, and the visual representation is displayed in order to provide a passenger of the first vehicle with information about the first vehicle'"'"'s perception of the environment with respect to the portion of roadway.

19. The system of claim 11, wherein the one or more processors are further configured to change relative positions of the representation of the first vehicle and the representation of the second vehicles based on a detected change in distance between the first vehicle and the second vehicle.

20. The system of claim 11, further comprising the first vehicle.