User interface for displaying object-based indications in an autonomous driving system

• US 10,139,829 B1
• Filed: 10/17/2016
• Issued: 11/27/2018
• Est. Priority Date: 03/12/2013
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Mariet, Robertus Christianus Elisabeth, Clement, Manuel Christian, Nemec, Philip, Cullinane, Brian Douglas
• Primary Examiner(s): Brown, Drew J
• Application Number:
  US15/295,433
• Time in Patent Office: 771 Days
• CPC Class Codes: B60T 2201/022 Collision avoidance systems
• CPC Class Codes: B60T 7/22 initiated by contact of vehicle, e.g. bumper, with an external object, e.g. another vehicle , or by means of contactless obstacle detectors mounted on the vehicle
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 2554/804 Relative longitudinal speed
• CPC Class Codes: B60W 2556/40 High definition maps
• CPC Class Codes: B60W 2556/50 of positioning data, e.g. GPS [Global Positioning System] data
• CPC Class Codes: B60W 30/12 Lane keeping
• CPC Class Codes: B60W 30/18159 Traversing an intersection
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: B60W 60/0016 of the vehicle or its occupants
• CPC Class Codes: G01C 21/367 Details, e.g. road map scale, orientation, zooming, illumination, level of detail, scrolling of road map or positioning of current position marker
• CPC Class Codes: G05D 1/0238 using obstacle or wall sensors G05D1/0246 and G05D1/0289 take precedence; lidar systems designed for anti-collision purposes G01S17/93
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G08G 1/0962 having an indicator mounted inside the vehicle, e.g. giving voice messages
• CPC Class Codes: G08G 1/09626 where the origin of the information is within the own vehicle, e.g. a local storage device, digital map
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: G08G 1/167 Driving aids for lane monitoring, lane changing, e.g. blind spot detection

Abstract

A vehicle has a plurality of control apparatuses, a user input, a geographic position component, an object detection apparatus, memory, and a display. A processor is also included and is programmed to receive the destination information, identify a route, and determine the current geographic location of the vehicle. The processor is also programmed to identify an object and object type based on object information received from the object detection apparatus and to determine at least one warning characteristic of the identified object based on at least one of: the object type, a detected proximity of the detected object to the vehicle, the location of the detected object relative to predetermined peripheral areas of the vehicle, the current geographic location of the vehicle, and the route. The processor is also configured to select and display on the display an object warning image based on the at least one warning characteristic.

Claims

1. An autonomous driving system of a vehicle, the autonomous driving system comprising:

a first display configured to provide an indicator identifying whether the autonomous driving system is engaged in an autonomous mode;

a second display configure to display navigation functionality of the autonomous driving system;

an object detection system configured to detect object information of one or more objects in proximity to the vehicle, the object detection system including at least one of a laser, a camera, radar and sonar;

a vehicle control system configured to control one or more of braking, acceleration, steering and signaling of the vehicle while in the autonomous mode;

a user input device configured to receive input regarding the autonomous mode; and

one or more processors operatively coupled to the first display, the second display, the object detection system, the vehicle navigation system, and the user input device, the one or more processors being configured to;

receive detected object information of the one or more objects from the object detection system;

identify a position and speed of a given one of the one or more objects relative to a current position and speed of the vehicle based on the received detected object information;

generate a notification image corresponding to one or more peripheral areas external to the vehicle;

update the notification image by modifying at least one peripheral area of the notification image when a first threshold condition of the identified position and speed of the given object relative to the current position and speed of the vehicle is met;

cause the updated notification image to be displayed on either the first display or the second display;

generate a warning indicator for the given object when a second threshold condition of the identified position and speed of the given object relative to the current position and speed of the vehicle is met; and

cause the updated notification image and the warning indicator to be displayed on either the first display or the second display, the warning indicator being displayed separate from the updated notification image.

2. The autonomous driving system of claim 1, wherein the one or more processors are further configured to:

detect a change in the identified position or speed of the given object relative to the vehicle; and

determine that the first or the second threshold condition is met in response to the detected change.

3. The autonomous driving system of claim 2, wherein the one or more processors are further configured to instruct the vehicle control system to control the vehicle to take corrective action in response to the determination that the second threshold condition is met.

4. The autonomous driving system of claim 1, wherein the warning indicator identifies a characteristic of the given object.

5. The autonomous driving system of claim 1, wherein the one or more processors are further configured to identify an object type for the given object based on the detected object information.

6. The autonomous driving system of claim 1, wherein the one or more processors are further configured to generate a subject vehicle icon representing the vehicle, and to cause the notification image to be displayed adjacent to the subject vehicle icon.

7. The autonomous driving system of claim 1, wherein the one or more processors are further configured to generate a route line for display, the route line indicating a route along which the vehicle is traveling.

8. The autonomous driving system of claim 1, wherein the first display is disposed along a dashboard of the vehicle.

9. The autonomous driving system of claim 1, wherein the user input device is disposed along a center console arranged between driver and passenger seats of the vehicle.

10. The autonomous driving system of claim 1, wherein the notification image includes one or more segments.

11. The autonomous driving system of claim 10, wherein the one or more segments of the notification image are displayable as a multi-segmented shape at least partly surrounding a vehicle icon, the vehicle icon representing the vehicle.

12. The autonomous driving system of claim 11, wherein the multi-segmented shape includes a plurality of arcuate or linear elements.

13. An autonomous driving system of a vehicle, the autonomous driving system comprising:

a first display element configured to provide an indicator identifying whether the autonomous driving system is engaged in an autonomous mode;

a second display element configure to display navigation functionality of the autonomous driving system;

an object detection system configured to detect object information of one or more objects in proximity to the vehicle, the object detection system including at least one of a laser, a camera, radar and sonar;

a vehicle control system configured to control one or more of braking, acceleration, steering and signaling of the vehicle while in the autonomous mode; and

one or more processors operatively coupled to the first display, the second display, the object detection system, and the vehicle navigation system, the one or more processors being configured to;

receive detected object information of the one or more objects from the object detection system;

identify a position and speed of a given one of the one or more objects relative to a current position and speed of the vehicle based on the received detected object information;

generate a notification image corresponding to one or more peripheral areas external to the vehicle;

generate a first indicator for the given object when a threshold condition of the identified position and speed relative to the current position and speed of the vehicle is not met;

cause the notification image and the first indicator to be displayed on either the first display or the second display; and

cause a second indicator to be displayed with the notification image and the first indicator on either the first display or the second display when the threshold condition is met, the second indicator providing information about the given object, wherein the second indicator is arranged for display adjacent to and separate from the first indicator.

14. The autonomous driving system of claim 13, wherein the first display element is part of a first display device and the second display element is part of a second display device separate from the first display device.

16. The autonomous driving system of claim 13, wherein the one or more processors are configured to:

detect a change in the identified position or speed of the given object relative to the vehicle;

determine that the threshold condition is met in response to the detected change.

17. The autonomous driving system of claim 16, wherein upon determination that the threshold condition is met, the one or more processors are configured to change the first indicator to the second indicator by changing a color of the first indicator.

18. The autonomous driving system of claim 16, wherein the one or more processors are further configured to instruct the vehicle control system to control the vehicle to take corrective action in response to the determination that the threshold condition is met.

19. The autonomous driving system of claim 13, wherein the second indicator is provided to indicate to a passenger of the vehicle that the autonomous driving system will not take a particular action because of a condition associated with the given object.

20. The autonomous driving system of claim 13, wherein the one or more processors are further configured to generate a route line for display, the route line indicating a route along which the vehicle is traveling.

15. The autonomous driving system of 13, wherein either the first display element or the second display element is further configured to provide information related to an entertainment system of the vehicle.