Methods for communicating state, intent, and context of an autonomous vehicle

• US 10,261,513 B2
• Filed: 12/19/2016
• Issued: 04/16/2019
• Est. Priority Date: 12/19/2016
• Status: Active Grant

• Current Assignee: Drive.ai, Inc. (Apple Inc.)
• Original Assignee: Drive.ai, Inc. (Apple Inc.)
• Inventors: Reiley, Carol, Espinel, Daniel, Wistort, Ryan, Paris, Michael
• Primary Examiner(s): Marc, McDieunel
• Application Number:
  US15/384,033
• Publication Number:
  US 20180173237A1
• Time in Patent Office: 848 Days
• CPC Class Codes: B60Q 1/26 the devices being primarily intended to indicate the vehicle, or parts thereof, or to give signals, to other traffic
• CPC Class Codes: B60Q 1/46 for giving flashing caution signals during drive, other than signalling change of direction, e.g. flashing the headlights or hazard lights
• CPC Class Codes: B60Q 1/50 for indicating other intentions or conditions, e.g. request for waiting or overtaking
• CPC Class Codes: B60Q 1/5035 electronic displays
• CPC Class Codes: B60Q 1/5037 the display content changing automatically, e.g. depending on traffic situation
• CPC Class Codes: B60Q 1/507 specific to autonomous vehicles
• CPC Class Codes: B60Q 1/508 specific to vehicles driving in fleets or convoys
• CPC Class Codes: B60Q 1/525 automatically indicating risk of collision between vehicles in traffic or with pedestrians, e.g. after risk assessment using the vehicle sensor data
• CPC Class Codes: B60Q 1/543 for indicating other states or conditions of the vehicle indicating taxis' state of hire or fare G07B13/045
• CPC Class Codes: B60Q 1/545 for indicating other traffic conditions, e.g. fog, heavy traffic
• CPC Class Codes: B60Q 1/547 for issuing requests to other traffic participants; for confirming to other traffic participants they can proceed, e.g. they can overtake
• CPC Class Codes: B60Q 1/549 for expressing greetings, gratitude or emotions
• CPC Class Codes: B60Q 2400/50 Projected symbol or information, e.g. onto the road or car body
• CPC Class Codes: B60Q 5/006 indicating risk of collision between vehicles or with pedestrians
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2554/4029 Pedestrians
• CPC Class Codes: B60W 2756/10 Involving external transmission of data to or from the vehicle
• CPC Class Codes: B60W 30/18154 Approaching an intersection
• CPC Class Codes: B60W 30/18159 Traversing an intersection
• CPC Class Codes: B60W 50/082 : Selecting or switching between different modes of propelling
• CPC Class Codes: B60W 60/001 : Planning or execution of driving tasks
• CPC Class Codes: B60W 60/0053 : from vehicle to occupant
• CPC Class Codes: G05D 1/0061 : for transition from automatic pilot to manual pilot and vice versa
• CPC Class Codes: G05D 1/0214 : in accordance with safety or protection criteria, e.g. avoiding hazardous areas monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13
• CPC Class Codes: G05D 1/0246 : using a video camera in combination with image processing means

Abstract

One variation of a method for communicating state, intent, and context of an autonomous vehicle includes: at a first time, displaying a first icon representing a current state of a vehicle on a rear-facing visual display arranged on the vehicle; navigating toward an intersection; at a second time, detecting a state of the intersection ahead of the vehicle; rendering a second icon representing the state of the intersection at the second time on the rear-facing visual display; detecting a change in the state of the intersection at a third time succeeding the second time; selecting a next navigation action for the vehicle responsive to the change in the state of the intersection at the third time; prior to executing the next navigation action, rendering a third icon representing the next navigation action on the rear-facing visual display; and autonomously executing the next navigation action.

Claims

1. A method comprising, at a vehicle:

recording a sensor stream via a sensor suite in the vehicle;

at a first time;

rendering a first representation of a current state of the vehicle on a rear-facing visual display arranged on the vehicle;

autonomously navigating toward an intersection based on the sensor stream;

during a second period of time succeeding the first time;

detecting proximity of the vehicle to the intersection based on the sensor stream;

detecting a current state of the intersection ahead of the vehicle based on the sensor stream; and

in response to the vehicle falling within a threshold distance of the intersection, rendering a second representation of the current state of the intersection on the rear-facing visual display;

during a third period of time;

in response to detecting a change in the state of the intersection, selecting a next navigation action for the vehicle based on the sensor stream;

in preparation for autonomously executing the next navigation action, rendering a third representation of the next navigation action on the rear-facing visual display; and

autonomously executing the next navigation action to autonomously navigate the vehicle through the intersection based on the sensor stream.

2. The method of claim 1, wherein rendering the first representation of the current state of the vehicle on the rear-facing visual display comprises displaying a first indicator that the vehicle is operating in a fully-autonomous driving mode on the rear-facing visual display at the first time.

3. The method of claim 2:

further comprising maintaining a substantially constant speed of the vehicle in the fully-autonomous driving mode during a first period of time comprising the first time;

wherein displaying the first indicator that the vehicle is operating in the fully-autonomous driving mode comprises displaying the first indicator comprising a first color representative of the fully-autonomous driving mode on the rear-facing visual display during the first period of time; and

further comprising;

in response to receipt of a command to enter a manual driving mode from an occupant of the vehicle, transitioning from the fully-autonomous driving mode to the manual driving mode; and

in response to transitioning into the manual driving mode, displaying a second indicator that the vehicle is operating in the manual driving mode on the rear-facing visual display, the second indicator comprising a second color distinct from the first color.

4. The method of claim 1, further comprising, during a fourth period of time:

recording a digital scan image of a scene around the vehicle through an optical sensor arranged on the vehicle;

detecting, in the digital scan image, proximity of a pedestrian to the vehicle and a position of the pedestrian relative to the vehicle;

identifying a particular display, in a set of displays on the vehicle, in a field of view of the pedestrian based on the position of the pedestrian relative to the vehicle; and

rendering, on the particular visual display, an icon indicating identification of the pedestrian by the vehicle.

5. The method of claim 1, wherein rendering the second representation of the current state of the intersection on the rear-facing visual display comprises rendering a second icon indicating the current state of the intersection on the rear-facing visual display in response to arrival of the vehicle at a head of the intersection and detection of a second vehicle behind the vehicle.

6. The method of claim 1:

wherein rendering the first representation of the current state of the vehicle on the rear-facing visual display comprises rendering a first icon indicating deceleration of the vehicle toward the intersection on the rear-facing visual display;

wherein detecting proximity of the vehicle to the intersection comprises confirming arrival of the vehicle within the threshold distance of the intersection;

wherein detecting the current state of the intersection ahead of the vehicle comprises;

recording a digital scan image of a scene around the vehicle through an optical sensor arranged on the vehicle; and

determining dispossession, of the vehicle, to right of way at the intersection based on traffic signage identified in the digital scan image;

wherein rendering the second representation of the current state of the intersection on the rear-facing visual display comprises rendering the second representation of the current state of the intersection during deceleration of the vehicle toward the intersection in response to arrival of the vehicle within the threshold distance of the intersection.

7. The method of claim 6:

wherein detecting proximity of the vehicle to the intersection comprises confirming arrival of the vehicle within the threshold distance of the intersection based on a geospatial location of the vehicle and a virtual roadmap indicating a geospatial location of the intersection;

wherein determining dispossession, of the vehicle, to right of way at the intersection comprises identifying the traffic signage in the digital scan image and determining dispossession, of the vehicle, to right of way at the intersection based on a state of the traffic signage.

8. The method of claim 7:

wherein selecting the next navigation action for the vehicle comprises;

confirming possession, by the vehicle, of right of way at the intersection based on a change in a state of the traffic signage;

selecting the next navigation action succeeding the intersection in a current preplanned driving path in process at the vehicle;

recording a second digital scan image of the scene around the vehicle through the optical sensor arranged on the vehicle; and

confirming lack of a pedestrian in a path of the next navigation action based on the second digital scan image;

wherein autonomously executing the next navigation action comprises accelerating forward to resume the current preplanned driving path in response to confirmation of possession, by the vehicle, of right of way at the intersection and lack of a pedestrian in the path of the next navigation action.

9. The method of claim 1, wherein rendering the first representation of the current state of the vehicle on the rear-facing visual display arranged on the vehicle comprises rendering the first representation of the current state of the vehicle on the rear-facing visual display mounted to a roof of the vehicle.

10. The method of claim 1:

wherein selecting the next navigation action for the vehicle comprises calculating a confidence score for reliable execution of the next navigation action based on the change in the state of the intersection;

wherein rendering the third representation of the next navigation action on the rear-facing visual display comprises displaying the third representation of the next navigation action on the rear-facing visual display in response to the confidence score exceeding a threshold score; and

further comprising, in response to the confidence score remaining below the threshold score;

rendering a fourth representation of a processing state of the vehicle on the rear-facing visual display; and

delaying autonomous execution of the next navigation action.

11. The method of claim 1, wherein rendering the second representation of the current state of the intersection on the rear-facing visual display and rendering the third representation of the next navigation action on the rear-facing visual display comprises oscillating between displaying the second representation and displaying the third representation on the rear-facing visual display while the vehicle is stopped at the intersection.

12. The method of claim 1:

wherein recording the sensor stream via the sensor suite in the vehicle comprises recording a digital scan image of a scene around the vehicle through an optical sensor arranged on the vehicle;

wherein detecting the current state of the intersection ahead of the vehicle comprises, during the second period of time, extracting the current state of the intersection ahead of the vehicle from the digital scan image; and

further comprising, during the second period of time;

extracting a current state of the scene behind the vehicle from the digital scan image; and

rendering a fourth representation of the current state of the scene behind the vehicle on a forward-facing visual display arranged on the vehicle.

13. The method of claim 12:

wherein extracting the current state of the scene behind the vehicle from the digital scan image comprises;

confirming dispossession, of the vehicle, to right of way at the intersection based on a state of a traffic signal identified in the digital scan image;

detecting, in the digital scan image, a pedestrian ahead of the vehicle; and

detecting, in the digital scan image, a second vehicle approaching the intersection from behind the vehicle;

wherein rendering the second representation of the current state of the intersection on the rear-facing visual display comprises rendering a second icon indicating presence of the pedestrian ahead of the vehicle and indicating the state of the traffic signal on the rear-facing visual display; and

wherein rendering the fourth representation of the current state of the scene behind the vehicle on the forward-facing visual display comprises rendering an icon representative of the second vehicle on the forward-facing display in response to detection of the pedestrian ahead of the vehicle.

14. The method of claim 13:

wherein detecting the second vehicle approaching the intersection comprises estimating a speed of the second vehicle; and

wherein rendering the icon representative of the second vehicle further comprises rendering an icon indicating the speed of the second vehicle.

15. The method of claim 12, further comprising, during the third period of time, replaying an audio track through an audio speaker arranged on the vehicle in conjunction with rendering the third representation of the next navigation action on the rear-facing visual display, the audio track comprising an audible sound configured to draw attention to the rear-facing visual display.

16. The method of claim 1:

wherein detecting the current state of the intersection ahead of the vehicle comprises detecting a second vehicle stopped ahead of the vehicle at the intersection; and

wherein rendering the second representation of the current state of the intersection on the rear-facing visual display comprises displaying an icon indicating the presence of the second vehicle ahead of the vehicle on the rear-facing visual display.

17. The method of claim 16:

further comprising receiving a specification of a state of the intersection ahead of the second vehicle broadcast from the second vehicle; and

wherein rendering the second representation of the current state of the intersection on the rear-facing visual display comprises displaying the second representation of the current state of the intersection based on the specification of the state of the intersection received from the second vehicle.

18. The method of claim 1:

wherein selecting the next navigation action for the vehicle comprises selecting a right turn action;

wherein rendering the third representation of the next navigation action on the rear-facing visual display comprises rendering an icon representative of the right turn action on the rear-facing visual display at a third time in the third period of time; and

wherein autonomously executing the next navigation action comprises autonomously executing the right turn action at a fourth time succeeding the third time in the third period of time.

19. The method of claim 18:

wherein selecting the next navigation action for the vehicle comprises setting an acceleration rate through the intersection; and

wherein rendering the icon representative of the right turn action comprises rendering the icon animated at a rate proportional to the acceleration rate and orientated according to a severity of the right turn action on the rear-facing visual display.

20. A method comprising, at a vehicle:

recording a sensor stream via a sensor suite in the vehicle;

at a first time;

displaying a first icon representing a current state of the vehicle on a visual display arranged on the vehicle;

autonomously navigating toward an intersection based on the sensor stream;

at a second time succeeding the first time, detecting a state of a roadway near the vehicle based on the sensor stream;

in response to the vehicle falling within a threshold distance of the intersection, rendering a second icon representing the state of the roadway at the second time on the visual display;

detecting a change in the state of the roadway at a third time succeeding the second time based on the sensor stream;

selecting a next navigation action for the vehicle responsive to the change in the state of the roadway at the third time;

prior to autonomously executing the next navigation action, rendering a third icon representing the next navigation action on the visual display; and

at a fourth time succeeding the third time, autonomously executing the next navigation action to autonomously navigate the vehicle through the intersection based on the sensor stream.

21. The method of claim 20,

wherein displaying the first icon representing the current state of the vehicle on the visual display comprises rendering the first icon in a first color corresponding to a fully-autonomous driving mode on the rear-facing visual display;

wherein rendering the second icon representing the state of the roadway on the visual display comprises rendering the second icon in the first color on the visual display;

wherein rendering the third icon representing the next navigation action on the visual display comprises rendering the third icon in the first color on the display; and

further comprising;

in response to receipt of a command to enter a manual driving mode from an occupant of the vehicle, transitioning from a fully-autonomous driving mode to the manual driving mode; and

in response to transitioning into the manual driving mode, displaying a fourth icon in a second color corresponding to the manual driving mode on the rear-facing visual display, the second color distinct from the first color.