Autonomous vehicle interaction with external environment

• US 9,855,890 B2
• Filed: 12/11/2014
• Issued: 01/02/2018
• Est. Priority Date: 12/11/2014
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: James, Michael R., Prokhorov, Danil V.
• Primary Examiner(s): Dager, Jonathan M
• Application Number:
  US14/566,974
• Publication Number:
  US 20160167648A1
• Time in Patent Office: 1,118 Days
• Field of Search: None
• CPC Class Codes: B60K 2360/146 Instrument input by gesture
• CPC Class Codes: B60K 2360/175 Autonomous driving
• CPC Class Codes: B60K 2360/334 Projection means
• CPC Class Codes: B60K 2360/797 at the vehicle exterior
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60K 35/60 Instruments characterised by their location or relative disposition in or on vehicles arrangements of lighting devices on dashboards B60Q3/10
• CPC Class Codes: B60Q 1/50 for indicating other intentions or conditions, e.g. request for waiting or overtaking
• CPC Class Codes: B60Q 1/5035 electronic displays
• CPC Class Codes: B60Q 1/507 specific to autonomous vehicles
• CPC Class Codes: B60Q 1/525 automatically indicating risk of collision between vehicles in traffic or with pedestrians, e.g. after risk assessment using the vehicle sensor data
• CPC Class Codes: B60Q 1/547 for issuing requests to other traffic participants; for confirming to other traffic participants they can proceed, e.g. they can overtake
• CPC Class Codes: B60Q 2400/50 Projected symbol or information, e.g. onto the road or car body
• CPC Class Codes: B60Q 5/006 indicating risk of collision between vehicles or with pedestrians
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

Arrangements relate to the interaction between an autonomous vehicle and an external environment of the autonomous vehicle. Such interaction can occur in various ways. For example, a non-verbal human gesture in the external environment can be detected. The non-verbal human gesture can be identified. A future driving maneuver can be determined based on the identified non-verbal human gesture. The autonomous vehicle can be caused to implement the determined future driving maneuver. As another example, the external environment of the autonomous vehicle can be detected to identify a person (e.g. a human pedestrian, a human bicyclist, a human driver or occupant of another vehicle, etc.) therein. The identified person can be located. It can be determined whether the person is potentially related to a future driving maneuver of the autonomous vehicle. The autonomous vehicle can be caused to send a directional message to the person.

Claims

1. A method of interaction between an autonomous vehicle and an external environment of the autonomous vehicle, the method comprising:

detecting the external environment of the autonomous vehicle to identify a person therein;

locating the identified person;

determining whether the identified person is potentially related to a future driving maneuver of the autonomous vehicle; and

responsive to determining that the identified person is potentially related to a future driving maneuver of the autonomous vehicle, causing the autonomous vehicle to send a directional audial message to the person, the directional audial message being sent directly to the person and being directly perceptible by the person without the person using a portable communication device, wherein causing the autonomous vehicle to send the directional audial message to the person includes;

adjusting a position of an external speaker mounted on the autonomous vehicle to be substantially aligned with the location of the identified person; and

causing the directional audial message to be emitted from the external speaker.

2. A method of interaction between an autonomous vehicle and an external environment of the autonomous vehicle, the method comprising:

detecting the external environment of the autonomous vehicle to identify a person therein;

locating the identified person;

determining whether the identified person is potentially related to a future driving maneuver of the autonomous vehicle; and

responsive to determining that the identified person is potentially related to a future driving maneuver of the autonomous vehicle, causing the autonomous vehicle to send a directional audial message to the person, the directional audial message being sent directly to the person and being directly perceptible by the person without the person using a portable communication device,

a plurality of external speakers being mounted on the autonomous vehicle, the plurality of external speakers including at least one external speaker on a front portion of the autonomous vehicle, at least one external speaker on a rear portion of the autonomous vehicle, at least one external speaker on a first lateral side of the autonomous vehicle, and at least one speaker on a second lateral side of the autonomous vehicle, and wherein causing the autonomous vehicle to send the directional audial message to the person includes;

selecting at least one of the plurality of external speakers that is substantially aligned with the location of the identified person; and

causing the directional audial message to be emitted from the selected at least one of the plurality of external speakers.

3. A method of interaction between an autonomous vehicle and an external environment of the autonomous vehicle, the method comprising:

detecting the external environment of the autonomous vehicle to identify a person therein;

locating the identified person;

determining whether the identified person is potentially related to a future driving maneuver of the autonomous vehicle; and

responsive to determining that the identified person is potentially related to a future driving maneuver of the autonomous vehicle, causing the autonomous vehicle to send a directional visual message to the person, the directional visual message being sent directly to the person and being directly perceptible by the person without the person using a portable communication device, and wherein causing the autonomous vehicle to send the directional visual message to the person includes;

adjusting a position of an externally facing display mounted on the autonomous vehicle to be substantially aligned with the location of the identified person; and

causing the directional visual message to be presented on the externally facing display.

4. A method of interaction between an autonomous vehicle and an external environment of the autonomous vehicle, the method comprising:

detecting the external environment of the autonomous vehicle to identify a person therein;

locating the identified person;

determining whether the identified person is potentially related to a future driving maneuver of the autonomous vehicle; and

responsive to determining that the identified person is potentially related to a future driving maneuver of the autonomous vehicle, causing the autonomous vehicle to send a directional visual message to the person, the directional visual message being sent directly to the person and being directly perceptible by the person without the person using a portable communication device,

the autonomous vehicle including a plurality of externally facing displays mounted on the autonomous vehicle, the plurality of externally facing displays including at least one externally facing display on a front portion of the autonomous vehicle, at least one externally facing display on a rear portion of the autonomous vehicle, at least one externally facing display on a first lateral side of the autonomous vehicle, and at least one externally facing display on a second lateral side of the autonomous vehicle, and wherein causing the autonomous vehicle to send the directional visual message includes;

selecting at least one of the plurality of externally facing displays that is substantially aligned with the location of the identified person; and

causing the directional visual message to be presented on the selected at least one of the plurality of externally facing displays.

5. A method of interaction between an autonomous vehicle and an external environment of the autonomous vehicle, the method comprising:

detecting the external environment of the autonomous vehicle to identify a person therein;

locating the identified person;

determining whether the identified person is potentially related to a future driving maneuver of the autonomous vehicle; and

responsive to determining that the identified person is potentially related to a future driving maneuver of the autonomous vehicle, causing the autonomous vehicle to send a directional message to the person, the directional message being sent directly to the person and being directly perceptible by the person without the person using a portable communication device;

wherein sending the directional message includes activating a humanoid figure to present a non-verbal gesture, and wherein the humanoid figure is a physical object located within or on the autonomous vehicle.

6. A system for interaction between an autonomous vehicle and an external environment of the autonomous vehicle, the system comprising:

an autonomous vehicle, the autonomous vehicle including;

a sensor system operable to detect the external environment of the autonomous vehicle to identify a person therein and operable to locate the identified person;

an external speaker, the external speaker being mounted on the autonomous vehicle, the external speaker having an associated position, the position of the external speaker being adjustable;

a processor located onboard the autonomous vehicle, the processor being programmed to initiate executable operations comprising;

determining whether the identified person is potentially related to a future driving maneuver of the autonomous vehicle; and

responsive to determining that the person is potentially related to a future driving maneuver of the autonomous vehicle, causing the autonomous vehicle to send a directional audial message to the person, the directional audial message being sent directly to the person and being directly perceptible by the person without the person using a portable communication device,

wherein causing the autonomous vehicle to send the directional audial message to the person includes;

adjusting the position of the external speaker to be substantially aligned with the location of the identified person, and

causing the directional audial message to be emitted from the external speaker.

7. A system for interaction between an autonomous vehicle and an external environment of the autonomous vehicle, the system comprising:

an autonomous vehicle, the autonomous vehicle including;

a sensor system operable to detect the external environment of the autonomous vehicle to identify a person therein and operable to locate the identified person;

a plurality of external speakers mounted on the autonomous vehicle, the plurality of external speakers including at least one external speaker on a front portion of the autonomous vehicle, at least one external speaker on a rear portion of the autonomous vehicle, at least one external speaker on a first lateral side of the autonomous vehicle, and at least one speaker on a second lateral side of the autonomous vehicle;

a processor located onboard the autonomous vehicle, the processor being programmed to initiate executable operations comprising;

determining whether the identified person is potentially related to a future driving maneuver of the autonomous vehicle; and

responsive to determining that the person is potentially related to a future driving maneuver of the autonomous vehicle, causing the autonomous vehicle to send a directional audial message to the person, the directional audial message being sent directly to the person and being directly perceptible by the person without the person using a portable communication device,

wherein causing the autonomous vehicle to send the directional audial message includes;

selecting at least one of the plurality of external speakers that is substantially aligned with the location of the identified person; and

causing the directional audial message to be emitted from the selected at least one of the plurality of external speakers.

8. A system for interaction between an autonomous vehicle and an external environment of the autonomous vehicle, the system comprising:

an autonomous vehicle, the autonomous vehicle including;

a sensor system operable to detect the external environment of the autonomous vehicle to identify a person therein and operable to locate the identified person;

an externally facing display mounted on the autonomous vehicle, the externally facing display having an associated position, the position of the externally facing display being adjustable;

a processor located onboard the autonomous vehicle, the processor being programmed to initiate executable operations comprising;

determining whether the identified person is potentially related to a future driving maneuver of the autonomous vehicle; and

responsive to determining that the person is potentially related to a future driving maneuver of the autonomous vehicle, causing the autonomous vehicle to send a directional visual message to the person, the directional visual message being sent directly to the person and being directly perceptible by the person without the person using a portable communication device,

wherein causing the autonomous vehicle to send a directional visual message to the person includes;

adjusting the position of the externally facing display to be substantially aligned with the location of the identified person; and

causing the directional visual message to be presented on the externally facing display.

9. A system for interaction between an autonomous vehicle and an external environment of the autonomous vehicle, the system comprising:

an autonomous vehicle, the autonomous vehicle including;

a sensor system operable to detect the external environment of the autonomous vehicle to identify a person therein and operable to locate the identified person;

a plurality of externally facing displays mounted on the autonomous vehicle, the plurality of externally facing displays including at least one externally facing display on a front portion of the autonomous vehicle, at least one externally facing display on a rear portion of the autonomous vehicle, at least one externally facing display on a first lateral side of the autonomous vehicle, and at least one externally facing display on a second lateral side of the autonomous vehicle;

a processor located onboard the autonomous vehicle, the processor being programmed to initiate executable operations comprising;

determining whether the identified person is potentially related to a future driving maneuver of the autonomous vehicle; and

responsive to determining that the person is potentially related to a future driving maneuver of the autonomous vehicle, causing the autonomous vehicle to send a directional visual message to the person, the directional visual message being sent directly to the person and being directly perceptible by the person without the person using a portable communication device,

wherein causing the autonomous vehicle to send a directional visual message to the person includes;

selecting at least one of the plurality of externally facing displays that is substantially aligned with the location of the identified person; and

causing the directional visual message to be presented on the selected at least one of the externally facing displays.

10. A system for interaction between an autonomous vehicle and an external environment of the autonomous vehicle, the system comprising:

an autonomous vehicle, the autonomous vehicle including;

a sensor system operable to detect the external environment of the autonomous vehicle to identify a person therein and operable to locate the identified person;

a humanoid figure, the humanoid figure being a physical object located within or on the autonomous vehicle;

a processor located onboard the autonomous vehicle, the processor being programmed to initiate executable operations comprising;

determining whether the identified person is potentially related to a future driving maneuver of the autonomous vehicle; and

responsive to determining that the person is potentially related to a future driving maneuver of the autonomous vehicle, causing the autonomous vehicle to send a directional message to the person, the directional message being sent directly to the person and being directly perceptible by the person without the person using a portable communication device,

wherein causing the autonomous vehicle to send a directional message to the person includes;

activating the humanoid figure to present a non-verbal gesture.

11. A computer program product for facilitating interaction between an autonomous vehicle and an external environment of the autonomous vehicle, the computer program product comprising a non-transitory computer readable storage medium having program code embodied therewith, the program code executable by a processor to perform a method comprising:

detecting the external environment of the autonomous vehicle to identify a person therein;

locating the identified person;

determining whether the identified person is potentially related to a future driving maneuver of the autonomous vehicle;

responsive to determining that the person is potentially related to a future driving maneuver of the autonomous vehicle, causing the autonomous vehicle to send a directional message to the person, the directional message being sent directly to the person and being directly perceptible by the person without the person using a portable communication device, the directional message being one of a directional audial message or a directional visual message,

when the directional message is a directional audial message, causing the autonomous vehicle to send the directional message to the person includes;

adjusting the position of an external speaker mounted on the autonomous vehicle to be substantially aligned with the location of the identified person; and

causing the directional audial message to be emitted from the external speaker, and

when the directional message is a directional visual message, causing the autonomous vehicle to send the directional message to the person includes;

adjusting a position of an externally facing display mounted on the autonomous vehicle to be substantially aligned with the location of the identified person; and

causing the directional visual message to be presented on the externally facing display.

12. A method of interaction between an autonomous vehicle and an external environment of the autonomous vehicle, the method comprising:

detecting a non-verbal human gesture in the external environment;

identifying the detected non-verbal human gesture;

determining, using a processor, a future driving maneuver based at least in part on the identified non-verbal human gesture; and

causing the autonomous vehicle to implement the determined future driving maneuver.

13. The method of claim 12, wherein identifying the detected non-verbal human gesture includes comparing the detected non-verbal human gesture to a set of predetermined non-verbal human gestures.

14. The method of claim 12, wherein the non-verbal human gesture is made by a person in the external environment, and further including:

locating the person; and

causing the autonomous vehicle to send a directional message to the person.

15. A system for interaction between an autonomous vehicle and an external environment of the autonomous vehicle, the system comprising:

an autonomous vehicle, the autonomous vehicle including;

a sensor system operable to detect a non-verbal gesture made by a person in the external environment;

a processor being programmed to initiate executable operations comprising;

identifying a detected non-verbal human gesture;

determining a future driving maneuver based at least in part on the identified non-verbal human gesture; and

causing the autonomous vehicle to implement the determined future driving maneuver.

16. The system of claim 15, wherein identifying the detected non-verbal human gesture includes comparing the detected non-verbal human gesture to a set of predetermined non-verbal human gestures.

17. The system of claim 15, wherein the non-verbal human gesture is made by a person in the external environment, and wherein the initiate executable operations further include:

locating the person; and

causing the autonomous vehicle to send a directional message to the person.

18. A computer program product for interaction between an autonomous vehicle and an external environment of the autonomous vehicle, the computer program product comprising a non-transitory computer readable storage medium having program code embodied therewith, the program code executable by a processor to perform a method comprising:

detecting a non-verbal human gesture in the external environment;

identifying the detected non-verbal human gesture;

determining a future driving maneuver based at least in part on the identified non-verbal human gesture; and

causing the autonomous vehicle to implement the determined future driving maneuver.