Method for robotic vehicle communication with an external environment via acoustic beam forming

• US 9,878,664 B2
• Filed: 11/04/2015
• Issued: 01/30/2018
• Est. Priority Date: 11/04/2015
• Status: Active Grant

• Current Assignee: Zoox Inc. (Amazon.com, Inc.)
• Original Assignee: Zoox Inc. (Amazon.com, Inc.)
• Inventors: Kentley-Klay, Timothy David, Levinson, Jesse Sol, Lind, Amanda Blair
• Primary Examiner(s): Smith, Jelani A
• Application Number:
  US14/756,993
• Publication Number:
  US 20170120814A1
• Time in Patent Office: 818 Days
• CPC Class Codes: B60C 5/008 Low pressure tyres, e.g. for all terrain vehicles
• CPC Class Codes: B60Q 5/006 indicating risk of collision between vehicles or with pedestrians
• CPC Class Codes: B60Q 5/008 for signaling silent vehicles, e.g. for warning that a hybrid or electric vehicle is approaching
• CPC Class Codes: G10K 11/346 using phase variation

Abstract

Systems and methods implemented in algorithms, software, firmware, logic, or circuitry may be configured to process data to determine whether an object external to an autonomous vehicle is a person (e.g., such as a pedestrian) or other classification (e.g., such as a vehicle), and may be further configured to determine a position of the person relative to the autonomous vehicle. Logic may be configured to direct acoustic energy (e.g., via vehicular acoustic beam-forming) to an object external to the autonomous vehicle as an audible acoustic alert. The vehicle-related acoustic beam may be directed to a driver in another vehicle. Logic may be configured to track the motion of external objects, such as a pedestrian crossing from one side of the street to the other, and may correspondingly steer the direction of the vehicle-related acoustic beam(s) to track the person'"'"'s movement.

Claims

1. A method comprising:

calculating a trajectory of an autonomous vehicle in an environment, the autonomous vehicle including an acoustic beam steering array, the acoustic beam steering array including at least a first speaker adjacent to a second speaker, wherein the first speaker is spaced from the second speaker by a wave front distance;

determining a location of an object disposed in the environment, the object having an object type;

predicting an object trajectory of the object based at least in part on the object type and the location of the object;

estimating a threshold location in the environment based at least in part on the object trajectory and the trajectory of the autonomous vehicle, the threshold location associated with an acoustic alert;

selecting an audio signal to associate with the acoustic alert;

detecting that the location of the object is coincident with the threshold location;

determining, based at least in part on the location of the object, a direction of propagation to emit a beam of steered acoustic energy;

calculating, based at least in part on the wave front distance and a speed of sound, a first signal delay associated with the first speaker;

calculating, based at least in part on the wave front distance and the speed of sound, a second signal delay associated with the second speaker; and

causing the acoustic beam steering array to emit the beam of steered acoustic energy by at least delaying-application of the audio signal to a signal input of one or more amplifiers coupled with the first speaker and the second speaker by one or more times based at least on one or more of the first signal delay or the second signal delay.

2. The method of claim 1, further comprising:

identifying a change in a driving operation of the autonomous vehicle,

and wherein selecting the audio signal is based at least in part on the change in the driving operation.

3. The method of claim 2, further comprising:

capturing sound associated with the driving operation of the autonomous vehicle, and

adjusting one or more characteristics of the audio signal based at least in part on the sound.

4. The method of claim 1, wherein the threshold location comprises a plurality of threshold locations, and wherein selecting the audio signal comprises selecting the audio signal based at least in part on a threat level.

5. The method of claim 1, wherein the object comprises a first object, and wherein the method further comprises:

detecting a second object in the environment;

identifying an object type for the second object;

associating a first priority with the first object based at least in part on the object type for the first object;

associating a second priority with the second object based at least in part on the object type for the second object; and

determining to process data related to the first object before processing data related to the second object based at least in part on the first priority being a higher priority than the second priority.

6. The method of claim 5, wherein associating the first priority comprises determining one or more of a motion of the first object the location of the first object, a predicted motion of the first object and the trajectory of the autonomous vehicle.

7. The method of claim 1, further comprising activating a system of the autonomous vehicle based at least in part on the object trajectory of the object, wherein the system is one or more of a braking system, a propulsion system, and a signaling system.

8. The method of claim 1, wherein the acoustic beam steering array is a first acoustic beam steering array and the autonomous vehicle includes a second acoustic beam steering array, and wherein the method further comprises:

selecting one of the first acoustic beam steering array or the second acoustic beam steering array to emit the beam of steered acoustic energy based at least in part on a direction of travel of the autonomous vehicle, wherein the autonomous vehicle is configured to travel bi-directionally, and wherein the first acoustic beam steering array is disposed on a first half of the autonomous vehicle, and the second acoustic beam steering array is disposed on a second half of the autonomous vehicle.

9. The method of claim 1, further comprising:

accessing drive operation sounds of the autonomous vehicle; and

modulating the audio signal using the drive operation sounds.

10. The method of claim 1, further comprising:

generating, at a perception system, the location of the object;

receiving, at the acoustic beam steering array, the location of the object, the acoustic beam steering array being in communication with the perception system;

receiving, at a planner system in communication with the perception system, the location of the object, an object track, and data representing an object classification;

accessing, by the planner system, data representing object types;

comparing, at the planner system, the object classification and the object track with the data representing the object types to determine the object type;

accessing, by the planner system, object dynamics; and

comparing, at the planner system, the object type and the location of the object with the data representing the object dynamics to determine the object trajectory.

11. A system, comprising:

sensors operative to generate sensor data for an autonomous vehicle located within an environment;

an acoustic beam steering array operative to emit a beam of steered acoustic energy into the environment, the acoustic beam steering array including at least a first speaker adjacent to a second speaker, wherein the first speaker is spaced from the second speaker by a wave front distance;

one or more amplifiers coupled to the first speaker and the second speaker; and

one or more processors configured to perform acts comprising;

determining a location of an object disposed in the environment based at least in part on the sensor data;

selecting an audio signal to associate with an acoustic alert;

determining a direction of propagation to emit the beam of steered acoustic energy, wherein the direction is based at least in part on the location of the object;

calculating a signal delay based at least in part on the wave front distance; and

causing the acoustic beam steering array to emit the beam of steered acoustic energy by at least delaying application of the audio signal based at least in part on the signal delay.

12. The system of claim 11, wherein the acts further comprise:

predicting a trajectory of the object based at least in part on an object type associated with the object and the location of the object; and

estimating a threshold location in the environment based at least in part on the trajectory of the object,

and wherein causing the acoustic beam steering array to emit the beam of steered acoustic energy is based at least in part on detecting that the object is coincident with the threshold location.

13. The system of claim 11, further comprising one or more microphones operative to capture sound;

and wherein the acts further comprise identifying a change in a driving operation of the autonomous vehicle based at least in part on the sound; and

wherein selecting the audio signal is based at least in part on the change in the driving operation.

14. The system of claim 11, further comprising one or more microphones operative to capture sound;

and wherein the acts further comprise adjusting one or more characteristics of the audio signal based at least in part on the sound.

15. The system of claim 11, wherein the acts further comprise:

determining at least a first threshold location and a second threshold location based at least in part on a first trajectory of the autonomous vehicle and a second trajectory of the object;

identifying a first threat level associated with the first threshold location; and

identifying a second threat level associated with the second threshold location,

and wherein selecting the audio signal comprises selecting a first audio signal based at least in part on identifying the first threat level and selecting a second audio signal based at least in part on identifying the second threat level.

16. The system of claim 11, wherein the acts further comprise:

determining calculating the signal delay comprises;

calculating a first signal delay associated with the first speaker based at least in part on the wave front distance; and

determining calculating a second signal delay associated with the second speaker based at least in part on the wave front distance; and

delaying application of the audio signal based at least in part on the signal delay comprises delaying application of the audio signal to a signal input of at least one of the one or more amplifiers by one or more times based at least in part on one or more of the first signal delay or the second signal delay.

17. The system of claim 11, wherein the acts further comprise:

accessing speaker data associated with the first speaker and the second speaker, the speaker data including a first speaker position in the acoustic beam steering array and a second speaker position in the acoustic beam steering array;

calculating a first gain associated with the first speaker and a second gain associated with the second speaker, wherein one or more of the first speaker or the second speaker are coupled to an output of an amplifier having a gain input; and

applying to the gain input one or more of the first gain or the second gain.

18. The system of claim 11, wherein the acts further comprise:

accessing drive operation sounds of the autonomous vehicle; and

modulating the audio signal using the drive operation sounds.

19. The system of claim 11, wherein the acts further comprise:

determining an object type for the object;

accessing object dynamics based at least in part on the object type; and

determining a trajectory of the object based at least in part on the object dynamics,

and wherein determining the direction of propagation to emit the beam of steered acoustic energy is based at least in part on the trajectory.

20. The system of claim 11, wherein the object comprises a first object, and wherein the acts further comprise:

detecting a second object in the environment;

identifying a first object type for the first object;

identifying a second object type for the second object;

associating a first priority with the first object based at least in part on the first object type for the first object;

associating a second priority with the second object based at least in part on the second object type for the second object; and

determining to process data related to the first object before processing data related to the second object based at least in part on the first priority being a higher priority than the second priority.

21. The system of claim 20, wherein associating the first priority comprises determining one or more of a motion of the first object, the location of the first object, a predicted motion of the first object, and a trajectory of the autonomous vehicle.

22. The system of claim 11, wherein the acts further comprise activating a control system of the autonomous vehicle based at least in part on a trajectory of the object, wherein the control system is one or more of a braking system, a propulsion system, and a signaling system.

23. The system of claim 11, wherein:

the acoustic beam steering array comprises a first acoustic beam steering array disposed on a first half of the autonomous vehicle;

the system further comprises a second acoustic beam steering array that is disposed on a second half of the autonomous vehicle; and

the autonomous vehicle is configured to travel bi-directionally.

24. The system of claim 11, the acts further comprising:

determining an initial location of the object;

predicting an object trajectory of the object based at least in part on the object type and the initial location of the object;

estimating a threshold location in the environment based at least in part on the object trajectory and the trajectory of the autonomous vehicle, the threshold location associated with the acoustic alert; and

detecting that the location of the object is coincident with the threshold location,

wherein causing the acoustic beam steering array to emit the beam of steered acoustic energy is based at least in part on detecting that the location of the object is coincident with the threshold location.

25. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:

calculating an autonomous vehicle trajectory in an environment;

determining a first location of an object disposed in the environment, the object having an object type;

predicting an object trajectory based at least in part on the object type and the first location of the object;

determining, based at least in part on the autonomous vehicle trajectory and the object trajectory, a threshold location in the environment;

associating the threshold location with emitting an acoustic alert;

selecting an audio signal to associate with the acoustic alert;

detecting that a second location of the object is coincident with the threshold location; and

based at least in part on detecting that the second location of the object is coincident with the threshold location, causing an acoustic beam steering array to emit a beam of steered acoustic energy indicative of the audio signal into the environment to convey information about the autonomous vehicle to the object.

26. The one or more non-transitory computer-readable media of claim 25, wherein determining the threshold location comprises determining the threshold location such that the autonomous vehicle can emit the beam of steered acoustic energy for a given period of time before the first trajectory of the autonomous vehicle intersects with the second trajectory of the object.

27. The one or more non-transitory computer-readable media of claim 25, wherein the threshold location comprises a first threshold location, the acoustic alert comprises a first acoustic alert, the audio signal comprises a first audio signal and the beam of steered acoustic energy comprises a first beam of steered acoustic energy, and wherein the acts further comprise:

determining, based at least in part on the autonomous vehicle and the object trajectory, a second threshold location in the environment;

associating the second threshold location with emitting a second acoustic alert;

selecting a second audio signal to associate with the second acoustic alert;

detecting that a third location of the object is coincident with the second threshold location; and

based at least in part on detecting that the third location of the object is coincident with the second threshold location, causing the acoustic beam steering array to emit a second beam of steered acoustic energy indicative of the second audio signal into the environment.

28. The one or more non-transitory computer-readable media of claim 27, wherein the first threshold location includes a greater distance from the autonomous vehicle than the second threshold location.