Collision avoidance using auditory data

• US 9,873,428 B2
• Filed: 10/27/2015
• Issued: 01/23/2018
• Est. Priority Date: 10/27/2015
• Status: Active Grant

• Current Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Original Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Inventors: Banvait, Harpreetsingh, Han, Kyu Jeong, Jain, Jinesh J
• Primary Examiner(s): Black, Thomas G
• Assistant Examiner(s): Lewandroski, Sara J
• Application Number:
  US14/924,187
• Publication Number:
  US 20170113684A1
• Time in Patent Office: 819 Days
• CPC Class Codes: B60W 10/04 including control of propulsion units
• CPC Class Codes: B60W 10/06 including control of combustion engines
• CPC Class Codes: B60W 10/18 including control of braking systems
• CPC Class Codes: B60W 10/20 including control of steering systems
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/54 Audio sensitive means, e.g. ultrasound
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 30/09 Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: B60W 30/0956 the prediction being responsive to traffic or environmental parameters
• CPC Class Codes: G05D 1/0255 using acoustic signals, e.g. ultra-sonic singals sonar systems designed for anti-collision purposes G01S15/93
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks

Abstract

A controller for an autonomous vehicle receives audio signals from one or more microphones. The outputs of the microphones are pre-processed to enhance audio features that originated from vehicles. The outputs may also be processed to remove noise. The audio features are input to a machine learning model that classifies the source of the audio features. For example, features may be classified as originating from a vehicle. A direction to a source of the audio features is determined based on relative delays of the audio features in signals from multiple microphones. Where audio features are classified with an above-threshold confidence as originating from a vehicle, collision avoidance is performed with respect to the direction to the source of the audio features.

Claims

1. A system for an autonomous vehicle comprising:

two or more microphones mounted to the autonomous vehicle;

a controller executing

a pre-processor programmed to detect audio features in two or more audio streams from the two or more microphones;

a collision avoidance module programmed to classify the audio features and a direction to a source thereof, and, if the class for the sound source is a vehicle, invoke obstacle avoidance with respect to the direction;

wherein the collision avoidance module is further programmed to;

classify the audio features by inputting the audio features into a machine learning model; and

wherein the machine-learning model outputs a confidence value indicating a probability that the audio features correspond to a vehicle.

2. The system of claim 1, wherein the collision avoidance module is further programmed to invoke obstacle avoidance with respect to the direction to the source of the audio features by actuating at least one of a steering actuator, accelerator actuator, and brake actuator of the autonomous vehicle effective to avoid collision with the source of the audio features.

3. The system of claim 1, wherein the collision avoidance module is further programmed to:

receive two or more image streams from two or more cameras mounted to the autonomous vehicle;

identify a set of potential obstacles among the plurality of vehicle images;

evaluate possible collisions between the autonomous vehicle and the set of potential obstacles and the source of the audio features; and

activate at least one of a steering actuator, accelerator actuator, and brake actuator of the autonomous vehicle effective to avoid collisions with the set of potential obstacles.

4. The system of claim 3, wherein the two or more microphones are configured to detect sounds from the source of the audio features when the source of the audio features is not in a field of view of any of the two or more cameras.

5. The system of claim 1, wherein the collision avoidance module is further programmed to identify the audio features by filtering the two or more audio streams to obtain two or more filtered signals each including one or more of the audible features.

6. The system of claim 5, wherein the collision avoidance module is further programmed to filter the two or more audio streams by removing ambient noise from the two or more audio streams.

7. The system of claim 1, wherein the machine learning model is a deep neural network.

8. The system of claim 1, wherein the collision avoidance module is further programmed to detect the direction to the source of the audio features by evaluating a time difference between the audio features.

9. A method for obstacle detection in an autonomous vehicle, the method comprising:

receiving, by a controller including one or more processing devices, two or more audio streams from two or more microphones mounted to the autonomous vehicle;

detecting, by the controller, audio features in the two or more audio streams;

detecting, by the controller, a direction to a sound source according to the audio features;

identifying, by the controller, a class for the sound source according to the audio features; and

determining, by the controller, that the class for the sound source is a vehicle,

in response to determining that the class for the sound source is a vehicle, invoking obstacle avoidance with respect to the direction to the sound source;

wherein identifying the class for the sound source according to the audio features comprises inputting, by the controller, the audio features into a machine learning model; and

wherein the method further comprises outputting, by the machine-learning model, a confidence value indicating a probability that the audio features correspond to a vehicle.

10. The method of claim 9, wherein the machine learning model is a deep neural network.

11. The method of claim 9, wherein detecting the direction to the sound source according to the audio features comprises evaluating, by the controller, a time difference between the audio features.

12. The method of claim 9, further comprising invoking obstacle avoidance with respect to the direction to the sound source by actuating at least one of a steering actuator, accelerator actuator, and brake actuator of the autonomous vehicle effective to avoid collision with the sound source.

13. The method of claim 9, further comprising:

receiving, by the controller, two or more image streams from two or more cameras mounted to the autonomous vehicle;

identifying, by the controller, a set of potential obstacles among the plurality of vehicle images;

evaluating, by the controller, possible collisions between the autonomous vehicle and the set of potential obstacles and the sound source; and

activating, by the controller, at least one of a steering actuator, an accelerator actuator, and a brake actuator of the autonomous vehicle effective to avoid collisions with the set of potential obstacles.

14. The method of claim 13, wherein the sound source is not in a field of view of any of the two or more cameras.

15. The method of claim 9, wherein identifying the audible features comprises filtering, by the controller, the two or more audio streams to obtain two or more filtered signals each including one or more of the audible features.

16. The method of claim 15, wherein filtering the two or more audio streams further comprises removing ambient noise from the two or more audio streams.