Training algorithm for collision avoidance using auditory data

• US 10,055,675 B2
• Filed: 06/15/2016
• Issued: 08/21/2018
• Est. Priority Date: 06/15/2016
• Status: Active Grant

• Current Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Original Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Inventors: Micks, Ashley Elizabeth, Jain, Jinesh J, Han, Kyu Jeong, Banvait, Harpreetsingh
• Primary Examiner(s): Allison, Andrae S
• Application Number:
  US15/183,610
• Publication Number:
  US 20170364776A1
• Time in Patent Office: 797 Days
• Field of Search: None
• CPC Class Codes: G01S 13/006 Theoretical aspects G01S7/418, G01S13/9094, G01S13/958 take precedence
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 17/006 Theoretical aspects
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G06F 30/15 Vehicle, aircraft or watercraft design
• CPC Class Codes: G06F 30/20 Design optimisation, verification or simulation optimisation, verification or simulation of circuit designs G06F30/30
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G09B 9/54 Simulation of radar G09B9/40 takes precedence
• CPC Class Codes: Y02T 90/00 Enabling technologies or technologies with a potential or indirect contribution to GHG emissions mitigation

Abstract

A machine learning model is trained by defining a scenario including models of vehicles and a typical driving environment. A model of a subject vehicle is added to the scenario and sensor locations are defined on the subject vehicle. A perception of the scenario by sensors at the sensor locations is simulated. The scenario further includes a model of a parked vehicle with its engine running. The location of the parked vehicle and the simulated outputs of the sensors perceiving the scenario are input to a machine learning algorithm that trains a model to detect the location of the parked vehicle based on the sensor outputs. A vehicle controller then incorporates the machine learning model and estimates the presence and/or location of a parked vehicle with its engine running based on actual sensor outputs input to the machine learning model.

Claims

1. A method comprising:

defining, by a computer system, a three dimensional (3D) model;

simulating, by the computer system, two or more sensor outputs from sound from a parked vehicle with its engine running incident on two or more sensor locations of a subject vehicle in the 3D model; and

training, by the computer system, a machine-learning model using a location of the parked vehicle in the 3D model and the two or more sensor outputs.

2. The method of claim 1, further comprising:

defining on the subject vehicle one or more camera locations;

simulating detection of images at the one or more camera locations; and

training the machine-learning model using both the images and the two or more sensor outputs.

3. The method of claim 2 further comprising:

defining on the subject vehicle a RADAR sensor location;

simulating a RADAR sensor output according to the 3D model; and

training the machine learning model using all of the images, the RADAR sensor output, and the two or more sensor outputs.

4. The method of claim 3, further comprising:

defining on the subject vehicle a LIDAR sensor location;

simulating a sequence of point clouds detected from the LIDAR sensor location of the 3D model; and

training the machine learning model using all of the images, the RADAR sensor output, the sequence of point clouds, and the two or more sensor outputs.

5. The method of claim 1, wherein defining the 3D model further includes defining a plurality of intervening vehicles between the parked vehicle and the subject vehicle.

6. The method of claim 1, wherein the machine-learning model is a deep neural network.

7. The method of claim 1, wherein simulating the two or more sensor outputs from the sound of the plurality of vehicles incident on the two or more sensor locations comprises simulating propagation of sound from an engine of the parked vehicle, around one or more intervening vehicles between the parked vehicle, and incident on the subject vehicle.

8. The method of claim 1, wherein the parked vehicle is modeled as a vehicle emitting sounds corresponding to a running vehicle engine.

9. The method of claim 1, further comprising:

providing a vehicle including a vehicle controller and two or more microphones;

programming the vehicle controller with the machine-learning model;

receiving, by the vehicle controller, two or more audio streams from the two or more microphones;

inputting, by the vehicle controller, the two or more audio streams to the machine-learning model;

(a) determining, by the vehicle controller, that the machine-learning model indicates that the two or more audio streams currently indicate an actual parked vehicle with its engine running is present;

in response to (a), performing obstacle avoidance with respect to a potential path of the actual parked vehicle.

10. The method of claim 9, further comprising:

(b) determining, by the vehicle controller, subsequent to (a) that the machine-learning model indicates that the two or more audio streams do not indicate that an engine of a second actual parked vehicle is running;

in response to (b), refraining from performing obstacle avoidance with respect to a potential path of the second actual parked vehicle.

11. The method of claim 9, further comprising:

if the machine-learning model indicates that the two or more audio streams do not currently indicate an actual parked vehicle with its engine running is present, at least one of refrain from outputting the alert by the vehicle controller and permit, refrain from performing obstacle avoidance with respect to a potential path of the actual parked vehicle.

12. A system comprising one or more processors and one or more memory devices coupled to the one or more processors, the one or more memory devices storing executable code effective to:

define a three dimensional model including a parking area and a plurality of stationary parked vehicles, a running parked vehicle, and a subject vehicle including two or more sensor locations positioned on the parking area;

simulate two or more sensor outputs from sound from the running parked vehicle incident on the two or more sensor locations; and

train a machine-learning model using a location of the stationary parked vehicle and the two or more sensor outputs over time.

13. The system of claim 12, wherein the executable code is further effective to cause the one or more processors to:

define on the subject vehicle one or more camera locations;

simulate detection of images at the one or more camera locations; and

train the machine-learning model using both the images and the two or more sensor outputs.

14. The system of claim 13, wherein the executable code is further effective to cause the one or more processors to:

define on the subject vehicle a RADAR sensor location;

simulate a RADAR sensor output according to the 3D model; and

train the machine learning model using all of the images, the RADAR sensor output, and the two or more sensor outputs.

15. The system of claim 14, wherein the executable code is further effective to cause the one or more processors to:

define on the subject vehicle a LIDAR sensor location;

simulate a sequence of point clouds detected from the LIDAR sensor location of the 3D model; and

train the machine learning model using all of the images, the RADAR sensor output, the sequence of point clouds, and the two or more sensor outputs.

16. The system of claim 12, wherein the executable code is further effective to cause the one or more processors to define the 3D model defining a plurality of intervening vehicles between the parked vehicle and the subject vehicle.

17. The system of claim 12, wherein the machine-learning model is a deep neural network.

18. The system of claim 12, wherein the executable code is further effective to cause the one or more processors to simulate the two or more sensor outputs from the sound of the plurality of vehicles incident on the two or more sensor locations by simulating Doppler effects and propagation distances of the sound.

19. The system of claim 12, wherein the executable code is further effective to cause the one or more processors to model the parked vehicle as a vehicle emitting sounds corresponding to a running vehicle engine.

20. The system of claim 12, further comprising a vehicle including a vehicle controller and two or more microphones;

wherein the vehicle controller is programmed with the machine-learning model;

wherein the vehicle controller is further programmed to;

receive two or more audio streams from the two or more microphones;

input the two or more audio streams to the machine-learning model;

if the machine-learning model indicates that the two or more audio streams currently indicate an actual parked vehicle with its engine running, performing obstacle avoidance with respect to a potential path of the actual parked vehicle.