Vehicle ambient audio classification via neural network machine learning

• US 10,276,187 B2
• Filed: 10/19/2016
• Issued: 04/30/2019
• Est. Priority Date: 10/19/2016
• Status: Active Grant

• Current Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Original Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Inventors: Gross, Ethan
• Primary Examiner(s): Ortiz-Sanchez, Michael
• Application Number:
  US15/298,077
• Publication Number:
  US 20180108369A1
• Time in Patent Office: 923 Days
• CPC Class Codes: B60R 2025/1016 Remote signals alerting owner or authorities, e.g. radio signals
• CPC Class Codes: B60R 25/1001 Alarm systems associated with another car fitting or mechanism, e.g. door lock or knob, pedals features related to the general power supply B60R25/1018
• CPC Class Codes: B60R 25/1009 Sonic sensors; Signal treatment therefor
• CPC Class Codes: B60R 25/302 using recording means, e.g. black box
• CPC Class Codes: B60R 25/31 of human presence inside or outside the vehicle
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G07C 5/085 using electronic data carriers
• CPC Class Codes: G07C 9/00896 specially adapted for particular uses
• CPC Class Codes: G08B 13/1672 using sonic detecting means, e.g. a microphone operating in the audio frequency range
• CPC Class Codes: G08B 21/00 Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
• CPC Class Codes: G08B 21/22 responsive to presence or absence of persons
• CPC Class Codes: G08B 21/24 Reminder alarms, e.g. anti-loss alarms
• CPC Class Codes: G10L 25/24 the extracted parameters being the cepstrum
• CPC Class Codes: G10L 25/30 using neural networks
• CPC Class Codes: G10L 25/51 for comparison or discrimination

Abstract

A method and an apparatus for detecting and classifying sounds around a vehicle via neural network machine learning are described. The method involves an audio recognition system that may determine the origin of the sounds being inside or outside of a vehicle and classify the sounds into different categories such as adult, child, or animal sounds. The audio recognition system may communicate with a plurality of sensors in and around the vehicle to obtain information of conditions of the vehicle. Based on information of the sounds and conditions of the vehicles, the audio recognition system may determine whether an occupant or the vehicle is at risk and send alert messages or issue warning signals.

Claims

1. A method, comprising:

receiving audible information from one or more microphones;

receiving vehicle information of one or more conditions of a vehicle from one or more sensors;

determining whether the vehicle is at risk of theft or whether an occupant of the vehicle is at risk of danger based on the vehicle information and the audible information by;

determining that the vehicle is at risk of theft responsive to the audible information indicating a sound from outside the vehicle and the vehicle information indicating that the vehicle is tilted or moved, doors of the vehicle are locked with no occupant inside, and a key of the vehicle is not inside the vehicle; and

determining that the occupant is at risk of danger responsive to the audible information indicating that the occupant is inside the vehicle and the vehicle information indicating that the doors of the vehicle are locked and a temperature inside the vehicle exceeds a threshold temperature; and

performing some or all actions of a plurality of actions upon determining that the occupant or the vehicle is at risk, the plurality of actions comprising;

wirelessly transmitting alert messages to a wireless receiver;

activating headlights of the vehicle to flash the headlights;

activating a horn of the vehicle to honk the horn; and

controlling locks on the doors to unlock the doors.

2. The method of claim 1, wherein the receiving of the audible information from the one or more microphones comprises determining whether the occupant is inside the vehicle based on information of a first neural network and a second neural network.

3. The method of claim 2, wherein the determining of whether the occupant is inside the vehicle based on information of the first neural network and the second neural network by performing operations comprising:

detecting a plurality of sounds in and around the vehicle;

recording the sounds into a plurality of audio files;

determining whether the sounds are originated from inside or outside of the vehicle based on information of the first neural network; and

classifying the sounds into a plurality of categories based on information of the second neural network.

4. The method of claim 3, wherein the detecting of the sounds in and around the vehicle comprises initiating the detecting of the sounds by turning off the vehicle or by pushing an engine stop button, a key not inserted in an ignition keyhole, and one or more doors of the vehicle being locked.

5. The method of claim 3, wherein the determining of whether the sounds are originated from inside or outside of the vehicle comprises machine learning of the audio files via the first neural network.

6. The method of claim 3, wherein the classifying of the sounds into the plurality of categories comprises machine learning of the audio files via the second neural network.

7. The method of claim 5, wherein the machine learning of the audio files via the first neural network comprises:

processing the audio files with a mel-frequency cepstrum algorithm by calculating a respective set of mel-frequency cepstrum coefficients for each of the audio files; and

processing the mel-frequency cepstrum coefficients by the first neural network to learn a set of characteristics of the respective audio file.

8. The method of claim 7, wherein the processing of the audio files comprises performing logarithmic and linear Fourier Transform of the audio files on a Mel scale.

9. The method of claim 5, wherein the machine learning of the audio files via the second neural network comprises:

processing the audio files with a mel-frequency cepstrum algorithm by calculating a respective set of mel-frequency cepstrum coefficients for each of the audio files; and

processing the mel-frequency cepstrum coefficients by the second neural network to learn a set of characteristics of the respective audio file.

10. The method of claim 9, wherein the processing of the audio files comprises performing logarithmic and linear Fourier Transform of the audio files on a Mel scale.

11. An apparatus implemented as an audio recognition system of a vehicle, comprising:

one or more microphones configured to detect a plurality of sounds;

one or more sensors configured to monitor one or more conditions of the vehicle;

a memory configured to store raw data of the sounds and a plurality of audio files recorded from the sounds;

a processor configured to perform operations comprising;

recording the sounds in the memory as the audio files;

processing the audio files with a mel-frequency cepstrum algorithm by calculating a respective set of mel-frequency cepstrum coefficients for each of the audio files;

receiving information of the one or more conditions of the vehicle from the sensors;

determining whether an occupant is inside the vehicle based on information of the first neural network and the second neural network such that the occupant is determined to be at risk of danger responsive to the audible files indicating that the occupant is inside the vehicle and the vehicle information indicating that doors of the vehicle are locked and a temperature inside the vehicle exceeds a threshold temperature;

determining whether the vehicle is at risk of theft or vandalism based on information of the one or more conditions of the vehicle and information of the first neural network and the second neural network such that the vehicle is determined to be at risk of theft responsive to the audible files indicating a sound from outside the vehicle and the vehicle information indicating that the vehicle is tilted or moved, the doors of the vehicle are locked with no occupant inside, and a key of the vehicle is not inside the vehicle; and

performing some or all actions of a plurality of actions upon determining that the occupant or the vehicle is at risk, the plurality of actions comprising;

wirelessly transmitting alert messages to a wireless receiver;

activating headlights of the vehicle to flash the headlights;

activating a horn of the vehicle to honk the horn; and

controlling locks on the doors to unlock the doors;

a first neural network comprising at least a first hidden layer with a first set of nodes; and

a second neural network comprising at least a second hidden layer with a second set of nodes.

12. The apparatus of claim 11, wherein, in processing of the audio files, the processor is configured to perform a logarithmic operation and a linear Fourier Transform of the audio files on a Mel scale.

13. The apparatus of claim 11, wherein the first neural network is configured to perform operations comprising:

processing the mel-frequency cepstrum coefficients to learn a set of characteristics of the respective audio file recorded from the sounds; and

determining whether the sounds are originated from inside or outside of the vehicle.

14. The apparatus of claim 11, wherein the second neural network is configured to perform operations comprising:

processing the mel-frequency cepstrum coefficients to learn a set of characteristics of the respective audio file recorded from the sounds; and

classifying the sounds into a plurality of categories.

15. The apparatus of claim 11, wherein the one or more conditions of the vehicle comprise one or more of:

a key being not inserted in an ignition keyhole;

a key being not inside the vehicle;

the vehicle being lifted;

the vehicle being in motion;

one or more doors of the vehicle being locked; and

a temperature inside the vehicle reaching or exceeding a threshold temperature.

16. The apparatus of claim 11, further comprising:

a communication device configured to convey a plurality of instructions to a plurality of components of the vehicle regarding the plurality of actions to be performed by the components,

wherein, in performing the some or all actions, the processor is configured to provide, via the communication device, one or more instructions to one or more components of the vehicle.