Method and system for detecting an audio event for smart home devices

• US 9,965,685 B2
• Filed: 06/12/2015
• Issued: 05/08/2018
• Est. Priority Date: 06/12/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Matsuoka, Yoky, Nongpiur, Rajeev Conrad, Dixon, Michael
• Primary Examiner(s): WOZNIAK, JAMES S
• Application Number:
  US14/737,678
• Publication Number:
  US 20160364963A1
• Time in Patent Office: 1,061 Days
• Field of Search: 704231, 704233, 704236, 704239, 704274
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 1/08 using electric transmission ; transformation of alarm signals to electrical signals from a different medium, e.g. transmission of an electric alarm signal upon detection of an audible alarm signal
• CPC Class Codes: G08B 21/0208 Combination with audio or video communication, e.g. combination with "baby phone" function
• CPC Class Codes: G10L 25/27 characterised by the analysis technique
• CPC Class Codes: G10L 25/51 for comparison or discrimination
• CPC Class Codes: G10L 25/72 for transmitting results of analysis
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04R 2410/05 Noise reduction with a separate noise microphone
• CPC Class Codes: H04R 3/00 Circuits for transducers , loudspeakers or microphones

Abstract

This application discloses a method implemented by an electronic device to detect a signature event (e.g., a baby cry event) associated with an audio feature (e.g., baby sound). The electronic device obtains a classifier model from a remote server. The classifier model is determined according to predetermined capabilities of the electronic device and ambient sound characteristics of the electronic device, and distinguishes the audio feature from a plurality of alternative features and ambient noises. When the electronic device obtains audio data, it splits the audio data to a plurality of sound components each associated with a respective frequency or frequency band and including a series of time windows. The electronic device further extracts a feature vector from the sound components, classifies the extracted feature vector to obtain a probability value according to the classifier model, and detects the signature event based on the probability value.

Claims

1. A method for detecting a signature event associated with an audio feature, comprising:

on an electronic device having one or more processors and memory storing one or more programs for execution by the one or more processors, automatically and without user intervention;

obtaining from a remote server a classifier model that distinguishes an audio feature from a plurality of alternative features and ambient noises, wherein the classifier model is determined by the remote server according to a number of false positives generated by the classifier model, predefined capabilities of the electronic device and ambient sound characteristics of the electronic device, the predefined capabilities of the electronic device comprising one or more of computational capabilities, storage capabilities, and caching capabilities;

obtaining audio data associated with an audio signal;

splitting the audio data to a plurality of sound components each associated with a respective frequency or frequency band and including a series of time windows;

statistically analyzing each of the plurality of sound components across the series of time windows;

extracting a feature vector from the plurality of sound components based on the statistical analysis, the feature vector including a plurality of elements that are arranged according a predetermined order;

classifying the extracted feature vector based on the classifier model to obtain a probability value indicating whether the audio signal includes the audio feature within the series of time windows; and

detecting the signature event associated with the audio feature based on the probability value and issuing an alert indicating occurrence of the signature event.

2. The method of claim 1, the feature vector including a first subset of elements associated with energy levels of a first subset of sound components, and a second subset of elements associated with harmonic characteristics of a second subset of sound components, wherein the first and second subsets of elements in the feature vector are arranged according a predetermined order.

3. The method of claim 2, wherein the first subset of elements are associated with variations of the energy levels for each of the first subset of sound components with respect to the series of time windows.

4. The method of claim 2, wherein the first subset of elements includes one or more of a maximum energy level, a minimum energy level, a median energy level, a mean energy level and a difference between the maximum and minimum energy levels that each of the first subset of sound components has across the series of time windows.

5. The method of claim 2, wherein the first subset of elements includes one or more of a maximum energy variation, a minimum energy variation, a median energy variation, a mean energy variation and a difference between the maximum and minimum energy variations that each of the first subset of sound components has across the series of time windows.

6. The method of claim 2, wherein the harmonic characteristics of the second subset of sound components are associated with a respective harmonic peak for each sound component at each of the series of time windows, and include one or more of an intensity value, a harmonic frequency and a variation of the harmonic frequency of the respective harmonic peak.

7. The method of claim 6, wherein the second subset of elements includes one or more of a maximum value, a minimum value, a median value, a mean value and a difference between the maximum and minimum values of each harmonic characteristic.

8. The method of claim 2, wherein statistically analyzing each of the plurality of sound components across the series of time windows includes:

for each sound component at each of the series of time windows, statistically analyzing the energy levels for the first subset of sound components to obtain a respective energy level.

9. The method of claim 1, wherein statistically analyzing each of the plurality of sound components across the series of time windows further includes:

for each sound component at each of the series of time windows;

identifying a respective harmonic peak;

obtaining the intensity and the frequency of the respective harmonic peak; and

obtaining the variation of the frequency of the respective harmonic peak with respect to that of another time window preceding to the respective time window.

10. The method of claim 1, wherein splitting the audio data to the plurality of sound components includes:

for each executive time window;

applying a Fast Fourier Transform (FFT) to obtain a plurality of FFT coefficients associated with the energy levels and the harmonic characteristics for the plurality of sound components each associated with the respective frequency or frequency band.

11. The method of claim 1, wherein the feature vector further includes a plurality of Cepstral coefficients obtained by a FFT.

12. The method of claim 1, wherein at least two of the time windows are consecutive time windows that partially overlap in time.

13. The method of claim 1, wherein each of the series of time windows lasts 30 msec.

14. The method of claim 1, wherein the plurality of sound components includes at least three sound components that are associated respectively with a low frequency band, an intermediate frequency band and a high frequency band.

15. The method of claim 1, wherein each of the plurality of sound components is associated with one or more of the following frequency bands:

0-900 Hz, 1000-5000 Hz and 6000 Hz and higher.

16. The method of claim 1, wherein the plurality of sound components includes at least one sound component that is associated with a frequency or frequency band related to a baby cry.

17. The method of claim 1, wherein the classifier model is selected from a group consisting of:

a neural network, a linear support vector machine (SVM), a naï

ve Bayes classifier, a Gaussian Mixture Model.

18. The method of claim 1, wherein the audio signal further includes an alternative series of time windows that are distinct from the series of time windows and is associated with at least one additional probability value indicating whether the audio signal includes the audio feature within the alternative series of time windows, wherein the signature event associated with the audio feature is detected based on both the probability value associated with the series of time windows and the at least one additional probability value.

19. The method of claim 18, wherein the signature event associated with the audio feature is detected when both the probability value associated with the series of time windows and the at least one additional probability value are larger than a predetermined probability threshold.

20. The method of claim 1, wherein the probability value that indicates whether the audio signal includes the audio feature has a magnitude between 0 and 1.

21. The method of claim 1, wherein the audio feature is associated with a baby sound, and the signature event is associated with an extended baby cry event.

22. A method for detecting a signature event associated with an audio feature, comprising:

on an electronic device having one or more processors and memory storing one or more programs for execution by the one or more processors, automatically and without user intervention;

obtaining audio data associated with an audio signal;

splitting the audio data to a plurality of sound components each associated with a respective frequency or frequency band and including a series of time windows;

statistically analyzing each of the plurality of sound components across the series of time windows;

extracting a feature vector from the plurality of sound components based on the statistical analysis, the feature vector including a first subset of elements associated with energy levels of a first subset of sound components, and a second subset of elements associated with harmonic characteristics of a second subset of sound components, wherein the first and second subsets of elements in the feature vector are arranged according a predetermined order;

classifying the extracted feature vector based on a classifier model provided by a remote server to obtain a probability value indicating whether the audio signal includes the audio feature within the series of time windows, wherein the classifier is configured to recognize the audio feature according to feature vectors that include elements arranged according to the predetermined order, wherein the classifier model is determined by the remote server based on a number of false positives generated by the classifier model and the predefined capabilities of the electronic device comprising one or more of computational capabilities, storage capabilities, and caching capabilities;

detecting the signature event associated with the audio feature based on the probability value and issuing an alert indicating occurrence of the signature event.

23. An electronic device for detecting a signature event associated with an audio feature, the electronic device comprising:

one or more processors; and

memory storing one or more programs to be executed by the one or more processors, the one or more programs comprising instructions for;

obtaining acoustic data associated with an audio signal;

splitting the audio data to a plurality of sound components each associated with a respective frequency or frequency band and including a series of time windows;

statistically analyzing each of the plurality of sound components across the series of consecutive time windows;

extracting a feature vector from the plurality of sound components based on the statistical analysis, the feature vector including a first subset of elements associated with energy levels of a first subset of sound components, and a second subset of elements associated with harmonic characteristics of a second subset of sound components, wherein the first and second subsets of elements in the feature vector are arranged according a predetermined order;

classifying the extracted feature vector based on a classifier model provided by a remote server to obtain a probability value indicating whether the audio signal includes the audio feature within the series of consecutive time windows, wherein the classifier is configured to recognize the audio feature according to feature vectors that include elements arranged according to the predetermined order, wherein the classifier model is determined by the remote server based on a number of false positives generated by the classifier model and the predefined capabilities of the electronic device comprising one or more of computational capabilities, storage capabilities, and caching capabilities;

detecting the signature event associated with the audio feature based on the probability value and issuing an alert indicating occurrence of the signature event.