Sound event detection

• US 9,805,739 B2
• Filed: 05/15/2015
• Issued: 10/31/2017
• Est. Priority Date: 05/15/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Nongpiur, Rajeev Conrad, Dixon, Michael
• Primary Examiner(s): Zhang, Leshui
• Application Number:
  US14/713,619
• Publication Number:
  US 20160335488A1
• Time in Patent Office: 900 Days
• Field of Search: 381 56, 381302, 381 86, 381 12, 381 14, 381 16, 381 26, 381 28, 381 57, 381 61, 381321, 381333, 381334, 381 91, 381 92, 381 93, 381 941- 948, 381 95, 381 97, 381 98, 381100, 381101, 381102, 381103, 381118, 381119, 381120, 381122, 381 731, 381 74, 704E17002, 704231, 704246, 704251, 704E17011, 704E15001, 704E15002, 704E15003, 704E15004, 704E15038, 704E15039, 704E1504, 704E15044, 704E15047, 700 94
• CPC Class Codes: G08B 13/1672 using sonic detecting means, e.g. a microphone operating in the audio frequency range
• CPC Class Codes: G10L 25/18 the extracted parameters being spectral information of each sub-band
• CPC Class Codes: G10L 25/51 for comparison or discrimination

Abstract

A system and method for the use of sensors and processors of existing, distributed systems, operating individually or in cooperation with other systems, networks or cloud-based services to enhance the detection and classification of sound events in an environment (e.g., a home), while having low computational complexity. The system and method provides functions where the most relevant features that help in discriminating sounds are extracted from an audio signal and then classified depending on whether the extracted features correspond to a sound event that should result in a communication to a user. Threshold values and other variables can be determined by training on audio signals of known sounds in defined environments, and implemented to distinguish human and pet sounds from other sounds, and compensate for variations in the magnitude of the audio signal, different sizes and reverberation characteristics of the environment, and variations in microphone responses.

Claims

1. An environmental data monitoring and reporting system, comprising:

a device sensor that detects sound in an area and generates an audio signal based on the detected sound;

a device processor communicatively coupled to the device sensor, wherein the device processor is configured to convert the audio signal received from the device sensor into low-resolution audio signal data comprising a plurality of low-resolution feature vectors representative of the detected sound, and to analyze the low-resolution audio signal data, at the device processor level, to identify the detected sound as one of either a sound related to area human or pet occupancy, or a sound generated by a source other than the area human or pet occupancy, and provide a communication regarding the detected area human or pet occupancy-related sound; and

a device communication interface communicatively coupled to the device processor, wherein the device communication interface is configured to send the communication regarding the detected area human or pet occupancy-related sound,

wherein the device sensor, device processor and device communication interface are integrated into a single premises management device, and

wherein the device processor is configured to;

implement a Fast Fourier Transform element to perform a frequency domain conversion of the audio signal;

implement a plurality of bandwidth filters, a plurality of median filters, a plurality of range filters, and a plurality of summers, to extract the low-resolution feature vectors that distinguish the detected sound;

implement a state classifier element to determine state transition conditions by comparing the low-resolution feature vectors to threshold values that distinguish sound categories and generate outputs indicating occurrences of the distinguished sound categories; and

implement a detector element to detect an occurrence of a sound category indicating the area human or pet occupancy and generate a user message in response.

2. The environmental data monitoring and reporting system of claim 1, further comprising the Fast Fourier Transform element, implemented by the device processor, to perform the frequency domain conversion of the audio signal, on a frame-by-frame basis.

3. The environmental data monitoring and reporting system of claim 1, further comprising:

the plurality of bandwidth filters, implemented by the device processor, to divide the bands of the frequency domain conversion;

the plurality of median filters, implemented by the device processor, to median filter the divided bands;

the plurality of range filters, implemented by the device processor, to filter a range of sample lengths; and

the plurality of summers, implemented by the device processor, to subtract a minimum sample range value from a maximum sample range value to calculate the plurality of low-resolution feature vectors that distinguish the detected sound, on a frame-by-frame basis.

4. The environmental data monitoring and reporting system of claim 1, further comprising:

the state classifier element, implemented by the device processor, to determine the state transition conditions by comparing the plurality of low-resolution feature vectors to threshold values and generate the outputs indicating the occurrences of distinguished sound categories, on a frame-by-frame basis.

5. The environmental data monitoring and reporting system of claim 4, wherein the device processor is configured to train on low-resolution audio signal data of known sound categories in defined areas to determine the threshold values that distinguish the sound categories and that compensate for low-resolution audio signal data, area and sensor variations.

6. The environmental data monitoring and reporting system of claim 1, further comprising:

the detector element, implemented by the device processor, to detect the occurrence of the sound category indicating the area human or pet occupancy; and

the device communication interface, implemented by the device processor, to communicate the user message in response to the detected occurrence of the sound category indicating the area human or pet occupancy.

7. The environmental data monitoring and reporting system of claim 6, wherein the detector element is configured to analyze each output indicating the occurrence of the sound category as received to detect an output denoting the occurrence of the sound category indicating the area human or pet occupancy.

8. The environmental data monitoring and reporting system of claim 6, wherein the detector element is configured to analyze a set of the outputs indicating the occurrences of the sound categories to detect a first output of the set denoting the occurrence of the sound category indicating the area human or pet occupancy.

9. The environmental data monitoring and reporting system of claim 6, wherein the detector element is configured to statistically analyze a set of the outputs indicating the occurrences of the distinguished sound categories to detect a likelihood of an occurrence of the sound category indicating the area human or pet occupancy.

10. An environmental data monitoring and reporting system, comprising:

a device sensor, comprising a microphone, that detects a condition comprising one or more sounds in an area and generates an audio signal based on the detected condition;

a device processor communicatively coupled to the device sensor, wherein the device processor is configured to receive the audio signal and convert the audio signal received from the device sensor into low-resolution signal data comprising a plurality of low-resolution feature vectors representative of the one or more sounds in the area and to analyze the low-resolution signal data, at the device processor level, by;

implementing a Fast Fourier Transform element, a plurality of bandwidth filters, a plurality of median filters, a plurality of range filters, and a plurality of summers, to perform a frequency domain conversion of the audio signal and extract the low-resolution feature vectors that distinguish detected conditions,

implementing a state classifier element to compare the low-resolution feature vectors to threshold values that distinguish condition categories,

generating outputs indicating occurrences of the distinguished condition categories, and

implementing a detector element to detect one of the distinguished condition categories, which represents one of either a sound related to an area human or pet occupancy, or a sound generated by a source other than the area human or pet occupancy, and generate a user message in response; and

a device communication interface communicatively coupled to the device processor, wherein the device communication interface is configured to send the user message regarding the detected area human or pet occupancy-related condition,

wherein the device sensor, device processor and device communication interface are integrated into a single premises management device.

11. The environmental data monitoring and reporting system of claim 10, further comprising:

the Fast Fourier Transform element, implemented by the device processor, to perform the frequency domain conversion of the audio signal;

the plurality of bandwidth filters, implemented by the device processor, to divide the bands of the frequency domain conversion;

the plurality of median filters, implemented by the device processor, to median filter the divided bands;

the plurality of range filters, implemented by the device processor, to filter a range of sample lengths; and

the plurality of summers, implemented by the device processor, to subtract a minimum sample range value from a maximum sample range value to calculate the plurality of low-resolution feature vectors that distinguish the detected conditions.

12. The environmental data monitoring and reporting system of claim 10, further comprising:

the state classifier element, implemented by the device processor, to compare the plurality of low-resolution feature vectors to the threshold values and generate the outputs indicating the occurrences of the distinguished condition categories.

13. The environmental data monitoring and reporting system of claim 12, wherein the device processor is configured to train on low-resolution signal data of known condition categories in defined areas to determine the threshold values that distinguish the condition categories and that compensate for signal data, area and sensor variations.

14. The environmental data monitoring and reporting system of claim 10, further comprising:

the detector element, implemented by the device processor, to detect the occurrence of the condition category indicating the area human or pet occupancy; and

the device communication interface, implemented by the device processor, to communicate the user message in response to the detected occurrence of the condition category indicating the area human or pet occupancy.

15. A method for controlling an environmental data monitoring and reporting system, comprising:

detecting sound in an area and generating an audio signal based on the detected sound;

converting the audio signal into low-resolution audio signal data comprising a plurality of low-resolution feature vectors representative of the sound in the area, and analyzing the low-resolution audio signal data, at a device processor level, to identify the detected sound as one of either a sound related to area human or pet occupancy, or a sound generated by a source other than the area human or pet occupancy, and provide a communication regarding the detected area human or pet occupancy-related sound; and

sending the communication regarding the detected area human or pet occupancy-related sound,

wherein the detecting step, converting-step, analyzing step and sending are performed by a single premises management device,

wherein the converting comprises performing a frequency domain conversion of the audio signal using a Fast Fourier Transform and extracting the low-resolution feature vectors that distinguish detected sounds, where the extracting is performed using a plurality of bandwidth filters, a plurality of median filters, a plurality of range filters, and a plurality of summers, to extract the low-resolution feature vectors, and

the analyzing step comprises determining state transition conditions by comparing the low-resolution feature vectors to threshold values that distinguish sound categories and generating outputs indicating occurrences of the distinguished sound categories.

16. The method of claim 15, wherein the analyzing step further comprises detecting the occurrence of the sound category indicating an area human or pet occupancy and generating a user message in response.

17. The method of claim 15, further comprising training on low-resolution audio signal data of known sound categories in defined areas to determine the threshold values that distinguish the sound categories and that compensate for audio signal, area and sensor variations.