Apparatus and method for wireless sound recognition to notify users of detected sounds

• US 9,870,719 B1
• Filed: 04/17/2017
• Issued: 01/16/2018
• Est. Priority Date: 04/17/2017
• Status: Expired due to Fees

• Current Assignee: Hz Innovations Inc.
• Original Assignee: Hz Innovations Inc.
• Inventors: Watkins, Greyson Kendall, Baltzer, Zachary, Lamb, Nicholas
• Primary Examiner(s): Ton, David
• Application Number:
  US15/489,195
• Time in Patent Office: 274 Days
• Field of Search: 381 56, 381315, 704231, 704232, 704236, 704202, 701 9
• CPC Class Codes: G09B 21/009 Teaching or communicating with deaf persons G09B21/04 takes precedence; deaf and dumb persons G09B21/00
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 21/0208 Noise filtering
• CPC Class Codes: G10L 25/21 the extracted parameters being power information
• CPC Class Codes: G10L 25/51 for comparison or discrimination
• CPC Class Codes: H04R 2225/43 Signal processing in hearing aids to enhance the speech intelligibility
• CPC Class Codes: H04R 25/554 using a wireless connection, e.g. between microphone and amplifier or using Tcoils
• CPC Class Codes: H04R 29/00 Monitoring arrangements; Testing arrangements for hearing aids H04R25/30; detection of loudspeaker connection H04R5/04; sound-field adaptation dependent on speaker detection H04S7/308
• CPC Class Codes: H04R 3/00 Circuits for transducers , loudspeakers or microphones
• CPC Class Codes: H04W 76/10 Connection setup
• CPC Class Codes: H04W 88/02 Terminal devices

Abstract

The described examples provide for a sound recognition and notification process and apparatus that provides a notification to a hearing impaired person via a portable device in response to the recognition of a selected sound. A processor of the sound notification apparatus is configured to compare the detected sound data to a model for recognition. If the sound data matches the model, features are extracted from the detected sound data and used by the processor to generate reference data, e.g. an updated sound model. The reference is retained in a memory of the apparatus to facilitate subsequent sound recognition without having to rely upon an external device for analysis. Recognition of a sound, based on a substantial match to stored reference data, may trigger transmission of a notification message indicating an occurrence of the corresponding sound.

Claims

1. An apparatus, comprising:

a housing;

a wireless transceiver configured to communicate via a first radio frequency communication protocol;

a sound detection circuit comprising a microphone and a converter coupled to the microphone, the sound detection circuit configured to;

detect sound in an area of a premises in which the apparatus is located, and

generate sound data based on the detected sound;

a memory storing programming instructions and one or more simplified sound models; and

a processor coupled to the sound detection circuit, the wireless transceiver and the memory, the processor when executing the programming instructions is configured to;

establish a communication path with a portable device via the wireless transceiver;

identify the portable device as at least one intended recipient device to receive notifications from the apparatus when the apparatus is in a sound recognition mode;

send, via the communication path, a request to the portable device requesting selection of an exigent sound of interest to a hearing-impaired person;

upon receiving from the portable device an exigent sound selection in response to the request, retrieve a pre-stored, simplified sound model corresponding to the exigent sound selection from the memory;

receive sound data generated by the sound detection circuit in response to sound detected in the area of the premises in which the apparatus is located;

compare the generated sound data received from the sound detection circuit to the retrieved, simplified sound model corresponding to the exigent sound selection;

in response to a result of the comparison indicating that the generated sound data is similar to the simplified sound model corresponding to the exigent sound selection, extract features from the generated sound data;

store the extracted features in the memory as reference sound data for the selected exigent sound; and

enter a sound recognition mode to monitor the area of the premises in which the apparatus is located to recognize an occurrence of an exigent sound that, based on the reference sound data, is similar to the selected exigent sound,

wherein the processor, the memory, the sound detection circuit and the wireless transceiver are coupled together within the housing.

2. The apparatus of claim 1, wherein the extracted features include at least one of an attack value of the generated sound data, a sustain value of the generated sound data, a decay value of the generated sound data, or a frequency track of the generated sound data.

3. The apparatus of claim 1, wherein when the apparatus is in the sound recognition mode:

the processor, upon execution of program instructions stored in the memory, is further configured to;

retrieve sound data generated by the sound detection circuit over a predetermined amount of time from a memory

sample the sound data retrieved from the memory;

determine whether the sampled sound data retrieved from memory exceeds a noise floor;

in response to the determination that the sampled sound data retrieved from memory exceeds the noise floor, extract features from the sampled sound data retrieved from memory;

determine a reference similarity value between the features extracted from the sampled retrieved sound data and the extracted features of the reference sound data, the reference similarity value indicating how similar the generated sound data is to the reference sound data for the selected exigent sound;

in response to the reference similarity value exceeding a predetermined threshold, output a notification to a device capable of receiving a notification via a network transceiver, the notification indicating an occurrence of the selected exigent sound.

4. The apparatus of claim 3, wherein when extracting features from the sampled retrieved sound data, the processor is further configured to:

identify features of the sampled retrieved sound related to at least one of the attack of the sound represented by the sampled sound data, the sustain of the sound represented by the sampled sound data, the decay of the sound represented by the sampled sound data, or tracking of the frequency of the sound represented by the sampled data.

5. The apparatus of claim 3, wherein prior to determining the reference similarity value between the features extracted from the sampled retrieved sound data and the extracted features of the reference sound data, the processor is further configured to:

test the validity of the extracted features of the sampled sound using a neural network; and

receive an indication of that the extracted features are valid for the sampled sound.

6. The apparatus of claim 1, wherein, after entering the sound recognition mode, the processor is further configured to:

calculate an average sound energy from sound collected during a preset window of time;

set a noise floor value based upon the calculated average sound energy, wherein the noise floor value is a minimum sound energy value that is equal to the predetermined sound energy;

sample additional sound data received from the sound detection circuit;

determine that the sampled additional sound data exceeds the noise floor value; and

in response to the sampled additional sound data exceeding the noise floor value, compare the sampled additional sound data to the reference sound data.

7. The apparatus of claim 1, wherein the first radio frequency protocol is a Bluetooth standard compatible protocol.

8. The apparatus of claim 1, wherein the network transceiver is a Zigbee transceiver, a Wi-Fi transceiver, or a cellular radio frequency transceiver.

9. The apparatus of claim 1, wherein the processor, upon execution of program instructions stored in the memory, is further configured to:

search an local area network using the network transceiver for a device capable of receiving notifications of a subsequent detection of the selected sound;

after locating a device capable of receiving notifications via the network transceiver, confirming via the communication path with the portable device that the located device is a permitted to receive notifications; and

upon receipt of the confirmation, close the communication path with the portable device.

10. The apparatus of claim 1, wherein the processor, upon execution of program instructions stored in the memory, is further configured to:

update the simplified model based on the stored reference data of the matching sound.

11. The apparatus of claim 10, wherein the processor, when updating the simplified model of the selected sound using the reference sound data, is further configured to:

incorporating the extracted features from the generated sound data of the detected exigent sound into the simplified model to produce a supplemented model, wherein the supplemented model further distinguishes the generated sound data as the selected sound; and

store the supplemented model in the memory.

12. The apparatus of claim 11, wherein when extracting features to update the simplified model, the processor is further configured to:

identify features of the sampled sound related to at least one of the attack of the sound represented by the sampled data, the sustain of the sound represented by the sampled data, the decay of the sound represented by the sampled data, or tracking of the frequency of the sound represented by the sampled data.

13. The apparatus of claim 1, further comprising:

a wall plug coupled to the housing and configured to be inserted into an electrical outlet from which the apparatus receives electrical power.

14. The apparatus of claim 1, further comprising:

a network transceiver, different from the wireless transceiver, coupled to the housing, and configured to couple to a data communication network, wherein;

the network transceiver transmits notifications formatted for non-audio output by the recipient device via a communication path with the data communication network, and

the network transceiver communicates via a second radio frequency communication protocol.

15. A method, comprising:

establishing a communication path between a portable device and a sound notification apparatus via a wireless transceiver of the sound notification apparatus;

in response to receiving by the sound notification apparatus a selection of an exigent sound, retrieve from a memory of the sound notification apparatus a pre-stored, simplified sound model corresponding to the selected exigent sound;

receiving sound data generated by a sound detection circuit of the sound notification apparatus in response to detecting a sound in an area of a premises where the sound notification apparatus is located;

comparing by a processor of the sound notification apparatus the generated sound data to the retrieved simplified model;

in response to a result of the comparison indicating that the generated sound data is similar to the simplified model, extracting features from the generated sound data;

storing the extracted features from the generated sound data as reference sound data for the selected exigent sound in the memory; and

entering a sound recognition mode to monitor the area of the premises to recognize a subsequent occurrence of an exigent sound that, based on the reference sound data, is similar to the selected exigent sound.

16. The method of claim 15, further comprising:

identifying the portable device as at least one recipient device to receive non-audio notifications from the sound notification apparatus, when the sound notification apparatus is in a sound recognition mode;

while the sound notification system is in the sound recognition mode;

retrieving sound data generated by the sound detection circuit from the memory;

sampling the retrieved sound data;

determining whether the sampled sound data exceeds a noise floor;

in response to the determination that the sampled sound exceeds the noise floor, extracting features from the sampled sound data;

determining based on the features extracted from the sampled sound data a similarity value indicating how similar the sampled sound data is to the reference sound data; and

in response to the similarity value exceeding a predetermined threshold, outputting a notification directed to the identified portable device.

17. The method of claim 15, further comprising:

upon establishing the communication path via the wireless transceiver, receiving information related to the area in which the sound notification apparatus is located.

18. The method of claim 15, wherein extracting features from the generated sound data, further comprises:

identifying features of the sampled sound related to at least one of an attack value of the generated sound data, a sustain of the generated sound data, a decay of the generated sound data, or a frequency track of the generated sound data.

19. The method of claim 15, further comprising:

prior to determining the reference similarity value between the features extracted from the sampled retrieved sound data and the extracted features of the reference sound data, determining that the extracted features are valid by testing the validity of the extracted features of the sampled sound using a neural network; and

receiving an indication of that the extracted features are valid for the sampled sound.

20. A sound notification apparatus, comprising:

a housing;

a wall plug coupled to the housing and configured to be inserted into an electrical outlet from which the apparatus receives electrical power;

a wireless transceiver configured to communicate via a first radio frequency communication protocol;

a sound detection circuit configured to;

detect a sound in an area in which the apparatus is receiving electrical power, and

generate sound data based on the detected sound;

a memory storing programming instructions and a set of simplified sound models; and

a processor coupled to the sound detection circuit, the transceiver and the memory within the housing, the processor when executing the programming instructions is configured to;

establish a communication path with a portable device via the wireless transceiver to conduct a sound notification apparatus setup;

request selection of a sound of interest to a hearing-impaired person for the sound notification apparatus setup;

in response to receiving a sound selection, retrieve a simplified sound model of the selected sound from the set of simplified sound models stored in the memory, the simplified sound model having additional sound-related information including an area of a premises in which the selected sound is most probable to occur;

sample sound data of a sound detected by the sound detection circuit that exceeds a noise floor; and

compare the sampled sound data to the retrieved simplified model of the selected sound;

in response to the comparison indicating a substantial match between the sampled sound data and the simplified model of the selected sound, extract features from the sampled sound that uniquely identify the sampled sound as the selected sound;

store the extracted features in the memory;

enter a sound recognition mode to notify an identified device of detection of the selected sound based upon data generated from a subsequent sound substantially matching the extracted features; and

in response to detection of data generated from the subsequent sound substantially matching extracted features while in the sound recognition mode, output a notification message indicating an occurrence of the selected sound for transmission to the identified device.

21. The apparatus of claim 20, further comprising:

a network transceiver coupled to a data communication network, wherein the transceiver communicates via a wired or wireless communication path with the data communication network, wherein the wireless communication path utilizes a second radio frequency communication protocol; and

the network transceiver is configured;

transmit the notification output by the processor to the identified device.

22. The apparatus of claim 20, wherein the processor, upon execution of program instructions stored in the memory, is further configured to:

search a local area network for a device different from the portable device that is capable of receiving notifications of subsequent detection of the selected sound;

after locating a device capable of receiving notifications, storing an identifier of located device as the identified device; and

close communication path with the portable device.

23. The apparatus of claim 20, wherein the processor, upon execution of program instructions stored in the memory, is further configured to:

extract features from the sampled sound data by identifying features of the sampled sound related to at least one of the attack of the sampled data, the sustain of the sampled data, the decay of the sampled data, or tracking of the frequency of the sampled data.