Apparatus and method for wireless sound recognition to notify users of detected sounds

• US 10,062,304 B1
• Filed: 12/08/2017
• Issued: 08/28/2018
• Est. Priority Date: 04/17/2017
• Status: Active Grant

• Current Assignee: Hz Innovations Inc.
• Original Assignee: Hz Innovations Inc.
• Inventors: Watkins, Greyson Kendall, Baltzer, Zachary, Lamb, Nicholas
• Primary Examiner(s): Ton, David
• Application Number:
  US15/836,176
• Time in Patent Office: 263 Days
• Field of Search: 381 56, 381315, 704231, 704232, 704236, 704202, 701 9
• CPC Class Codes: G09B 21/009 Teaching or communicating with deaf persons G09B21/04 takes precedence; deaf and dumb persons G09B21/00
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 21/0208 Noise filtering
• CPC Class Codes: G10L 25/21 the extracted parameters being power information
• CPC Class Codes: G10L 25/51 for comparison or discrimination
• CPC Class Codes: H04R 2225/43 Signal processing in hearing aids to enhance the speech intelligibility
• CPC Class Codes: H04R 25/554 using a wireless connection, e.g. between microphone and amplifier or using Tcoils
• CPC Class Codes: H04R 29/00 Monitoring arrangements; Testing arrangements for hearing aids H04R25/30; detection of loudspeaker connection H04R5/04; sound-field adaptation dependent on speaker detection H04S7/308
• CPC Class Codes: H04R 3/00 Circuits for transducers , loudspeakers or microphones
• CPC Class Codes: H04W 76/10 Connection setup
• CPC Class Codes: H04W 88/02 Terminal devices

Abstract

The described examples provide for a sound recognition and notification process and apparatus that provides a notification to a hearing impaired person via a portable device in response to the recognition of a selected sound. A processor of the sound notification apparatus is configured to compare the detected sound data to a model for recognition. If the sound data matches the model, features are extracted from the detected sound data and used by the processor to generate reference data, e.g. an updated sound model. The reference is retained in a memory of the apparatus to facilitate subsequent sound recognition without having to rely upon an external device for analysis. Recognition of a sound, based on a substantial match to stored reference data, may trigger transmission of a notification message indicating an occurrence of the corresponding sound.

Claims

1. An apparatus, comprising:

a wireless transceiver;

a sound detection circuit configured to detect sound and generate sound data based on detected sound;

memory configured to store one or more sound models; and

a processor, coupled to the memory, configured to;

identify a device to receive notifications from the apparatus;

send, via the wireless transceiver, a request to the device for selection of a sound;

identify a sound model corresponding to a received sound selection;

receive sound data generated by the sound detection circuit in response to detected sound;

extract features from the generated sound data in response to a comparison of the generated sound data to the sound model corresponding to the sound selection, based on the comparison producing a first similarity value that meets or exceeds a first similarity threshold;

store the extracted features in the memory as reference sound data for the selected sound; and

enter a sound recognition mode to recognize a sound based on the reference sound data having a second similarity value that meets or exceeds a second similarity threshold with respect to the selected sound,

wherein the processor, the sound detection circuit and the wireless transceiver are within a housing.

2. The apparatus of claim 1, wherein the extracted features include at least one of an attack value of the generated sound data, a sustain value of the generated sound data, a decay value of the generated sound data, or a frequency track of the generated sound data.

3. The apparatus of claim 1, wherein when the apparatus is in the sound recognition mode:

the processor is further configured to;

retrieve sound data generated by the sound detection circuit over a predetermined amount of time from a memory

sample the sound data retrieved from the memory;

determine whether the sampled sound data retrieved from memory exceeds a noise floor;

extract features from the sampled sound data retrieved from memory;

in response to the second similarity value exceeding the second similarity threshold, wherein the second similarity threshold is a predetermined threshold, output a notification to a device capable of receiving a notification via a network transceiver, the notification indicating an occurrence of the selected sound.

4. The apparatus of claim 3, wherein when extracting features from the sampled retrieved sound data, the processor is further configured to:

identify features of the sampled retrieved sound related to at least one of the attack of the sound represented by the sampled sound data, the sustain of the sound represented by the sampled sound data, the decay of the sound represented by the sampled sound data, or tracking of the frequency of the sound represented by the sampled data.

5. The apparatus of claim 3, wherein prior to determining the second similarity value between the features extracted from the sampled retrieved sound data and the extracted features of the reference sound data, the processor is further configured to:

test the validity of the extracted features of the sampled sound using a neural network; and

receive an indication of that the extracted features are valid for the sampled sound.

6. The apparatus of claim 1, wherein, after entering the sound recognition mode, the processor is further configured to:

calculate an average sound energy from sound collected during a preset window of time;

set a noise floor value based upon the calculated average sound energy, wherein the noise floor value is a minimum sound energy value equal to the predetermined sound energy;

sample additional sound data received from the sound detection circuit;

determine that the sampled additional sound data exceeds the noise floor value; and

compare the sampled additional sound data to the reference sound data.

7. The apparatus of claim 1, wherein the first radio frequency protocol is a Bluetooth standard compatible protocol.

8. The apparatus of claim 1, wherein the processor is further configured to establish a communication path with a portable device via the wireless transceiver.

9. The apparatus of claim 8, wherein the processor is further configured to:

search an local area network using the network transceiver for a device capable of receiving notifications of a subsequent detection of the selected sound;

after locating a device capable of receiving notifications via the network transceiver, confirming via the communication path with the portable device that the located device is a permitted to receive notifications; and

upon receipt of the confirmation, close the communication path with the portable device.

10. The apparatus of claim 1, wherein the processor is further configured to update the model based on the stored reference data of the matching sound.

11. The apparatus of claim 10, wherein the processor, when updating the model of the selected sound using the reference sound data, is further configured to:

incorporating the extracted features from the generated sound data of the detected sound into the model to produce a supplemented model, wherein the supplemented model further distinguishes the generated sound data as the selected sound; and

store the supplemented model in the memory.

12. The apparatus of claim 11, extracting features to update the model, the processor is further configured to:

identify features of the sampled sound related to at least one of the attack of the sound represented by the sampled data, the sustain of the sound represented by the sampled data, the decay of the sound represented by the sampled data, or tracking of the frequency of the sound represented by the sampled data.

13. The apparatus of claim 1, further comprising:

a wall plug coupled to the housing and configured to be inserted into an electrical outlet from which the apparatus receives electrical power.

14. The apparatus of claim 1, further comprising:

a network transceiver, different from the wireless transceiver, coupled to the housing, and configured to couple to a data communication network, wherein;

the network transceiver transmits notifications formatted for non-audio output by the recipient device via a communication path with the data communication network, and

the network transceiver communicates via a second radio frequency communication protocol.

15. A method, comprising:

retrieving from memory of a sound notification apparatus a pre-stored sound model corresponding to a selected sound;

receiving sound data generated by a sound detection circuit of the sound notification apparatus in response to detecting a sound in an area where the sound notification apparatus is located;

extracting features from the generated sound data in response to a comparison of the pre-stored sound model to the generated sound data, based on the comparison producing a first similarity value that meets or exceeds a first similarity threshold;

storing the extracted features from the generated sound data as reference sound data for the selected sound in the memory; and

entering a sound recognition mode to monitor the area to recognize a subsequent occurrence of a sound, based on the reference sound data having a second similarity value with respect to the selected sound that meets or exceeds a second similarity threshold with respect.

16. The method of claim 15, further comprising:

identifying the portable device as at least one recipient device to receive non-audio notifications from the sound notification apparatus, when the sound notification apparatus is in a sound recognition mode;

while the sound notification system is in the sound recognition mode;

retrieving sound data generated by the sound detection circuit from the memory;

sampling the retrieved sound data;

determining whether the sampled sound data exceeds a noise floor;

in response to the determination that the sampled sound exceeds the noise floor, extracting features from the sampled sound data; and

in response to the similarity value exceeding the second similarity threshold, wherein the second similarity threshold is a predetermined threshold, outputting a notification directed to the identified portable device.

17. The method of claim 15, further comprising:

establishing a communication path between a portable device and a sound notification apparatus via a wireless transceiver of the sound notification apparatus; and

upon establishing the communication path via the wireless transceiver, receiving information related to the area in which the sound notification apparatus is located.

18. The method of claim 15, wherein extracting features from the generated sound data, further comprises:

identifying features of the sampled sound related to at least one of an attack value of the generated sound data, a sustain of the generated sound data, a decay of the generated sound data, or a frequency track of the generated sound data.

19. The method of claim 15, further comprising:

prior to determining the second similarity value, determining that the extracted features are valid by testing the validity of the extracted features of the sampled sound using a neural network; and

receiving an indication of that the extracted features are valid for the sampled sound.

20. A sound notification apparatus, comprising:

a wireless transceiver configured to communicate via a first wireless communication protocol;

a sound detection circuit;

a memory storing programming instructions and a set of sound models; and

a processor coupled to the memory, the processor configured to;

request selection of a sound of interest to a hearing-impaired person for a sound notification apparatus setup;

retrieve a sound model of the selected sound from the set of sound models stored in the memory, the sound model having additional sound-related information including an area in which the selected sound is most probable to occur;

sample sound data of a sound detected by the sound detection circuit that exceeds a noise floor; and

compare the sampled sound data to the retrieved model of the selected sound;

extract features from the sampled sound that uniquely identify the sampled sound as the selected sound;

store the extracted features in the memory;

enter a sound recognition mode to notify an identified device of detection of the selected sound based upon data generated from a subsequent sound substantially matching the extracted features; and

output a notification message indicating an occurrence of the selected sound for transmission to the identified device.

21. The apparatus of claim 20, further comprising:

a network transceiver coupled to a data communication network, wherein the transceiver communicates via a wired or wireless communication path with the data communication network, wherein the wireless communication path utilizes a second radio frequency communication protocol; and

the network transceiver is configured;

transmit the notification output by the processor to the identified device.

22. The apparatus of claim 20, wherein the processor is further configured to:

establish a communication path with a portable device via the wireless transceiver to conduct the sound notification apparatus setup;

search a local area network for a device different from the portable device that is capable of receiving notifications of subsequent detection of the selected sound;

after locating a device capable of receiving notifications, storing an identifier of located device as the identified device; and

close the communication path with the portable device.

23. The apparatus of claim 20, wherein the processor is further configured to:

extract features from the sampled sound data by identifying features of the sampled sound related to at least one of the attack of the sampled data, the sustain of the sampled data, the decay of the sampled data, or tracking of the frequency of the sampled data.