Integration of hearing aids with smart glasses to improve intelligibility in noise

• US 9,264,824 B2
• Filed: 07/31/2013
• Issued: 02/16/2016
• Est. Priority Date: 07/31/2013
• Status: Active Grant

• Current Assignee: Starkey Laboratories Incorporated
• Original Assignee: Starkey Laboratories Incorporated
• Inventors: Pruthi, Tarun, Chabot, Xavier
• Primary Examiner(s): EASON, MATTHEW A
• Application Number:
  US13/956,092
• Publication Number:
  US 20150036856A1
• Time in Patent Office: 930 Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G10L 15/25 using position of the lips, movement of the lips or face analysis
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: G10L 21/0216 characterised by the method used for estimating noise
• CPC Class Codes: G10L 25/84 for discriminating voice from noise
• CPC Class Codes: H04R 2460/01 Hearing devices using active noise cancellation
• CPC Class Codes: H04R 25/00 Deaf-aid sets , i.e. electro-acoustic or electro-mechanical hearing aids; Electric tinnitus maskers providing an auditory perception electrical stimulation of auditory nerves to promote the auditory function A61N1/36038; optical stimulation of auditory nerves to promote the auditory function A61N5/0622
• CPC Class Codes: H04R 25/453 electronically
• CPC Class Codes: H04R 25/505 using digital signal processing

Abstract

Disclosed herein, among other things, are methods and apparatuses for integration of hearing aids with an MSD to improve intelligibility in noisy environments. One aspect of the present subject matter relates to a method of providing voice audio to a hearing aid or text in a hearing aid user'"'"'s field of view, where the voice audio or text corresponds to a speaker of interest within a noisy environment, and where the speaker of interest is identified using smart glasses. An MSD graphical user interface is provided for a hearing aid user to identify a speaker of interest. The lip movement patterns of the speaker of interest are recorded by the smart glasses, and voice activity detection is performed on the lip movement patterns. Noise reduction is performed using the results of the voice activity detection. The noise-reduced voice audio may be transmitted to the hearing aid, or the speech recognition text may be transmitted to and displayed on the MSD.

Claims

1. A method comprising:

providing a graphical user interface in a user field of vision for a user to identify and select a speaker of interest from a plurality of speakers;

recording a plurality of lip movement patterns of the speaker of interest;

generating voice activity detection data from the plurality of lip movement patterns;

recording voice audio data; and

generating noise-reduced voice audio data, wherein generating the noise-reduced voice audio data uses the voice activity detection data to isolate voice audio within the recorded voice audio data corresponding to the speaker of interest.

2. The method of claim 1, wherein recording a plurality of lip movement patterns of the speaker of interest includes steering a focus of a microphone array to a location corresponding to the speaker of interest.

3. The method of claim 1, further comprising:

transmitting the noise-reduced voice audio data to a hearing aid worn by the user.

4. The method of claim 3, wherein the noise-reduced voice audio data is transmitted continually to the hearing aid.

5. The method of claim 3, wherein transmitting the noise-reduced voice audio data to the hearing aid is initiated by the user.

6. The method of claim 5, wherein the noise-reduced voice audio data is stored in an audio data buffer, and wherein a predetermined duration of stored noise-reduced voice audio data is transmitted to the hearing aid worn by the user.

7. The method of claim 1, further comprising:

generating speech recognition text using the noise-reduced voice audio;

displaying the speech recognition text on the graphical user interface.

8. The method of claim 7, wherein the speech recognition text is displayed continually on the graphical user interface.

9. The method of claim 7, wherein displaying the speech recognition text on the graphical user interface is initiated by the user.

10. The method of claim 9, wherein the speech recognition text is stored in a text buffer, and wherein a predetermined amount of stored speech recognition text is displayed on the graphical user interface.

11. The method of claim 1, wherein providing a graphical user interface includes providing a user-worn personal computer.

12. The method of claim 1, wherein providing a graphical user interface includes providing a user-worn input device.

13. The method of claim 1, wherein providing a graphical user interface includes providing a gesture sensing input.

14. The method of claim 1, wherein providing a graphical user interface includes providing a speech recognition input.

15. The method of claim 1, wherein providing a graphical user interface includes providing for patient input to adjust the settings in real-time.

16. A system comprising:

a smart glasses graphical user interface configured to receive a user input to identify and select a speaker of interest from a plurality of speakers;

a video recording module configured to record a plurality of lip movement patterns of the speaker of interest;

a voice activity detection module configured to receive the plurality of lip movement patterns and generate voice activity detection data from the plurality of lip movement patterns;

a voice audio data recording module; and

a noise-reduced voice audio data generation module configured to receive the voice activity detection data and voice audio data, and configured to generate noise-reduced voice audio data corresponding to the speaker of interest.

17. The system of claim 16, further comprising:

a directional microphone array configured to focus audio detection in a direction corresponding to the speaker of interest.

18. The system of claim 16, further comprising:

a hearing assistance device configured to receive noise-reduced voice audio data and reproduce audible noise-reduced voice audio corresponding to the speaker of interest.

19. The system of claim 18, further comprising:

an audio data buffer configured to store a predetermined duration of voice audio data, wherein the smart glasses graphical user interface is further configured to receive a user input to cause the hearing assistance device to reproduce the predetermined duration of audible noise-reduced voice audio.

20. The system of claim 16, further comprising:

a speech recognition text generation module configured to receive the noise-reduced voice audio data and generate speech recognition text; and

a smart glasses graphical user display configured to receive and display speech recognition text corresponding to the speaker of interest.

21. The system of claim 17, further comprising:

a speech recognition text buffer configured to store a predetermined amount of speech recognition text, wherein the smart glasses graphical user interface is further configured to receive a user input to cause the smart glasses graphical user display configured to receive and display the predetermined amount of speech recognition text.

22. The system of claim 18, wherein the hearing assistance device includes a hearing aid.

23. The system of claim 18, wherein the hearing aid includes an in-the-ear (ITE) hearing aid.

24. The system of claim 18, wherein the hearing aid includes a behind-the-ear (BTE) hearing aid.

25. The system of claim 18, wherein the hearing aid includes an in-the-canal (ITC) hearing aid.

26. The system of claim 18, wherein the hearing aid includes a receiver-in-canal (RIC) hearing aid.

27. The system of claim 18, wherein the hearing aid includes a completely-in-the-canal (CIC) hearing aid.

28. The system of claim 18, wherein the hearing aid includes a receiver-in-the-ear (RITE) hearing aid.