Technique for directing audio in augmented reality system

• US 10,536,783 B2
• Filed: 02/02/2017
• Issued: 01/14/2020
• Est. Priority Date: 02/04/2016
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Sanger, George Alistair, Miller, Samuel A., Schmidt, Brian, Tajik, Anastasia Andreyevna
• Primary Examiner(s): Ojo, Oyesola C
• Application Number:
  US15/423,415
• Publication Number:
  US 20170230760A1
• Time in Patent Office: 1,076 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04R 1/028 associated with devices performing functions other than acoustics, e.g. electric candles
• CPC Class Codes: H04R 1/1016 Earpieces of the intra-aural type
• CPC Class Codes: H04R 1/1041 Mechanical or electronic switches, or control elements switches in general H01H
• CPC Class Codes: H04R 2205/041 Adaptation of stereophonic signal reproduction for the hearing impaired
• CPC Class Codes: H04R 2499/11 Transducers incorporated or for use in hand-held devices, e.g. mobile phones, PDA's, camera's
• CPC Class Codes: H04R 2499/15 Transducers incorporated in visual displaying devices, e.g. televisions, computer displays, laptops
• CPC Class Codes: H04R 25/40 Arrangements for obtaining a desired directivity characteristic
• CPC Class Codes: H04R 25/407 Circuits for combining signals of a plurality of transducers
• CPC Class Codes: H04R 25/505 using digital signal processing
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04R 5/033 Headphones for stereophonic communication details thereof, e.g. relating to batteries, cables or control elements H04R1/10
• CPC Class Codes: H04S 7/304 For headphones

Abstract

A hearing aid comprises a sensor configured for detecting a focus of an end user on a real sound source, a microphone assembly configured for converting sounds into electrical signals, a speaker configured for converting the electrical signals into sounds, and a control subsystem configured for modifying the direction and/or distance of a greatest sensitivity of the microphone assembly based on detected focus. A virtual image generation system comprises memory storing a three-dimensional scene, a sensor configured for detecting a focus of the end user on a sound source, a speaker configured for conveying sounds to the end user, and a control subsystem configured for causing the speaker to preferentially convey a sound originating from the sound source in response to detection of the focus, and for rendering image frames of the scene, and a display subsystem configured for sequentially displaying the image frames to the end user.

Claims

1. A hearing aid for use by an end user, comprising:

at least one sensor configured for detecting a focus of the end user on a real sound source;

an adaptive microphone assembly configured for converting sounds into electrical signals;

at least one speaker configured for converting the electrical signals to sounds for perception by the end user; and

a control subsystem configured for modifying, based on

the detected focus of the end user,

a direction of a greatest sensitivity of the adaptive microphone assembly, and

a distance of the greatest sensitivity of the adaptive microphone assembly,

wherein the control subsystem is configured for processing the electrical signals to compare characteristics of a first sound originating from the real sound source to characteristics of a second sound originating from a different source, emphasizing sounds having the same type of characteristics as the characteristics of the first sound, and deemphasizing sounds having the same type of characteristics as the second sound.

2. The hearing aid of claim 1, wherein the adaptive microphone assembly is configured for converting the sounds into the electrical signals in accordance with one of a dipole, cardioid, super-cardioid, or hyper-cardioid pattern.

3. The hearing aid of claim 1, wherein the control subsystem is configured for setting the greatest sensitivity of the adaptive microphone assembly to a direction and a distance of the detected focus of the end user.

4. The hearing aid of claim 1, wherein the at least one sensor comprises a camera.

5. The hearing aid of claim 1, wherein the at least one sensor is configured for detecting an angular position of an eye of the end user to detect a direction of the focus of the end user.

6. The hearing aid of claim 1, wherein the at least one sensor is configured for detecting a convergence of eyes of the end user to detect a distance of the focus of the end user.

7. The hearing aid of claim 1, wherein the at least one sensor is configured for detecting an angular position of a head of the end user to detect the direction of the focus of the end user.

8. The hearing aid of claim 1, wherein the adaptive microphone assembly comprises a phased array of microphone elements.

9. The hearing aid of claim 1, further comprising a frame structure configured for being worn by the end user, the frame structure carrying the at least one sensor, the microphone assembly, and the at least one speaker.

10. The hearing aid of claim 1, wherein the control subsystem comprises a digital signal processor (DSP) configured for modifying the direction and/or distance of the greatest sensitivity of the adaptive microphone assembly.

11. The hearing aid of claim 1, wherein the control subsystem is configured for processing the electrical signals to localize the sounds, such that a sound originating from the real sound source is perceived by the end user as originating from the real sound source.

12. The hearing aid of claim 1, wherein the control subsystem is configured for processing the electrical signals to selectively raising or lowering a volume of certain frequency components of the sounds, such that a sound originating from the real sound source is emphasized and/or a sound originating from a different sound source is deemphasized.

13. A virtual image generation system for use by an end user, comprising:

memory storing a three-demensional scene;

the hearing aid of claim 1, wherein the control subsystem is further configured for rendering a plurality of synthetic image frames of the three-dimensional scene; and

a display subsystem configured for sequentially displaying the plurality of image frames to the end user.

14. The virtual image generation system of claim 13, wherein the display subsystem is configured to being positioned in front of the eyes of the end user.

15. The vitual image generation system of claim 13, wherein the display subsystem includes a projection subsystem and a partially transparent display surface, the projection subsystem configured for projecting the image frames on the partially transparent display surface, and the partially transparent display surface is configured for being positioned in the field of view between the eyes of the end user and an ambient environment.

16. The virtual image generation system of claim 13, further comprising a frame structure configured for being worn by the end user, the frame structure carrying the at least one sensor, the adaptive microphone assembly, the at least one speaker, and the display subsystem.

17. The virtual image generation system of claim 13, wherein the control subsystem comprises a graphics control subsystem unit (GPU).

18. The hearing aid of claim 1, wherein the type of characteristics of the first sound is music.