Method and device for adjusting sound levels of sources based on sound source priority

• US 9,384,737 B2
• Filed: 06/29/2012
• Issued: 07/05/2016
• Est. Priority Date: 06/29/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Lamb, Mathew J., Sugden, Ben J., Crocco, Robert L. Jr., Keane, Brian E., Miles, Christopher E., Perez, Kathryn Stone, Massey, Laura K., Kipman, Alex Aben-Athar
• Primary Examiner(s): Lerner, Martin
• Application Number:
  US13/538,731
• Publication Number:
  US 20140006026A1
• Time in Patent Office: 1,467 Days
• Field of Search: 704/225, 704/226, 704/227, 704/233, 704/246, 381/57, 381/92, 381/313
• US Class Current: 1/1
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: H03G 3/10 having semiconductor devices

Abstract

A system for generating one or more enhanced audio signals such that one or more sound levels corresponding with sounds received from one or more sources of sound within an environment may be dynamically adjusted based on contextual information is described. The one or more enhanced audio signals may be generated by a head-mounted display device (HMD) worn by an end user within the environment and outputted to earphones associated with the HMD such that the end user may listen to the one or more enhanced audio signals in real-time. In some cases, each of the one or more sources of sound may correspond with a priority level. The priority level may be dynamically assigned depending on whether the end user of the HMD is focusing on a particular source of sound or has specified a predetermined level of importance corresponding with the particular source of sound.

Claims

1. A method for adjusting sound levels based on contextual information, comprising:

capturing at a mobile device one or more sounds from an environment using a microphone array, the mobile device comprises a head-mounted display device;

identifying one or more sources of sound within the environment based on the one or more sounds captured using the microphone array, the one or more sources of sound include a first sound source and a second sound source different from the first sound source, the first sound source comprises a particular person, the second sound source comprises a second person different from the particular person;

determining a location of the mobile device, the location of the mobile device is determined using GPS location information corresponding with the location of the mobile device;

determining using one or more processors an identification of the particular person;

assigning a first priority level to the first sound source based on the location of the mobile device and the identification of the particular person, the assigning the first priority level to the first sound source includes determining whether an end user of the mobile device has gazed at the particular person for at least a particular period of time;

assigning a second priority level to the second sound source based on the location of the mobile device;

determining one or more weighting coefficients based on the first priority level and the second priority level, the determining one or more weighting coefficients is performed by the mobile device;

generating using the one or more processors one or more enhanced audio signals based on the one or more weighting coefficients and the one or more sounds captured using the microphone array, the generating the one or more enhanced audio signals is performed by the mobile device; and

outputting the one or more enhanced audio signals from the mobile device.

2. The method of claim 1, wherein:

the assigning the first priority level to the first sound source includes determining whether the end user of the mobile device is gazing at the particular person.

3. The method of claim 1, wherein:

the assigning the first priority level to the first sound source includes acquiring a user profile associated with the particular person in response to determining the identification of the particular person and assigning the first priority level to the first sound source based on a priority level specified by the user profile.

4. The method of claim 1, wherein:

the identifying one or more sources of sound includes identifying the first sound source using voice recognition techniques.

5. The method of claim 1, further comprising:

determining a first direction associated with the first sound source; and

determining a second direction associated with the second sound source, the generating the one or more enhanced audio signals includes generating a first new audio signal based on the first direction using beamforming techniques and generating a second new audio signal based on the second direction using beamforming techniques, the generating the one or more enhanced audio signals includes generating a combined audio signal based on the first new audio signal and the second new audio signal.

6. The method of claim 1, wherein:

the determining one or more weighting coefficients includes determining the one or more weighting coefficients such that a minimum volume ratio is maintained.

7. An electronic device for adjusting sound levels based on contextual information, comprising:

a plurality of microphones configured to capture one or more sounds from an environment;

one or more processors in communication with the plurality of microphones, the one or more processors configured to identify one or more sources of sound within the environment based on the one or more sounds captured using the plurality of microphones, the one or more sources of sound include a first sound source and a second sound source different from the first sound source, the first sound source comprises a particular person, the one or more processors configured to determine a location of the electronic device and determine an identification of the particular person, the one or more processors configured to assign a first priority level to the first sound source based on the location of the electronic device and the identification of the particular person and assign a second priority level to the second sound source based on the location of the electronic device, the one or more processors configured to assign the first priority level to the first sound source based on whether an end user of the electronic device has gazed at the particular person for at least a particular period of time, the one or more processors configured to determine one or more weighting coefficients based on the first priority level and the second priority level, the one or more processors configured to generate one or more enhanced audio signals based on the one or more weighting coefficients and the one or more sounds captured using the plurality of microphones; and

an audio output device in communication with the one or more processors, the audio output device configured to output the one or more enhanced audio signals, the electronic device comprises a head-mounted display device.

8. The electronic device of claim 7, wherein:

the one or more processors configured to assign the first priority level to the first sound source based on whether the end user of the electronic device is gazing at the particular person.

9. The electronic device of claim 7, wherein:

the audio output device comprises earphones; and

the one or more processors configured to identify the first sound source using voice recognition techniques.

10. An apparatus, comprising:

a microphone array configured to capture one or more sounds from an environment;

one or more processors configured to identify one or more sources of sound within the environment based on the one or more sounds captured using the microphone array, the one or more sources of sound include a first sound source and a second sound source different from the first sound source, the first sound source comprises a particular person, the one or more processors configured to determine a location of the apparatus and determine an identification of the particular person, the one or more processors configured to assign a first priority level to the first sound source based on the location of the apparatus and the identification of the particular person, the one or more processors configured to assign the first priority level to the first sound source in response to detecting that an end user of the apparatus has gazed at the particular person for at least a particular period of time, the one or more processors configured to assign a second priority level to the second sound source based on the location of the apparatus, the one or more processors configured to determine one or more weighting coefficients based on the first priority level and the second priority level, the one or more processors configured to generate one or more enhanced audio signals based on the one or more weighting coefficients and the one or more sounds captured using the microphone array; and

an audio output device configured to output the one or more enhanced audio signals, the apparatus comprises a see-through head-mounted display device.

11. The apparatus of claim 10, wherein:

the location of the apparatus is determined using GPS location information corresponding with the location of the apparatus.

12. The apparatus of claim 10, wherein:

the apparatus comprises a mobile electronic device.

13. The apparatus of claim 10, wherein:

the second sound source comprises a second person different from the particular person.

14. The apparatus of claim 10, wherein:

the microphone array comprises a plurality of microphones.

15. The apparatus of claim 10, wherein:

the one or more processors configured to determine a first direction associated with the first sound source and determine a second direction associated with the second sound source, the one or more processors configured to generate a first new audio signal based on the first direction using beamforming techniques and generate a second new audio signal based on the second direction using beamforming techniques, the one or more processors configured to generate a combined audio signal based on the first new audio signal and the second new audio signal.