Filtering sounds for conferencing applications

• US 9,530,426 B1
• Filed: 06/24/2015
• Issued: 12/27/2016
• Est. Priority Date: 06/24/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wright, Shawn Crispin, Osborn, Dan, Thompson, Joe, Huai, Hongwu, Gouin, Forest Woodcroft, Saunders, Megan
• Primary Examiner(s): Sing, Simon
• Application Number:
  US14/749,446
• Publication Number:
  US 20160379660A1
• Time in Patent Office: 552 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 2021/02082 the noise being echo, reverberation of the speech
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: G10L 21/0216 characterised by the method used for estimating noise
• CPC Class Codes: G10L 21/0364 for improving intelligibility
• CPC Class Codes: H04M 2203/359 Augmented reality
• CPC Class Codes: H04M 2203/509 Microphone arrays
• CPC Class Codes: H04M 3/568 audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04R 2410/01 Noise reduction using microphones having different directional characteristics
• CPC Class Codes: H04R 2430/01 Aspects of volume control, not necessarily automatic, in sound systems
• CPC Class Codes: H04R 2430/20 Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic H04R2203/12 takes precedence
• CPC Class Codes: H04R 2430/21 Direction finding using differential microphone array [DMA]
• CPC Class Codes: H04R 29/005 Microphone arrays
• CPC Class Codes: H04R 29/008 Visual indication of individual signal levels visual indication of stereophonic sound image H04S7/40
• CPC Class Codes: H04S 1/002 Non-adaptive circuits, e.g. manually adjustable or static, for enhancing the sound image or the spatial distribution control circuits for electronic adaptation of the sound field H04S7/30
• CPC Class Codes: H04S 2400/11 : Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/13 : Aspects of volume control, not necessarily automatic, in stereophonic sound systems
• CPC Class Codes: H04S 7/30 : Control circuits for electronic adaptation of the sound field
• CPC Class Codes: H04S 7/302 : Electronic adaptation of stereophonic sound system to listener position or orientation H04S7/301 takes precedence

Abstract

A conferencing system includes a near-eye display device that displays video received from a remote communication device of a communication partner. An audio stream is transmitted to the remote communication device. The audio stream includes real-world sounds produced by one or more real-world audio sources captured by a spatially-diverse microphone array and virtual sounds produced by one or more virtual audio sources. A relative volume of background sounds in the audio stream is selectively reduced based, at least in part, on real-world positioning of corresponding audio sources, including real-world and/or virtualized audio sources.

Claims

1. A conferencing system, comprising:

a near-eye display device configured to display video received from a remote communication device of a communication partner;

a spatially-diverse microphone array including a plurality of microphones;

an audio communication interface configured to transmit, to the remote communication device of the communication partner, an audio stream including real-world sounds produced by one or more real-world audio sources captured by the spatially-diverse microphone array and virtual sounds produced by one or more virtual audio sources; and

an audio filter configured to selectively reduce a relative volume of background sounds in the audio stream based, at least in part, on real-world positioning of corresponding audio sources.

2. The conferencing system of claim 1, wherein the audio filter is configured to selectively reduce the relative volume of background virtual sounds in the audio stream based, at least in part, on virtualized real-world positioning of the one or more virtual audio sources.

3. The conferencing system of claim 2, wherein the virtualized real-world positioning of the one or more virtual audio sources are mapped to a physical space.

4. The conferencing system of claim 1, wherein the audio filter is configured to selectively reduce the relative volume of background virtual sounds in the audio stream based, at least in part, on an observed real-world positioning of the one or more real-world audio sources.

5. The conferencing system of claim 4, wherein the observed real-world positioning of the one or more real-world audio sources is identified using receive-beamforming of the real-world sounds captured by each of the plurality of microphones having spatially diverse positioning within a physical space.

6. The conferencing system of claim 1, wherein responsive to user selection of a first filter mode, a sound is included within the background sounds;

and

wherein responsive to user selection of a second filter mode, the sound is excluded from the background sounds.

7. The conferencing system of claim 1, wherein an audio source is individually selectable for volume reduction relative to other audio sources.

8. The conferencing system of claim 1, wherein a sound is included within the background sounds if a real-world positioning of a corresponding audio source is not within a three-dimensional audio-inclusion region;

and

wherein the sound is excluded from the background sounds if the real-world positioning of the corresponding audio source is within the three-dimensional audio-inclusion region.

9. The conferencing system of claim 8, wherein the three-dimensional audio-inclusion region is based on a field of view or gaze axis of the near-eye display device.

10. The conferencing system of claim 8, wherein the three-dimensional audio-inclusion region is user-defined.

11. The conferencing system of claim 1, wherein a sound is included within the background sounds if a real-world positioning of a corresponding audio source is within a three-dimensional audio-exclusion region;

and

wherein the sound is excluded from the background sounds if the real-world positioning of the corresponding audio source is not within the three-dimensional audio-exclusion region.

12. The conferencing system of claim 11, wherein the three-dimensional audio-exclusion region is based on a field of view or gaze axis of the near-eye display device.

13. The conferencing system of claim 11, wherein the three-dimensional audio-exclusion region is user-defined.

14. The conferencing system of claim 1, wherein the near-eye display device is configured to display a graphical indicator overlaying a real-world view of a physical space and identifying an audio source selected for volume reduction.

15. The conferencing system of claim 1, wherein the near-eye display device is configured to display a graphical indicator overlaying a real-world view of a physical space and identifying a real-world positioning of an audio source.

16. A conferencing method, comprising:

capturing, via a plurality of microphones of a spatially-diverse microphone array, real-world sounds produced by one or more real-world sources having real-world positioning within a physical environment;

combining the real-world sounds and virtual sounds into an audio stream, the virtual sounds produced by one or more virtual audio sources having virtualized real-world positioning within the physical environment;

displaying a virtual object via an augmented reality display device, the virtual object representing the virtual audio source at a corresponding virtualized real-world positioning;

selectively filtering the virtual sounds from the audio stream based on the virtualized real-world positioning of the one or more virtual audio sources; and

transmitting the audio stream to a remote communication device of a communication partner.

17. The conferencing method of claim 16, further comprising:

selectively filtering the real-world sounds from the audio stream based on observed real-world positioning of the one or more real-world audio sources.

18. The conferencing method of claim 17, further comprising:

receiving a user selection from the remote communication device over a communication network;

filtering from the audio stream, an audio source selected from among the plurality of real-world audio sources and the virtual audio source responsive to the user selection.

19. A conferencing system, comprising:

a spatially-diverse microphone array including a plurality of microphones;

an audio communication interface configured to transmit, to a remote communication device of a communication partner, an audio stream including real-world sounds produced by one or more real-world audio sources captured by the spatially-diverse microphone array and virtual sounds produced by one or more virtual audio sources having virtualized real-world positioning;

an audio filter configured to selectively amplify or dampen a relative volume of one or more of the real-world sounds and the virtual sounds within the audio stream; and

an augmented reality display device configured to display, within an augmented reality view of a physical space, video received from a remote communication device of the communication partner, and to display graphical indicators identifying observed real-world positioning of the one or more real-world audio sources and the virtualized real-world positioning of the one or more virtual audio sources.

20. The conferencing system of claim 19, wherein the relative volume of one or more of the real-world sounds and the virtual sounds within the audio stream is selectively amplified or dampened responsive to user input directed at the one or more graphical indicators.