Filtering sounds for conferencing applications

• US 10,127,917 B2
• Filed: 12/16/2016
• Issued: 11/13/2018
• Est. Priority Date: 06/24/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wright, Shawn Crispin, Osborn, Dan, Thompson, Joe, Huai, Hongwu, Gouin, Forest Woodcroft, Saunders, Megan
• Primary Examiner(s): Gauthier, Gerald
• Application Number:
  US15/382,313
• Publication Number:
  US 20170098453A1
• Time in Patent Office: 697 Days
• Field of Search: 345 8, 345156, 348 1401, 348 1403, 348 1407, 348 1408, 34821112, 381 17, 381 57, 381 74, 381 92, 381303, 381 22, 381 66, 381 86, 381 87, 381 97, 704201, 704257, 704226, 704233, 370260, 37942002, 386241, 455570
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 2021/02082 the noise being echo, reverberation of the speech
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: G10L 21/0216 characterised by the method used for estimating noise
• CPC Class Codes: G10L 21/0364 for improving intelligibility
• CPC Class Codes: H04M 2203/359 Augmented reality
• CPC Class Codes: H04M 2203/509 Microphone arrays
• CPC Class Codes: H04M 3/568 audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04R 2410/01 Noise reduction using microphones having different directional characteristics
• CPC Class Codes: H04R 2430/01 Aspects of volume control, not necessarily automatic, in sound systems
• CPC Class Codes: H04R 2430/20 Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic H04R2203/12 takes precedence
• CPC Class Codes: H04R 2430/21 Direction finding using differential microphone array [DMA]
• CPC Class Codes: H04R 29/005 Microphone arrays
• CPC Class Codes: H04R 29/008 Visual indication of individual signal levels visual indication of stereophonic sound image H04S7/40
• CPC Class Codes: H04S 1/002 Non-adaptive circuits, e.g. manually adjustable or static, for enhancing the sound image or the spatial distribution control circuits for electronic adaptation of the sound field H04S7/30
• CPC Class Codes: H04S 2400/11 : Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/13 : Aspects of volume control, not necessarily automatic, in stereophonic sound systems
• CPC Class Codes: H04S 7/30 : Control circuits for electronic adaptation of the sound field
• CPC Class Codes: H04S 7/302 : Electronic adaptation of stereophonic sound system to listener position or orientation H04S7/301 takes precedence
• CPC Class Codes: H05K 999/99 : dummy group

Abstract

A conferencing system includes a display device that displays video received from a remote communication device of a communication partner. An audio stream is transmitted to the remote communication device. The audio stream includes real-world sounds produced by one or more real-world audio sources captured by a microphone array and virtual sounds produced by one or more virtual audio sources. A relative volume of sounds in the audio stream is selectively adjusted based, at least in part, on real-world positioning of corresponding audio sources, including real-world and/or virtualized audio sources.

Claims

1. A conferencing system, comprising:

a display device configured to display video received from a remote communication device of a communication partner;

a microphone array including one or more microphones;

an audio communication interface configured to transmit, to the remote communication device of the communication partner, an audio stream including real-world sounds produced by one or more real-world audio sources captured by the microphone array and virtual sounds produced by one or more virtual audio sources not actually present in a physical space of the one or more real-world audio sources, wherein a virtual audio source of the one or more virtual audio sources is mapped to a virtualized real-world positioning within the physical space; and

an audio filter configured to selectively reduce a relative volume of a background virtual sound in the audio stream based, at least in part, on one or more parameters of the virtual audio source producing the background virtual sound.

2. The conferencing system of claim 1, further comprising a user input device configured to recognize user input for toggling one or more aspects of volume reduction via the audio filter.

3. The conferencing system of claim 1, wherein the audio filter is configured to selectively reduce the relative volume of the background virtual sound in the audio stream based, at least in part, on virtualized real-world positioning of the one or more virtual audio sources.

4. The conferencing system of claim 1, wherein the audio filter is further configured to selectively reduce a relative volume of a background real-world sound in the audio stream based, at least in part, on real-world positioning of a real-world audio source producing the background real-world sound.

5. The conferencing system of claim 1, wherein the audio filter is configured to selectively reduce the relative volume of the background virtual sound in the audio stream based, at least in part, on an observed real-world positioning of the one or more real-world audio sources.

6. The conferencing system of claim 5, wherein the observed real-world positioning of the one or more real-world audio sources is identified using receive-beamforming of the real-world sounds captured by each of a plurality of the one or more microphones having spatially diverse positioning within the physical space.

7. The conferencing system of claim 4, wherein responsive to user selection of a first filter mode, a relative volume of a background virtual sound or a background real-world sound is selectively reduced;

and

wherein responsive to user selection of a second filter mode, the relative volume of the background virtual sound or the background real-world sound is not selectively reduced.

8. The conferencing system of claim 1, wherein an audio source is individually selectable for volume reduction relative to other audio sources.

9. The conferencing system of claim 4, wherein a relative volume of a background virtual sound or a background real-world sound is selectively reduced if a real-world positioning of a corresponding audio source is not within a three-dimensional audio-inclusion region;

and

wherein the relative volume of the background virtual sound or the background real-world sound is not selectively reduced if the real-world positioning of the corresponding audio source is within the three-dimensional audio-inclusion region.

10. The conferencing system of claim 9, wherein the three-dimensional audio-inclusion region is based on a field of view or gaze axis of the display device.

11. The conferencing system of claim 9, wherein the three-dimensional audio-inclusion region is user-defined.

12. The conferencing system of claim 4, wherein a relative volume of a background virtual sound or a background real-world sound is selectively reduced if a real-world positioning of a corresponding audio source is within a three-dimensional audio-exclusion region;

and

wherein the relative volume of the background virtual sound or the background real-world sound is not selectively reduced if the real-world positioning of the corresponding audio source is not within the three-dimensional audio-exclusion region.

13. The conferencing system of claim 12, wherein the three-dimensional audio-exclusion region is based on a field of view or gaze axis of the display device.

14. The conferencing system of claim 12, wherein the three-dimensional audio-exclusion region is user-defined.

15. The conferencing system of claim 1, wherein the display device is configured to display a graphical indicator overlaying a real-world view of the physical space and identifying an audio source selected for volume reduction.

16. The conferencing system of claim 1, wherein the display device is configured to display a graphical indicator overlaying a real-world view of the physical space and identifying a real-world positioning of an audio source.

17. A conferencing method, comprising:

capturing, via a plurality of microphones of a spatially-diverse microphone array, real-world sounds produced by one or more real-world audio sources having real-world positioning within a physical environment;

combining the real-world sounds and virtual sounds into an audio stream, the virtual sounds produced by one or more virtual audio sources having virtualized real-world positioning within the physical environment;

displaying a virtual object via an augmented reality display device, the virtual object representing the virtual audio source at a corresponding virtualized real-world positioning;

selectively filtering the virtual sounds from the audio stream based on the virtualized real-world positioning of the one or more virtual audio sources and user selection; and

transmitting the audio stream to a remote communication device of a communication partner.

18. The conferencing method of claim 17, further comprising:

selectively filtering the real-world sounds from the audio stream based on observed real-world positioning of the one or more real-world audio sources.

19. The conferencing method of claim 18, further comprising:

receiving a user selection from the remote communication device over a communication network;

filtering from the audio stream, an audio source selected from among the one or more real-world audio sources and the virtual audio source responsive to the user selection.

20. A conferencing system, comprising:

a spatially-diverse microphone array including a plurality of microphones;

an audio communication interface configured to transmit, to a remote communication device of a communication partner, an audio stream including real-world sounds produced by one or more real-world audio sources captured by the spatially-diverse microphone array and virtual sounds produced by one or more virtual audio sources having virtualized real-world positioning;

an audio filter configured to selectively amplify or dampen a relative volume of one or more of the real-world sounds and the virtual sounds within the audio stream; and

a user input device configured to recognize user input for toggling one or more aspects of volume amplification or dampening via the audio filter.