Three-dimensional sound capturing and reproducing with multi-microphones

• US 9,552,840 B2
• Filed: 10/24/2011
• Issued: 01/24/2017
• Est. Priority Date: 10/25/2010
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Kim, Lae-Hoon, Visser, Erik, Xiang, Pei, Liu, Ian Ernan, Ramakrishnan, Dinesh
• Primary Examiner(s): Chin, Vivian
• Assistant Examiner(s): SUTHERS, DOUGLAS JOHN
• Application Number:
  US13/280,303
• Publication Number:
  US 20120128160A1
• Time in Patent Office: 1,919 Days
• Field of Search: 381/17, 381/92, 381/313, 381/333, 381/334
• US Class Current: 1/1
• CPC Class Codes: G11B 20/00 Signal processing not specific to the method of recording or reproducing; Circuits therefor
• CPC Class Codes: H04R 2201/401 2D or 3D arrays of transducers
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction

Abstract

A method for audio signal processing is described. The method includes decomposing a recorded auditory scene into a first category of localizable sources and a second category of ambient sound. The method also includes recording an indication of the directions of each of the localizable sources. The method may be performed with a device having a microphone array.

Claims

1. A method of audio signal processing by a device, said method comprising:

decomposing a recorded auditory scene, recorded using three microphones that form two microphone pairs on the device, into a first category of localizable sources and a second category of ambient sound, wherein each localizable source represents one of multiple different directions, wherein each direction represents one of multiple corners of the device, wherein the corners of the device include a front left direction, a back left direction, a front right direction, and a back right direction, and wherein the directions are associated with a standard channel format surround sound system;

recording an indication of the direction of each of the localizable sources; and

applying at least one multichannel filter to a plurality of recorded input channels of the auditory scene by combining two overlapping beams to enhance one of the localizable sources associated with one of the directions and applying a null beam in all other of the directions.

2. A method according to claim 1, wherein said method comprises receiving user input that comprises a desired recording direction.

3. A method according to claim 1, wherein said method comprises estimating a number of the localizable sources, including estimating a direction of arrival of each source.

4. A method according to claim 1, wherein said method comprises encoding a recorded multichannel signal according to a three-dimensional audio encoding scheme.

5. A method according to claim 4, wherein the three-dimensional audio encoding scheme is at least one among an Ambisonic scheme and a wavefield synthesis scheme.

6. A method according to claim 1, wherein the combining two overlapping beams comprises:

applying a beam in one endfire direction to obtain a first filtered signal;

applying a beam in another endfire direction to obtain a second filtered signal; and

combining the first filtered signal with a delayed version of the second filtered signal.

7. A method according to claim 1, wherein the combining two overlapping beams comprises:

applying a beam in one endfire direction to obtain a first filtered signal;

applying a beam in another endfire direction to obtain a second filtered signal; and

combining the first and second filtered signals to obtain a combined signal,

wherein each of the first and second filtered signals has at least two channels, and

wherein one channel of the combined signal is delayed relative to another channel of the combined signal.

8. The method according to claim 7, wherein said method comprises, prior to said combining, delaying one channel of the first filtered signal relative to another channel of the first filtered signal and delaying one channel of the second filtered signal relative to another channel of the second filtered signal.

9. The method according to claim 7, wherein said method comprises delaying one channel of the combined signal relative to another channel of the combined signal.

10. A method according to claim 1, wherein the combining two overlapping beams comprises:

applying a filter having a beam in a first direction to a signal produced by a first pair of microphones to obtain a first spatially filtered signal;

applying a filter having a beam in a second direction different from the first direction to a signal produced by a second pair of microphones that is different from the first pair of microphones to obtain a second spatially filtered signal; and

combining the first and second spatially filtered signals to obtain an output signal corresponding to a source in a third direction different than the first and second directions.

11. The method according to claim 10, wherein an axis of the first pair of microphones is at least substantially orthogonal to an axis of the second pair of microphones.

12. A method according to claim 10, wherein said applying the filter having the beam in the first direction is limited to frequencies between a low threshold and a high threshold, and

wherein at least one of the low and high thresholds is based on a distance between microphones.

13. A method according to claim 1, wherein said method comprises:

for each of the three microphones in an array, recording a corresponding input channel; and

for each of the directions, applying a corresponding multichannel filter to the plurality of the recorded input channels to obtain a corresponding output channel.

14. The method according to claim 13, wherein an axis of a first pair of the three microphones is not more than fifteen degrees from orthogonal to an axis of a second pair of the three microphones.

15. The method according to claim 13, wherein there are at least four directions that each represent one of the corners of the device.

16. The method according to claim 13, wherein said method comprises processing the plurality of output channels to produce a binaural recording,

wherein said processing includes, for each of the plurality of output channels, applying a corresponding binaural impulse response to the output channel to obtain a corresponding binaural signal,

wherein said binaural recording is based on a sum of said binaural signals.

17. A method according to claim 1, wherein said method comprises selecting a recording direction based on an orientation of a portable audio sensing device.

18. A method according to claim 1, wherein said method includes training a plurality of filters to generate a corresponding plurality of fixed converged filters;

and

applying the fixed converged filters to the recorded auditory scene to perform said decomposing.

19. The method of claim 1, wherein the three microphones are substantially in a same plane as a display of the device.

20. An apparatus for audio signal processing, said apparatus comprising:

means for decomposing a recorded auditory scene, recorded using three microphones that form two microphone pairs on the apparatus, into a first category of localizable sources and a second category of ambient sound, wherein each localizable source represents one of multiple different directions, wherein each direction represents one of multiple corners of the apparatus, wherein the corners of the apparatus include a front left direction, a back left direction, a front right direction, and a back right direction, and wherein the directions are associated with a standard channel format surround sound system;

means for recording an indication of the direction of each of the localizable sources; and

means for applying at least one multichannel filter to a plurality of recorded input channels of the auditory scene by combining two overlapping beams to enhance one of the localizable sources associated with one of the directions and applying a null beam in all other of the directions.

21. An apparatus according to claim 20, wherein said apparatus comprises means for receiving user input that comprises a desired recording direction.

22. An apparatus according to claim 20, wherein said apparatus comprises means for estimating a number of the localizable sources, including means for estimating a direction of arrival of each source.

23. An apparatus for audio signal processing, said apparatus comprising:

a display configured to present a user interface; and

one or more processors configured to;

decompose a recorded auditory scene, recorded using three microphones that form two microphone pairs on the apparatus, into a first category of localizable sources and a second category of ambient sound, wherein each localizable source represents one of multiple different directions, wherein each direction represents one of multiple corners of the apparatus, wherein the corners of the apparatus include a front left direction, a back left direction, a front right direction, and a back right direction, and wherein the directions are associated with a standard channel format surround sound system;

record an indication of the direction of each of the localizable sources; and

apply at least one multichannel filter to a plurality of recorded input channels of the auditory scene by combining two overlapping beams to enhance one of the localizable sources associated with one of the directions and applying a null beam in all other of the directions.

24. An apparatus according to claim 23, wherein the one or more processors are configured to receive user input that comprises a desired recording direction.

25. An apparatus according to claim 23, wherein the one or more processors are configured to estimating a number of the localizable sources, including estimating a direction of arrival of each source.

26. An apparatus according to claim 23, wherein the one or more processors are configured to encode a recorded multichannel signal according to a three-dimensional audio encoding scheme.

27. An apparatus according to claim 26, wherein the three-dimensional audio encoding scheme is at least one among an Ambisonic scheme and a wavefield synthesis scheme.

28. An apparatus according to claim 23, wherein the one or more processors configured to apply at least one multichannel filter by combining two overlapping beams are further configured to:

apply a beam in one endfire direction to obtain a first filtered signal;

apply a beam in another endfire direction to obtain a second filtered signal; and

combine the first filtered signal with a delayed version of the second filtered signal.

29. An apparatus according to claim 28, wherein applying the beam in the one endfire direction is configured to apply the beam only to frequencies between a low threshold and a high threshold, and

wherein at least one of the low and high thresholds is based on a distance between microphones.

30. An apparatus according to claim 23, wherein the one or more processors configured to apply at least one multichannel filter by combining two overlapping beams are further configured to:

apply a beam in one endfire direction to obtain a first filtered signal;

apply a beam in another endfire direction to obtain a second filtered signal; and

combine the first and second filtered signals to obtain a combined signal,

wherein each of the first and second filtered signals has at least two channels, and

wherein one channel of the combined signal is delayed relative to another channel of the combined signal.

31. An apparatus according to claim 30, wherein the one or more processors are configured to delay one channel of the first filtered signal relative to another channel of the first filtered signal and delay one channel of the second filtered signal relative to another channel of the second filtered signal, prior to said combining.

32. An apparatus according to claim 30, wherein the one or more processors are configured to delay one channel of the combined signal relative to another channel of the combined signal.

33. An apparatus according to claim 23, wherein the one or more processors configured to apply at least one multichannel filter by combining two overlapping beams are further configured to:

apply a filter having a beam in a first direction to a signal produced by a first pair of microphones to obtain a first spatially filtered signal;

apply a filter having a beam in a second direction different from the first direction to a signal produced by a second pair of microphones that is different from the first pair of microphones to obtain a second spatially filtered signal; and

combine the first and second spatially filtered signals to obtain an output signal corresponding to a source in a third direction different than the first and second directions.

34. An apparatus according to claim 33, wherein an axis of the first pair of microphones is at least substantially orthogonal to an axis of the second pair of microphones.

35. An apparatus according to claim 23, wherein the one or more processors are configured to:

record, for each of the three microphones in an array, a corresponding input channel; and

apply, for each of the directions, a corresponding multichannel filter to the plurality of the recorded input channels to obtain a corresponding output channel.

36. An apparatus according to claim 35, wherein an axis of a first pair of the three microphones is not more than fifteen degrees from orthogonal to an axis of a second pair of the three microphones.

37. The apparatus according to claim 35, wherein there are at least four directions that each represent one of the corners of the apparatus.

38. An apparatus according to claim 35, wherein the one or more processors are configured to process the plurality of output channels to produce a binaural recording, wherein said processing includes, for each of the plurality of output channels, applying a corresponding binaural impulse response to the output channel to obtain a corresponding binaural signal,

wherein said binaural recording is based on a sum of said binaural signals.

39. An apparatus according to claim 23, wherein the one or more processors are configured to select a recording direction based on an orientation of a portable audio sensing device.

40. A non-transitory machine-readable storage medium comprising tangible features that when read by a device cause the device to:

decompose a recorded auditory scene, recorded using three microphones that form two microphone pairs on the device, into a first category of localizable sources and a second category of ambient sound, wherein each localizable source represents one of multiple different directions, wherein each direction represents one of multiple corners of the device, wherein the corners of the device include a front left direction, a back left direction, a front right direction, and a back right direction, and wherein the directions are associated with a standard channel format surround sound system;

record an indication of the direction of each of the localizable sources; and

apply at least one multichannel filter to a plurality of recorded input channels of the auditory scene by combining two overlapping beams to enhance one of the localizable sources associated with one of the directions and applying a null beam in all other of the directions.