Converting multi-microphone captured signals to shifted signals useful for binaural signal processing and use thereof

• US 10,477,335 B2
• Filed: 09/11/2015
• Issued: 11/12/2019
• Est. Priority Date: 11/19/2010
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: Tammi, Mikko T, Vilermo, Miikka T
• Primary Examiner(s): Chin, Vivian C
• Assistant Examiner(s): Suthers, Douglas J
• Application Number:
  US14/851,266
• Publication Number:
  US 20160007131A1
• Time in Patent Office: 1,523 Days
• Field of Search: 381 92, 381 26, 381 941, 381 942, 381 943, 381122, 381 20, 381 22, 381 23
• CPC Class Codes: G10L 19/008 Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
• CPC Class Codes: H04R 2430/23 Direction finding using a sum-delay beam-former
• CPC Class Codes: H04S 1/002 Non-adaptive circuits, e.g. manually adjustable or static, for enhancing the sound image or the spatial distribution control circuits for electronic adaptation of the sound field H04S7/30
• CPC Class Codes: H04S 2400/01 Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 2420/07 Synergistic effects of band splitting and sub-band processing

Abstract

A method includes, estimating directional information based on multiple input channel signals representing at least one arriving sound from a sound source captured by respective multiple microphones that have respective known locations relative to each other, wherein said estimating comprises finding a time delay that removes a time difference between said first and second input channel signals; deriving a mid-signal and a side signal on a basis of a first input channel signal, a second input channel signal and said estimated directional information; and generating an output signal comprising a plurality of output channels using said mid-signal, said side signal and said estimated directional information such that the output signal retains a spatial representation of the captured at least one arriving sound. Apparatus and program products are also disclosed.

Claims

1. A method comprising:

estimating directional information based on multiple input channel signals representing at least one arriving sound from a sound source captured with respective multiple microphones that have respective known locations relative to each other, wherein the multiple input channel signals include at least a first input channel signal and a second input channel signal and said estimating comprises finding a time delay so as to remove a time difference between said first input channel signal and second input channel signal;

deriving a mid-signal and a side signal on a basis of said first input channel signal, said second input channel signal and said estimated directional information, wherein said deriving further includes deriving the mid-signal as a mid-signal combination based on at least the first input channel signal and the second input channel signal, and deriving the side signal as a side signal combination based on at least the first input channel signal and the second input channel signal, wherein at least one of the mid-signal combination and the side signal combination minimizes a distortion of the at least one arriving sound caused with the at least one arriving sound arriving at different times to at least two or more of the multiple microphones; and

generating an output signal comprising a plurality of output channels using said mid-signal, said side signal and said estimated directional information such that the output signal retains a spatial representation of the at least one arriving sound.

2. The method as claimed in claim 1, wherein the mid-signal combination includes one of the first input channel signal or the second input channel signal shifted with the time delay.

3. The method as claimed in claim 1, where the side signal combination includes one of the first input channel signal or the second input channel signal shifted with the time delay.

4. The method as claimed in claim 1, wherein at least one of the mid-signal combination and the side signal combination is a linear combination.

5. A method comprising:

capturing first, second and third audio signals from respective first, second and third microphones of at least three microphones spaced apart at predetermined distances and arranged in a predetermined geometric configuration;

forming a first resultant signal based on the first and second audio signals;

forming a second resultant signal based on the first and second audio signals;

determining, using at least said first and second audio signals in view of the predetermined geometric configuration, a potential direction of a sound source relative to the at least three microphones;

determining an angle of arriving sound relative to the first and second microphones, the angle having two possible values;

using a best correlation, selecting one of the two possible values of the angle as a direction of the sound source relative to the at least three microphones using the third microphone;

determining left and right output channel signals using the first and second resultant signals and information corresponding to the direction; and

outputting the left and right output channel signals.

6. The method as claimed in claim 5, further comprising:

determining a time delay between at least the first and second audio signals.

7. The method as claimed in claim 6, wherein forming the first resultant signal further comprises:

forming the first resultant signal comprising a sum signal of one of the first or second audio signals shifted with the time delay and the other one of the first or second audio signals.

8. The method as claimed in claim 7, wherein forming the second resultant signal further comprises:

forming the second resultant signal comprising a difference signal between the shifted one of the first or second audio signals and the other one of the first or second audio signals.

9. The method as claimed in claim 8, further comprising:

delaying the sum signal dependent on the two possible values to create two shifted sum audio signals, and

determining which of the two shifted sum audio signals has a best correlation with the third audio signal.

10. An apparatus, comprising:

one or more processors, and

one or more non-transitory memories including computer program code, the one or more non-transitory memories and the computer program code configured, with the one or more processors, to cause the apparatus to perform at least the following;

estimate directional information based on multiple input channel signals representing at least one arriving sound from a sound source captured with respective multiple microphones that have respective known locations relative to each other, wherein the multiple input channel signals include at least a first input channel signal and a second input channel signal and estimating comprises finding a time delay so as to remove a time difference between said first input channel signal and second input channel signal;

derive a mid-signal and a side signal on a basis of said first input channel signal, said second input channel signal and said estimated directional information, wherein deriving further includes deriving the mid-signal as a mid-signal combination based on at least the first input channel signal and the second input channel signal, and deriving the side signal as a side signal combination based on at least the first input channel signal and the second input channel signal, wherein at least one of the mid-signal combination and the side signal combination minimizes a distortion of the at least one arriving sound caused with the at least one arriving sound arriving at different times to at least two or more of the multiple microphones; and

generate an output signal comprising a plurality of output channels using said mid-signal, said side signal and said estimated directional information such that the output signal retains a spatial representation of the at least one arriving sound.

11. A computer program product embodied in a non-transitory computer memory and comprising instructions the execution of which with a processor results in performing operations that comprise:

estimating directional information based on multiple input channel signals representing at least one arriving sound from a sound source captured with respective multiple microphones that have respective known locations relative to each other, wherein the multiple input channel signals include at least a first input channel signal and a second input channel signal and said estimating comprises finding a time delay so as to remove a time difference between said first input channel signal and second input channel signal;

deriving a mid-signal and a side signal on a basis of said first input channel signal, said second input channel signal and said estimated directional information, wherein said deriving further includes deriving the mid-signal as a mid-signal combination based on at least the first input channel signal and the second input channel signal, and deriving the side signal as a side signal combination based on at least the first input channel signal and the second input channel signal, wherein at least one of the mid-signal combination and the side signal combination minimizes a distortion of the at least one arriving sound caused with the at least one arriving sound arriving at different times to at least two or more of the multiple microphones; and

generating an output signal comprising a plurality of output channels using said mid-signal, said side signal and said estimated directional information such that the output signal retains a spatial representation of the at least one arriving sound.

12. A method comprising:

receiving a first audio signal from a first microphone, a second audio signal from a second microphone, and a third audio signal from a third microphone, where locations of each of the first microphone, the second microphone, and the third microphone are known, and where each of the first audio signal, the second audio signal, and the third audio signal comprises sound arriving from a sound source;

determining a first potential direction of the sound arriving from the sound source based on analysis of the first audio signal and the second audio signal;

determining a second potential direction of the sound arriving from the sound source based on analysis of the first audio signal and the second audio signal;

determining a combined audio signal, where the combined audio signal comprises the first audio signal and a shifted version of the second audio signal;

determining one of the first potential direction or the second potential direction as a direction of the sound arriving from the sound source based on the third audio signal; and

generating one or more output signals based, at least partially, on the direction of the sound arriving from the sound source and the combined audio signal.

13. The method as claimed in claim 12, where the determining of the combined audio signal further comprises:

determining a delay that maximizes correlation between the first audio signal and the second audio signal; and

determining the shifted version of the second audio signal, where the determining of the shifted version of the second audio signal comprises shifting the second audio signal with the determined delay.

14. The method as claimed in claim 12, where the determining of the one of the first potential direction or the second potential direction as the direction further comprises:

determining a first distance between the third microphone and a first sound source located in the first potential direction;

determining a first delay based on the first distance;

determining a second distance between the third microphone and a second sound source located in the second potential direction;

determining a second delay based on the second distance;

determining a delay that provides better correlation between the third audio signal and the combined audio signal, where the delay comprises one of the first delay or the second delay; and

determining the one of the first potential direction or the second potential direction as the direction based, at least partially, on the delay.

15. The method as claimed in claim 12, where the generating of the one or more output signals is further based, at least partially, on a side signal, where the side signal is determined based on a difference between the first audio signal and the shifted version of the second audio signal.

16. The method as claimed in claim 15, where the generating of the one or more output signals further comprises:

processing the combined audio signal, where the processing of the combined audio signal comprises applying head related transfer functions to subbands of the combined audio signal; and

processing the side signal, where the processing of the side signal comprises applying a fixed delay to subbands of the side signal.

17. The method as claimed in claim 16, further comprising:

determining one or more left output channel signals and one or more right output channel signals, where the determining of the one or more left output channel signals and the one or more right output channel signals comprises combining the processed combined audio signal and the processed side signal, and where the one or more output signals comprise the one or more determined left output channel signals and the one or more determined right output channel signals.