Method and apparatus for three-dimensional audio display

• US 7,231,054 B1
• Filed: 09/24/1999
• Issued: 06/12/2007
• Est. Priority Date: 09/24/1999
• Status: Expired due to Fees

• Current Assignee: Creative Technology Ltd.
• Original Assignee: Creative Technology Ltd.
• Inventors: Jot, Jean-Marc, Wardle, Scott
• Primary Examiner(s): Chin; Vivian
• Assistant Examiner(s): KURR, JASON RICHARD
• Application Number:
  US09/806,193
• Time in Patent Office: 2,818 Days
• Field of Search: 381/309, 381/17, 381/23, 381/18, 381/22, 381/310
• US Class Current: 381/310
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 3/00 Systems employing more than two channels, e.g. quadraphonic H04S5/00, H04S7/00 take precedence

Abstract

This invention addresses sound recording and mixing methods for 3-D audio rendering of multiple sound sources over headphones or loudspeaker playback systems. Economical techniques are provided, whereby directional panning and mixing of sounds are performed in a multi-channel encoding format which preserves interaural time difference information and does not contain head-related spectral information. Decoders are provided for converting the multi-channel encoded signal into signals for playback over headphones or various loudspeaker arrangements. These decoders ensure faithful reproduction of directional auditory information at the eardrums of the listener and can be adapted to the number and geometrical layout of the loudspeakers and the individual characteristics of the listener. A particular multi-channel encoding format is disclosed, which, in addition to the above advantages, is associated with a practical microphone technique for producing 3-D audio recordings compliant with the decoders described.

Claims

1. A method for positioning of a plurality of audio signals, the method including:

selecting a set of spatial functions, each having an associated scaling factor;

providing a first set of amplifiers and a second set of amplifiers, the gains of the amplifiers being functions of the scaling factors;

receiving a first audio signal of the plurality of audio signals;

providing a first direction representing the direction of the source of the first audio signal;

adjusting the gains of the first and the second set of amplifiers depending on the first direction;

applying the first set of amplifiers to the first audio signal to produce first encoded signals;

delaying the first audio signal to produce a first delayed audio signal; and

applying the second set of amplifiers to the first delayed audio signal to produce second encoded signals;

providing a third set of amplifiers and a fourth set of amplifiers, the gains of the amplifiers being functions of the scaling factors;

receiving a second audio signal of the plurality of audio signals;

providing a second direction representing the direction of the source of the second audio signal;

adjusting the gains of the third and the fourth set of amplifiers depending on the second direction;

applying the third set of amplifiers to the second audio signal to produce third encoded signals;

delaying the second audio signal to produce a second delayed audio signal;

applying the fourth set of amplifiers to the second delayed audio signal to produce fourth encoded signals;

mixing the first and the third encoded signals or the first and the fourth encoded signals to provide a left-channel audio output;

mixing the second and the fourth encoded signals or the second and the third encoded signals to provide a right-channel audio output, the left-channel audio output excluding the second encoded signal and the right-channel audio output excluding the first encoded signal; and

decoding the encoded signals using filters that are defined based on the spatial functions.

2. The method of claim 1 wherein the spatial functions are spherical harmonic functions.

3. The method of claim 2 wherein the spherical harmonic functions include at least the first-order harmonics.

4. The method of claim 1 wherein the spatial functions are discrete panning functions.

5. The method of claim 1 wherein for each of the first and second sets of amplifiers, the gain of each amplifier is based on a B-format encoding scheme.

6. The method of claim 1 wherein the second signal is a synthesized audio signal.

7. A method of producing an audio signal from directionally encoded multi-channel audio signals, the method including:

selecting a set of spatial functions;

generating a set of spectral functions based on the spatial functions;

receiving a first set of directionally encoded audio signals encoded according to the set of spatial functions, the first set of directionally encoded signals providing an encoded left-channel input;

receiving a second of set directionally encoded audio signals encoded according to the set of spatial functions, the second set of directionally encoded signals providing an encoded right-channel input, the encoded left-channel input excluding the second set of directionally encoded signals and the encoded right-channel input excluding the first set of directionally encoded signals;

providing a first set of decoding filters defined by the set of spectral functions;

providing a second set of decoding filters defined by the set of spectral functions;

applying the first set of decoding filters to the first set of directionally encoded audio signals to produce a first set of filtered signals;

applying the second set of decoding filters to the second set of directionally encoded audio signals to produce a second set of filtered signals; and

providing the first set of filtered signals to a left-channel audio output and providing the second set of filtered signals to a right-channel audio output.

8. The method of claim 7 wherein the set of spatial functions is defined by {g_i(θ

, φ

), i=0, 1, . . . N−

1} and generating the spectral functions includes providing L_i(f) and R_i(f) such that Σ

<sub>{i=0, . . . N−

1}</sub> g_i(θ

_p, φ

_p) L_i(f) approximates L(θ

_p, φ

_p, f) and Σ

<sub>{i=0, . . . N−

1} gi</sub>(θ

_p, φ

_p) R_i(f) approximates R(θ

_p, φ

_p, f), where L(θ

_p, φ

_p, f) is a set of left-ear HRTFs and R(θ

_p, φ

_p, f) is a set of right-ear HRTFs, where {(θ

_p, φ

_p), p=1, 2, . . . P} is a set of directions and f is frequency.

9. The method of claim 8 wherein L(θ

_p, φ

_p, f) and R(θ

_p, φ

_p, f) are delay-free HRTFs.

10. The method of claim 8 wherein providing L_i(f) includes solving, at each frequency f, the vector equation L≈

GL, where;

the set of left-ear HRTFs L(θ

_p, φ

_p, f) define a P×

1 vector L,

G is a P×

N matrix whose columns are P×

1 vectors G_i, i=0, 1, . . . N−

1

each of the N spatial functions g_i(θ

_p, φ

_p, f) defines the vector G_i, and

the set of L_i(f) defines N×

1 vector L.

11. The method of claim 10 wherein providing L_i(f) is obtained by pseudo-inversion of the matrix G, resulting in L=(G^TG)⁻

1G^TL.

12. The method of claim 11 wherein providing L_i(f) includes projecting the P×

1 vector L formed by the set of left-ear HRTFs L(θ

_p, φ

_p, f) over each of the P×

1 vectors G_i formed by the spatial functions g_i(θ

_p, φ

_p, f) to compute the scalar product L_i.

13. The method according to claim 12 wherein an N×

1 vector L formed by the scalar products L_i is multiplied by the inverse of the Gram matrix G^TG.

14. The method of claim 10 wherein providing L_i(f) is obtained by L=(G^TΔ

G)<sup>−

1</sup>G^TΔ

L where Δ

is a diagonal P×

P matrix where the P diagonal elements are weights applied to the individual directions (θ

_p, φ

_p), p=1, 2, . . . P.

15. The method of claim 14 where each weight is proportional to a solid angle associated with the corresponding direction.

16. The method of claim 7 wherein the spatial functions are spherical harmonic functions.

17. The method of claim 16 wherein the spherical harmonic functions include at least zero- and first-order harmonics.

18. The method of claim 17 wherein the spectral functions define filters L_W(f), L_X(f), L_Y(f), and L_Z(f) effective for decoding binaural B-format encoded signals W_L, X_L, Y_L, Z_L, W_R, X_R, Y_R, Z_R, wherein the left-channel audio signal is defined by W_LL_W(f)+X_LL_X(f)+Y_LL_Y(f)+Z_LL_Z(f) and the right-channel audio signal is defined by W_RL_W(f)+X_RL_X(f)−

Y_RL_Y(f)+Z_RL_Z(f);

whereby left-and right-channel audio signals are suitable for playback with headphones.

19. The method of claim 17 wherein the spectral functions define filters L_W(f), L_X(f), L_Y(f), and L_Z(f) effective for decoding binaural B-format encoded signals W_L, X_L, Y_L, Z_L, W_R, X_R, Y_R, and Z_R;

wherein the left-channel audio signal comprises two signals

a first signal LF=0.5{[W_L+X_L][L_w(f)+L_X(f)]+Y_LL_Y(f)+Z_LL_Z(f)} and

a second signal LB=0.5{[W_L−

X_L][L_W(f)−

L_X(f)]+Y_LL_Y(f)+Z_LL_Z(f)};

and wherein the right-channel audio signal comprises two signals

a first signal RF=0.5{[W_R+X_R][L_W(f)+L_x(f)]+Y_RL_Y(f)+Z_RL_Z(f)} and

a second signal RB=0.5{[W_R−

X_R][L_W(f)−

L_X(f)]−

Y_RL_Y(f)+Z_RL_Z(f)};

whereby the left- and right-channel audio signals are suitable for playback over a pair of front speakers and a pair of rear speakers.

20. The method of claim 19 further including:

performing a first cross-talk cancellation on the LF and RF signals to feed the front speakers; and

performing a second cross-talk cancellation on the LB and RB signals to feed the rear speakers.

21. The method according to claim 20 including cross-talk cancellation of the left and right audio signals before feeding the loudspeakers.

22. The method of claim 7 wherein the spatial functions are discrete panning functions having a direction, called a principal direction, where the spatial function is maximum and wherein all other spatial functions are zero.

23. The method of claim 22 wherein the spectral function associated with each spatial function is the delay-free HRTF for the corresponding principal direction.

24. The method according to claims 22 or 23 wherein one or more of the spatial functions have their principal direction corresponding to a direction of one of the loudspeakers.

25. The method according to claim 24 including performing cross-talk cancellation of the left and right audio signals before feeding the loudspeakers.

26. The method of claims 22 or 23 further including:

producing left-front and left-back signals based on the left-channel audio signal;

producing right-front and right-back signals based on the right-channel audio signal; and

combining the left-front, left-back, right-front, and right-back signals to produce outputs suitable for playback with a pair of front speakers and a pair of rear speakers.

27. The method of claim 26 further including:

performing a first cross-talk cancellation on the left-front and right-front signals to feed the front speakers; and

performing a second cross-talk cancellation on the left-back and right-back signals to feed the rear speakers.

28. The method of claim 27 wherein one or more of the spatial functions have their principal direction corresponding to the direction of a loudspeaker.