Processing audio signals with head related transfer function filters and a reverberator

• US 7,949,141 B2
• Filed: 10/21/2004
• Issued: 05/24/2011
• Est. Priority Date: 11/12/2003
• Status: Active Grant

• Current Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Original Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Inventors: McKeag, Adam Richard, Reilly, Andrew Peter
• Primary Examiner(s): Chin; Vivian
• Assistant Examiner(s): Olaniran; Fatimat O
• Application Number:
  US10/970,123
• Publication Number:
  US 20050100171A1
• Time in Patent Office: 2,406 Days
• Field of Search: 381 61- 63, 381/310, 381/119, 381 1- 17
• US Class Current: 381/63
• CPC Class Codes: G10H 1/0091 Means for obtaining special acoustic effects combined with modulation G10H1/043
• CPC Class Codes: G10H 1/125 using a digital filter
• CPC Class Codes: G10H 2210/281 Reverberation or echo
• CPC Class Codes: G10H 2210/301 Soundscape or sound field simulation, reproduction or control for musical purposes, e.g. surround or 3D sound; Granular synthesis
• CPC Class Codes: G10K 15/12 using electronic time-delay networks
• CPC Class Codes: H04S 2400/01 Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 7/306 For headphones

Abstract

A method, an apparatus, and a software product to process a plurality of input audio signals. The apparatus accepts a plurality of input signals and includes a multi-input, multi-output reverberator arranged to generate a set of output signals including delayed reverberation components simulating the reverberations a listener is likely to hear in a listening environment. The apparatus further includes a multi-input, two-output filter accepting the outputs of the reverberator and the plurality of input terminals, providing outputs for the left and right ears, and configured to implement a set of head related transfer functions corresponding to a listening environment and a set of directions of a listener in the listening environment. The apparatus is such that a listener listening to the outputs through headphones has the sensation of listening to the plurality of input audio signals as if they are emanating from a plurality of loudspeakers spatially located in the listening environment at a corresponding plurality of directions.

Claims

1. An apparatus to process a plurality of input audio signals comprising:

a plurality of input terminals to accept a plurality of input signals;

a multi-input, multi-output reverberator accepting the plurality of input signals and arranged to generate a set of output signals, each reverberator output signal corresponding to a direction of sound arrival in a listening environment and including delayed echo components simulating the echoes a listener is likely to hear from the corresponding direction in the listening environment, such that each input to the multi-input, multi-output reverberator generates a plurality of outputs corresponding to a plurality of directions of sound arrival; and

a multi-input, two-output filter with inputs coupled to the outputs of the reverberator, the inputs further coupled to the plurality of input terminals, one filter output for the left ear and one filter output for the right ear, the filter arranged to implement a number of head related transfer function (HRTF) filter pairs, each HRTF filter pair corresponding to an HRTF direction of arrival of sound in a listening environment, the two outputs playable through headphones, each HRTF filter pair including a left ear HRTF filter and a right ear HRTF filter, the sum of the left ear HRTF filter outputs forming the left ear output, and the sum of the right ear HRTF filter outputs forming the right ear output of the multi-input, two-output filter,

wherein the reverberator includes a plurality of feedback signal paths, one for each HRTF direction for the listener, such that the coupling of the reverberator outputs to the multi-input, two-output filter couples each of the feedback signal paths to both the left ear HRTF filter and the right ear HRTF filter of a corresponding pair of the HRTF filter pairs, and wherein the reverberator further includes a multi-input, multi-output mixer with inputs coupled to the input terminals and to the outputs of the feedback signal paths, the mixer arranged to mix the plurality of inputs, the mixer outputs coupled to the feedback signal paths, the mixing describable by a non-diagonal matrix, such that at least one mixer output is generated by combining a plurality of mixer inputs,

such that each direct sound corresponding to an input and every separate echo generated by a corresponding reverberator output is processed by both HRTF filters of one pair of the HRTF filter pairs such that each reverberator output is associated with a corresponding binaurally rendered direction of sound arrival,

such that the listener listening to the left and right output signals in the listening environment through headphones has the sensation of listening to the plurality of input audio signals as if they are emanating from a plurality of loudspeakers spatially located at the loudspeaker locations in the listening environment.

2. An apparatus as recited in claim 1, wherein the reverberator is arranged in the forming of at least one of the echo components to combine a plurality of the accepted input signals, and wherein the reverberator is further arranged to process each of the input signals differently.

3. An apparatus as recited in claim 2, further comprising:

a first set of combiners coupled to the inputs of the reverberator and to the input terminals, arranged to combine the plurality of inputs with the set of reverberator outputs to generate a set of inputs for the multi-input, multi-output filters.

4. An apparatus as recited in claim 2, wherein the filter is arranged to generate two sets of outputs for each HRTF filter pair, one set for the left ear, and one set for the right ear, and wherein the filter includes a second set of combiners arranged to combine the left ear set of outputs and the right ear set of outputs to form the left-ear output signal and the right-ear output signal, respectively.

5. An apparatus as recited in claim 2, wherein the reverberator is arranged such that that the echo components include a series of mixed, delayed and filtered versions of the accepted input signals.

6. An apparatus as recited in claim 1, wherein the matrix is a unitary matrix to within each input being multiplied by a respective gain describable by pre-multiplying by a diagonal matrix and/or within each output being multiplied by a respective gain describable by post-multiplying by a diagonal matrix.

7. An apparatus as recited in claim 1, wherein the coupling of the mixer inputs to the input terminals is via a third set of combiners arranged to combine the inputs with delayed filtered versions of the mixer outputs, such that the reverberator includes a plurality of feedback signal paths, with at least one feedback signal path including a delay and filter.

8. An apparatus as recited in claim 1, wherein each of the feedback signal paths includes a delay and filter, each respective filter to produce a desired decay rate that varies with the frequency for echoes produced by the respective feedback signal path, and each respective delay being selected to provide a desired pattern of echoes for the listening environment.

9. An apparatus as recited in claim 8, wherein each filter is selected to achieve a desired reverberation time at low frequencies and a desired reverberation time at high frequencies.

10. An apparatus as recited in claim 8, wherein the delays of the different feedback signal paths are selected to be different with no common factor.

11. An apparatus as recited in claim 10, wherein each of the delays of the different feedback signal paths is selected to be approximately equal to the delay time of the first echo arrival in the listening environment.

12. An apparatus as recited in claim 10, wherein each of the delays of the different feedback signal paths is selected such that the patterns of echoes are uncorrelated for each feedback signal path.

13. An apparatus as recited in claim 1, wherein the number of HRTF directions is less than the number of input signals in the plurality of audio input signals.

14. An apparatus as recited in claim 1, wherein the number of HRTF directions is greater than the number of input signals in the plurality of audio input signals.

15. An apparatus as recited in claim 1, wherein the apparatus further includes a memory arranged to store at least one set of parameters for at least one listening environment, each set sufficient to simulate a listening environment.

16. An apparatus as recited in claim 15, wherein the memory is loaded with a plurality of sets of parameters for a plurality of sets of listening environments.

17. An apparatus as recited in claim 1, wherein the filter and reverberator are implemented by a DSP system having a memory.

18. A method to process a plurality of input audio signals comprising:

accepting a plurality of input signals;

generating a set of reverberator output signals from the plurality of input signals, each reverberator output signal corresponding to a direction of sound arrival in a listening environment, the generating including forming delayed echo components that simulate the echoes a listener is likely to hear from the corresponding direction in the listening environment, such that each input generates a plurality of reverberator output signals corresponding to a plurality of directions of sound arrival; and

filtering combinations of the input signals and reverberator output signals to produce two outputs, one filter output for the left ear and one filter output for the right ear, the filter implementing a number of head related transfer function (HRTF) filter pairs, each HRTF filter pair corresponding to an HRTF direction of arrival of sound in a listening environment, the two outputs playable through headphones, each HRTF filter pair including a left ear HRTF filter and a right ear HRTF filter, the sum of the left ear HRTF filter outputs forming the left ear output, and the sum of the right ear HRTF filter outputs forming the right ear output of the multi-input, two-output filter,

wherein the generating of the set of reverberator output signals includes mixing the plurality of input signals, the mixing describable by a non-diagonal matrix, such that at least one mixing output is generated by combining a plurality of mixing inputs, and wherein the generating of the set of reverberator output signals includes combining the plurality of input signals with delayed filtered versions of the mixer outputs, such that the generating of the set of reverberator output signals includes providing a plurality of feedback signal paths, with at least one feedback signal path including delaying and filtering

such that each direct sound corresponding to an input and every separate echo in a corresponding reverberator output signal is processed by both HRTF filters of one pair of the HRTF filter pairs, such that each reverberator output signal is associated with a corresponding binaurally rendered direction of sound arrival,

such that the listener listening to the left and right output signals in the listening environment through headphones has the sensation of listening to the plurality of input audio signals as if they are emanating from a plurality of loudspeakers spatially located at the loudspeaker locations in the listening environment.

19. A method as recited in claim 18, wherein the forming of at least one of the echo components includes combining a plurality of the accepted input signals, and wherein the generating of a set of reverberator output signals processes different input signals differently.

20. A method as recited in claim 19, further comprising:

combining the plurality of inputs with the set of reverberator outputs to generate a set of inputs for the reverberating.

21. A method as recited in claim 19, wherein the echo components include a series of mixed, delayed and filtered versions of the accepted input signals.

22. A method as recited in claim 19, wherein the matrix is a unitary matrix to within each input being multiplied by a respective gain describable by pre-multiplying by a diagonal matrix and/or within each output being multiplied by a respective gain describable by post-multiplying by a diagonal matrix.

23. A method as recited in claim 18,

wherein the filtering implements a plurality of HRTF filter pairs for a corresponding plurality of HRTF directions, one pair for each HRTF direction for the listener,

wherein the generating of reverberator outputs includes providing a plurality of feedback signal paths, one for each HRTF direction for the listener,

and wherein the method further includes coupling each of the feedback signal paths to both the left ear HRTF filter and the right ear HRTF filter of a corresponding pair of the HRTF filter pairs.

24. A method as recited in claim 23, wherein each of the feedback signal paths includes delaying and filtering, wherein each filter step is to produce a desired decay rate that varies with the frequency for echoes produced by the respective feedback signal path, and wherein each respective delaying step includes applying a respective delay selected to provide a desired pattern of echoes for the listening environment.

25. A method as recited in claim 24, wherein the filtering in each feedback signal path is selected to achieve a desired reverberation time at low frequencies and a desired reverberation time at high frequencies.

26. A method as recited in claim 24, wherein the delays of the different feedback signal paths are selected to be different with no common factor.

27. A method as recited in claim 26, wherein each of the delays of the different feedback signal paths is selected to be approximately equal to the delay time of the first echo arrival in the listening environment.

28. A method as recited in claim 26, wherein each of the delays of the different feedback signal paths is selected such that the patterns of echoes are uncorrelated for each feedback signal path.

29. A method as recited in claim 23, wherein the number of HRTF directions is less than the number of input signals in the plurality of audio input signals.

30. A method as recited in claim 23, wherein the number of HRTF directions is greater than the number of input signals in the plurality of audio input signals.

31. A non-transitory computer-readable storage medium on which is stored at least one code segment that when executed by at least one processor of a processing system causes carrying out a method, the method to process a plurality of input audio signals, the method comprising:

accepting a plurality of input signals;

generating a set of reverberator output signals from a plurality of input signals, each reverberator output signal corresponding to a direction of sound arrival in a listening environment, the generating including forming delayed echo components simulating the echoes a listener is likely to hear from the corresponding direction in the listening environment, such that each input to the multi-input, multi-output reverberator generates a plurality of outputs corresponding to a plurality of directions of sound arrival; and

filtering combinations of the input signals and reverberator output signals to produce two outputs, one filter output for the left ear and one filter output for the right ear, the filter implementing a number of head related transfer function (HRTF) filter pairs, each HRTF filter pair corresponding to an HRTF direction of arrival of sound in a listening environment, the two outputs playable through headphones, each HRTF filter pair including a left ear HRTF filter and a right ear HRTF filter, the sum of the left ear HRTF filter outputs forming the left ear output, and the sum of the right ear HRTF filter outputs forming the right ear output of the multi-input, two-output filter,

wherein the generating of the set of reverberator output signals includes mixing the plurality of input signals, the mixing describable by a non-diagonal matrix, such that at least one mixing output is generated by combining a plurality of mixing inputs, and wherein the generating of the set of reverberator output signals includes combining the accepted inputs with delayed filtered versions of the mixer outputs, such that the generating of the set of reverberator output signals includes providing a plurality of feedback signal paths, with at least one feedback signal path including delaying and filtering,

such that each direct sound corresponding to an input and every separate echo in a corresponding reverberator output signal is processed by both HRTF filters of one pair of the HRTF filter pairs such that each reverberator output signal is associated with a corresponding binaurally rendered direction of sound arrival,

such that the listener listening to the left and right output signals in the listening environment through headphones has the sensation of listening to the plurality of input audio signals as if they are emanating from a plurality of loudspeakers spatially located at the loudspeaker locations in the listening environment.

32. A non-transitory computer-readable storage medium as recited in claim 31, wherein the forming of at least one of the echo components includes combining a plurality of the accepted input signals, and wherein the generating of a set of reverberator output signals processes different input signals differently.

33. A non-transitory computer-readable storage medium as recited in claim 32, wherein the echo components include a series of mixed, delayed and filtered versions of the accepted input signals.

34. A non-transitory computer-readable storage medium as recited in claim 31,

wherein the filtering implements a plurality of HRTF filter pairs for a corresponding plurality of HRTF directions, one pair for each HRFT direction for the listener,

and wherein the method further includes coupling each of the feedback signal paths to both the left ear HRTF filter and the right ear HRTF filter of a corresponding pair of the HRTF filter pairs.

35. A non-transitory computer-readable storage medium as recited in claim 34, wherein each of the feedback signal paths includes delaying and filtering, wherein each filter step is to produce a desired decay rate that varies with the frequency for echoes produced by the respective feedback signal path, and wherein each respective delaying step includes applying a respective delay selected to provide a desired reverberation pattern for the listening environment.

36. A non-transitory computer-readable storage medium as recited in claim 35, wherein the filtering in each feedback signal path is selected to achieve a desired reverberation time at low frequencies and a desired reverberation time at high frequencies.

37. A non-transitory computer-readable storage medium as recited in claim 35, wherein the delays of the different feedback signal paths are selected to be different with no common factor.

38. A non-transitory computer-readable storage medium as recited in claim 37, wherein each of the delays of the different feedback signal paths is selected to be approximately equal to the delay time of the first echo arrival in the listening environment.

39. A non-transitory computer-readable storage medium as recited in claim 37, wherein each of the delays of the different feedback signal paths is selected such that the patterns of echoes are uncorrelated for each feedback signal path.

40. A non-transitory computer-readable storage medium as recited in claim 34, wherein the number of HRTF directions is less than the number of input signals in the plurality of audio input signals.

41. A non-transitory computer-readable storage medium as recited in claim 34, wherein the number of HRTF directions is greater than the number of input signals in the plurality of audio input signals.

42. An apparatus to process a plurality of input audio signals comprising:

means for accepting a plurality of input signals;

means for generating a set of reverberator output signals from a plurality of input signals, each reverberator output signal corresponding to a direction of sound arrival in a listening environment, including forming delayed echo components simulating the echoes a listener is likely to hear from the corresponding direction in the listening environment, such that each input generates a plurality of reverberator output signals corresponding to a plurality of directions of sound arrival; and

means for filtering combinations of the input signals and reverberator output signals to produce two outputs, one filter output for the left ear and one filter output for the right ear, the filter implementing a number of head related transfer (HRTF) filter pairs, each HRTF filter pair corresponding to an HRTF direction of arrival of sound in a listening environment, the two outputs playable through headphones, each HRTF filter pair including a left ear HRTF filter and a right ear HRTF filter, the sum of the left ear HRTF filter outputs forming the left ear output, and the sum of the right ear HRTF filter outputs forming the right ear output of the multi-input, two-output filter,

wherein the means for generating of the set of reverberator output signals is configured to mix the plurality of input signals, the mixing describable by a non-diagonal matrix, such that at least one mixing output is generated by combining a plurality of mixing inputs, and wherein the means for generating the set of reverberator output signals is configured to combine the plurality of input signals with delayed filtered versions of the mixer outputs, such that the means for generating of the set of reverberator output signals includes providing a plurality of feedback signal paths, with at least one feedback signal path including delaying and filtering,

such that each direct sound corresponding to an input and every separate echo in a corresponding reverberator output signal is processed by one of the HRTF filter pairs, such that each reverberator output signal is associated with a corresponding binaurally rendered direction of sound arrival,

such that the listener listening to the left and right output signals in the listening environment through headphones has the sensation of listening to the plurality of input audio signals as if they are emanating from a plurality of loudspeakers spatially located at the loudspeaker locations in the listening environment.

43. An apparatus as recited in claim 42, wherein the forming of at least one of the echo components includes combining a plurality of the accepted input, and wherein the generating of a set of reverberator output signals processes different input signals differently.

44. An apparatus as recited in claim 42, wherein the echo components include a series of mixed, delayed and filtered versions of the accepted input signals.

45. An apparatus as recited in claim 42, wherein the matrix is a unitary matrix to within each input being multiplied by a respective gain describable by pre-multiplying by a diagonal matrix and/or within each output being multiplied by a respective gain describable by post-multiplying by a diagonal matrix.

46. An apparatus as recited in claim 42, wherein each of the feedback signal paths includes delaying and filtering, wherein each filtering step is to produce a desired decay rate that varies with the frequency for echoes produced by the respective feedback signal path, and wherein each respective delaying step includes applying a respective delay selected to provide a desired pattern of echoes for the listening environment.

47. An apparatus as recited in claim 46, wherein the filtering in each feedback signal path is selected to achieve a desired reverberation time at low frequencies and a desired reverberation time at high frequencies.

48. An apparatus as recited in claim 46, wherein the delays of the different feedback signal paths are selected to be different with no common factor.

49. An apparatus as recited in claim 46, wherein each of the delays of the different feedback signal paths is selected to be approximately equal to the delay time of the first echo arrival in the listening environment.

50. An apparatus as recited in claim 46, wherein each of the delays of the different feedback signal paths is selected such that the patterns of echoes are uncorrelated for each feedback signal path.

51. An apparatus as recited in claim 42, wherein the number of HRTF directions is less than the number of input signals in the plurality of audio input signals.

52. An apparatus as recited in claim 42, wherein the number of HRTF directions is greater than the number of input signals in the plurality of audio input signals.