Producing headphone driver signals in a digital audio signal processing binaural rendering environment

• US 10,187,740 B2
• Filed: 09/23/2016
• Issued: 01/22/2019
• Est. Priority Date: 09/23/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Family, Afrooz
• Primary Examiner(s): Patel, Yogeshkumar
• Application Number:
  US15/275,217
• Publication Number:
  US 20180091920A1
• Time in Patent Office: 851 Days
• Field of Search: 381 17, 381300, 381 1, 381312
• CPC Class Codes: H04R 5/033 Headphones for stereophonic communication details thereof, e.g. relating to batteries, cables or control elements H04R1/10
• CPC Class Codes: H04S 1/005 For headphones
• CPC Class Codes: H04S 2400/01 Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 7/304 For headphones
• CPC Class Codes: H04S 7/306 For headphones

Abstract

A number of candidate binaural room impulse responses (BRIRs) are analyzed to select one of them as a selected first BRIR that is to be applied to diffuse audio, and another one as a selected second BRIR that is to be applied to direct audio, of a sound program. A first binaural rendering process is performed on the diffuse audio by applying the selected first BRIR and a first head related transfer function (HRTF) to the diffuse audio. A second binaural rendering process is performed on the direct audio by applying the selected second BRIR and a second HRTF to the direct audio. Results of the two binaural rendering processes are combined to produce headphone driver signals. Other embodiments are also described and claimed.

Claims

1. A method for rendering a sound program in a binaural rendering environment for headphones, comprising:

receiving an indication of diffuse audio in a sound program;

receiving an indication of direct audio in the sound program;

analyzing a plurality of candidate binaural room impulse responses (BRIRs) to determine a BRIR suitable for diffuse content and another BRIR suitable for direct content;

selecting the BRIR suitable for diffuse content as a selected first BRIR, and selecting the BRIR suitable for direct content as a selected second BRIR;

performing a first binaural rendering process on the diffuse audio to produce a plurality of first intermediate signals, wherein the first binaural rendering process applies the selected first BRIR and a first head related transfer function (HRTF) to the diffuse audio;

performing a second binaural rendering process on the direct audio to produce a plurality of second intermediate signals, wherein the second binaural rendering process applies the selected second BRIR and a second HRTF to the direct audio; and

summing the first and second intermediate signals to produce a plurality of headphone driver signals that are to drive the headphones.

2. The method of claim 1 wherein the diffuse audio and the direct audio overlap each other over time, in the sound program.

3. The method of claim 2 wherein the first and second binaural rendering processes are performed in parallel.

4. The method of claim 1 further comprising

receiving metadata associated with the sound program, wherein the metadata contains the indications of the diffuse and direct audio in the sound program.

5. The method of claim 1 wherein analyzing the plurality of candidate BRIRs to select the selected first and second BRIRs comprises:

classifying room acoustics of the BRIR, extrapolating room geometry, and extracting source directivity information.

6. The method of claim 1 wherein the plurality of candidate BRIRs comprise a plurality of early reflection impulse responses and a plurality of late reflection impulse responses,

wherein content of each of the early reflection impulses response is predominantly direct and early reflections, and

content of each of the late reflection impulse responses is predominantly late reverberation.

7. The method of claim 1 wherein the plurality of candidate BRIRs comprise a plurality of early reflection impulse responses and a plurality of late reflection impulse responses,

wherein one of the plurality of late reflection impulse responses is associated with a room that is larger than a room that is associated with one of the early reflection impulse responses.

8. The method of claim 1 wherein performing the second binaural rendering process to produce the second intermediate signals further comprises

processing the direct audio in accordance with a source model when producing the second intermediate signals, wherein the source model specifies directivity and orientation of a sound source that would produce the sound represented by the direct audio and is independent of room characteristics.

9. The method of claim 1 wherein the direct audio is voice, dialogue or commentary, and the diffuse audio is ambient sounds.

10. The method of claim 1 further comprising

head tracking of a wearer of the headphones,

wherein the second HRTF is updated based on the head tracking but the first HRTF is not updated based on the head tracking.

11. An audio playback system comprising:

a processor; and

memory having stored therein a plurality of candidate binaural room impulse responses (BRIRs), and instructions that when executed by the processor

receive an indication of diffuse audio in a sound program that is to be played back through headphones,

receive an indication of direct audio in the sound program,

analyze the plurality of candidate BRIRs to determine a BRIR suitable for diffuse content and another BRIR suitable for direct content,

select the BRIR suitable for diffuse content as a selected first BRIR, and select the BRIR suitable for direct content as a selected second BRIR,

perform a first binaural rendering process on the diffuse audio to produce a plurality of first intermediate signals, wherein the first binaural rendering process applies the selected first BRIR and a first head related transfer function (HRTF) to the diffuse audio,

perform a second binaural rendering process on the direct audio to produce a plurality of second intermediate signals, wherein the second binaural rendering process applies the selected second BRIR and a second HRTF to the direct audio, and

combine the first and second intermediate signals to produce a plurality of combined headphone driver signals that are to drive the headphones.

12. The audio playback system of claim 11 wherein the instructions program the processor to perform the first and second binaural rendering processes in parallel, and wherein the first and second HRTFs are the same.

13. The audio playback system of claim 11 wherein the instructions program the processor to analyze the plurality of candidate BRIRs to select the selected first and second BRIRs, by classifying room acoustics of each candidate BRIR, extrapolating room geometry of each candidate BRIR, and extracting source directivity information from each candidate BRIR.

14. The audio playback system of claim 11 wherein the plurality of candidate BRIRs comprise a plurality of early reflection impulse responses and a plurality of late reflection impulse responses,

wherein one of the plurality of late reflection impulse responses is associated with a room that is larger than a room that is associated with one of the early reflection impulse responses.

15. The audio playback system of claim 11 wherein the memory has stored therein further instructions that when executed by the processor track orientation of the headphones, wherein the second HRTF and the selected second BRIR are updated based on the tracked orientation of the headphones but the first HRTF and the selected first BRIR are not.

16. The audio playback system of claim 11 wherein the memory has stored therein a source model that specifies directivity and orientation of a sound source that would produce the sound represented by the direct audio and is independent of room characteristics, and instructions that when executed by the processor produce the second intermediate signals by processing the direct audio in accordance with the source model.

17. The audio playback system of claim 11 wherein the memory has stored therein instructions that when executed receive metadata associated with the sound program, wherein the metadata contains the indications of the diffuse and direct audio in the sound program.

18. An article of manufacture comprising:

a non-transitory machine readable storage medium having stored therein a plurality of candidate binaural room impulse responses (BRIRs) and instructions that when executed by a processor

analyze the plurality of candidate BRIRs to determine a BRIR suitable for diffuse content and another BRIR suitable for direct content;

select the BRIR suitable for diffuse content as a selected first BRIR that is to be applied to diffuse audio, and select the BRIR suitable for direct content as a selected second BRIR that is to be applied to direct audio,

perform a first binaural rendering process on the diffuse audio by applying the selected first BRIR and a first head related transfer function (HRTF) to the diffuse audio,

perform a second binaural rendering process on the direct audio by applying the selected second BRIR and a second HRTF to the direct audio, and

combining results of the first and second binaural rendering processes to produce a plurality of headphone driver signals that are to drive the headphones.

19. The article of manufacture of claim 18 wherein the first and second HRTFs are the same.

20. The article of manufacture of claim 18 wherein the diffuse audio and the direct audio overlap each other over time in a sound program that is to be played back through the headphones.

21. The article of manufacture of claim 18 wherein the instructions program the processor to analyze the plurality of candidate BRIRs to select the selected first and second BRIRs by analyzing and classifying number of channels or objects in a sound program that is being processed by the first and second binaural rendering processes, correlation between audio signals of the sound program over time, extraction of metadata associated with the sound program including genre of the sound program, to produce information about the sound program, and matching the sound program information with one or more of the candidate BRIRs.

22. The article of manufacture of claim 18 wherein the plurality of candidate BRIRs comprise a plurality of early reflection impulse responses and a plurality of late reflection impulse responses,

wherein one of the plurality of late reflection impulse responses is associated with a room that is larger than a room that is associated with one of the early reflection impulse responses.