Binaural synthesis

• US 10,531,217 B2
• Filed: 11/14/2018
• Issued: 01/07/2020
• Est. Priority Date: 10/08/2015
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Nair, Varun
• Primary Examiner(s): Tran, Thang V
• Application Number:
  US16/191,283
• Publication Number:
  US 20190116442A1
• Time in Patent Office: 419 Days
• CPC Class Codes: H04R 5/033 Headphones for stereophonic communication details thereof, e.g. relating to batteries, cables or control elements H04R1/10
• CPC Class Codes: H04S 2400/01 Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 7/30 Control circuits for electronic adaptation of the sound field
• CPC Class Codes: H04S 7/307 Frequency adjustment, e.g. tone control H04S7/301 takes precedence

Abstract

Embodiments relate to obtaining filter coefficients for a binaural synthesis filter; and applying a compensation filter to reduce artefacts resulting from the binaural synthesis filter; wherein the filter coefficients and compensation filter are configured to be used to obtain binaural audio output from a monaural audio input. The filter coefficients and compensation filter may be applied to the monaural audio input to obtain the binaural audio output. The compensation filter may comprise a timbre compensation filter.

Claims

1. A method comprising:

obtaining a monaural audio input representative of monoaural audio input from a plurality of audio sources;

for each audio source, selecting at least one binaural synthesis model from a plurality of binaural synthesis models and applying the at least one binaural synthesis model to process the monoaural audio input from that audio source to obtain at least one binaural audio output; and

obtaining a combined binaural audio output by combining binaural audio outputs corresponding to the processed monoaural audio inputs from the selected binaural synthesis models for each of the audio sources.

2. The method according to claim 1, wherein the plurality of binaural synthesis models comprises at least one of a head-related impulse response (HRIR) binaural synthesis model, a structural model, and a virtual speakers model.

3. The method according to claim 1, wherein the selected at least one binaural synthesis model of the plurality of binaural synthesis models comprises at least a first, higher-quality binaural synthesis model and a second, lower-quality binaural synthesis model, and wherein the first, higher-quality binaural synthesis model is selected for a first, higher-priority audio source;

and the second, lower-quality binaural synthesis model is selected for a second, lower-priority audio source.

4. The method according to claim 1, wherein the selecting of the binaural synthesis models is dependent on a distance of each audio source from a position with respect to which the binaural synthesis is performed.

5. The method according to claim 1, wherein, for an audio source of the plurality of audio sources:

selecting at least one binaural synthesis model for the audio source comprises selecting a first binaural synthesis model and a second binaural synthesis model different from the first binaural synthesis model, and

the combined audio output comprises a first proportion of an audio output for the audio source from the first binaural synthesis model and a second proportion of an audio output for the audio source from the second binaural synthesis model.

6. The method according to claim 5, wherein the position of the audio source changes over time, and the first proportion and second proportion change with time in accordance with the changing position of the audio source.

7. The method according to claim 1, wherein each of the plurality of binaural synthesis models comprises a respective timbre compensation filter, the timbre compensation filters being configured to match timbre between the binaural synthesis models.

8. The method according to claim 1, wherein the binaural synthesis models are selected depending on at least one of:

a central processing unit (CPU) frequency, a computational resource limit, a computational resource parameter, or a quality requirement.

9. The method according to claim 1, wherein the binaural synthesis models are selected in dependence on a priority of each audio source, a distance associated with each audio source, a quality requirement of each audio source, or an amplitude of each audio source.

10. A non-transitory computer readable storage medium storing instructions, the instructions when executed by a processor cause the processor to:

obtain a monaural audio input representative of monoaural audio input from a plurality of audio sources;

for each audio source, select at least one binaural synthesis model from a plurality of binaural synthesis models and applying the at least one binaural synthesis model to process the monoaural audio input from that audio source to obtain at least one binaural audio output; and

obtain a combined binaural audio output by combining binaural audio outputs corresponding to the processed monoaural audio inputs from the selected binaural synthesis models for each of the audio sources.

11. The non-transitory computer readable storage medium according to claim 10, wherein the plurality of binaural synthesis models comprises at least one of a head-related impulse response (HRIR) binaural synthesis model, a structural model, and a virtual speakers model.

12. The non-transitory computer readable storage medium according to claim 10, wherein the selected at least one binaural synthesis model of the plurality of binaural synthesis models comprises at least a first, higher-quality binaural synthesis model and a second, lower-quality binaural synthesis model, and wherein the first, higher-quality binaural synthesis model is selected for a first, higher-priority audio source;

and the second, lower-quality binaural synthesis model is selected for a second, lower-priority audio source.

13. The non-transitory computer readable storage medium according to claim 10, wherein the selecting of the binaural synthesis models is dependent on a distance of each audio source from a position with respect to which the binaural synthesis is performed.

14. The non-transitory computer readable storage medium according to claim 10, wherein instructions to select at least one binaural synthesis model comprises instructions to:

select at least one binaural synthesis model for the audio source comprises selecting a first binaural synthesis model and a second binaural synthesis model different from the first binaural synthesis model, and

the combined audio output comprises a first proportion of an audio output for the audio source from the first binaural synthesis model and a second proportion of an audio output for the audio source from the second binaural synthesis model.

15. The non-transitory computer readable storage medium according to claim 14, wherein the position of the audio source changes over time, and the first proportion and second proportion change with time in accordance with the changing position of the audio source.

16. The non-transitory computer readable storage medium according to claim 10, wherein each of the plurality of binaural synthesis models comprises a respective timbre compensation filter, the timbre compensation filters being configured to match timbre between the binaural synthesis models.

17. The non-transitory computer readable storage medium according to claim 10, wherein the binaural synthesis models are selected depending on at least one of:

a central processing unit (CPU) frequency, a computational resource limit, a computational resource parameter, or a quality requirement.

18. The non-transitory computer readable storage medium according to claim 10, wherein the binaural synthesis models are selected in dependence on a priority of each audio source, a distance associated with each audio source, a quality requirement of each audio source, or an amplitude of each audio source.