Content-adaptive surround sound virtualization

• US 10,306,392 B2
• Filed: 11/02/2016
• Issued: 05/28/2019
• Est. Priority Date: 11/03/2015
• Status: Active Grant

• Current Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Original Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Inventors: Liu, Xin, Lu, Lie, Seefeldt, Alan J.
• Primary Examiner(s): Gay, Sonia L
• Application Number:
  US15/772,383
• Publication Number:
  US 20180324540A1
• Time in Patent Office: 937 Days
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: H04R 1/26 Spatial arrangements of separate transducers responsive to two or more frequency ranges
• CPC Class Codes: H04S 2400/03 Aspects of down-mixing multi-channel audio to configurations with lower numbers of playback channels, e.g. 7.1 -> 5.1 H04S2400/01 takes precedence
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 5/00 Pseudo-stereo systems, e.g. in which additional channel signals are derived from monophonic signals by means of phase shifting, time delay or reverberation
• CPC Class Codes: H04S 5/005 of the pseudo five- or more-channel type, e.g. virtual surround
• CPC Class Codes: H04S 7/30 Control circuits for electronic adaptation of the sound field
• CPC Class Codes: H04S 7/307 Frequency adjustment, e.g. tone control H04S7/301 takes precedence

Abstract

Example embodiments disclosed herein relate to content-adaptive surround sound virtualization. A method of virtualizing surround sound is disclosed. The method includes receiving a set of input audio signals, each of the input audio signals being indicative of sound from one of different sound sources, and determining a probability of the set of input audio signals belonging to a predefined audio content category. The method also includes determining a virtualization amount based on the determined probability, the virtualization amount indicating to which extent the set of input audio signals is virtualized as surround sound. The method further includes performing surround sound virtualization on two or more input audio signals in the set based on the determined virtualization amount and generating output audio signals based on the virtualized input audio signals and other input audio signals in the set. Corresponding system and computer program product for virtualizing surround sound are also disclosed.

Claims

1. A method of virtualizing surround sound, comprising:

receiving a set of input audio signals, each of the input audio signals being indicative of sound from one of different sound sources;

obtaining respective location information of the sound sources of the input audio signals, and respective locations of speakers which are to be driven by output audio signals;

determining a probability of the set of input audio signals belonging to a predefined audio content category;

determining a virtualization amount based on the determined probability, the virtualization amount indicating to which extent the set of input audio signals is virtualized as surround sound;

performing surround sound virtualization on two or more input audio signals in the set based on the determined virtualization amount; and

generating the output audio signals based on the virtualized input audio signals and one or more other input audio signals in the set,

wherein said performing surround sound virtualization comprises adjusting the obtained location information of the sound sources such that, for each of the sound sources of the input audio signals, a shortest distance between a virtual location of the sound source and a virtual line through a predefined listening location and one of the obtained speaker locations corresponds to the virtualization amount, whereby the shortest distance is substantially zero when the virtualization amount is zero.

2. The method according to claim 1, further comprising:

modifying the virtualization amount in a nonlinear manner, and

wherein adjusting the predetermined location information comprises;

adjusting the predetermined location information based on the modified virtualization amount.

3. The method according to claim 2, wherein performing the surround sound virtualization on the two or more input audio signals comprises:

obtaining a set of coefficients for head related transfer functions (HRTFs) or binaural room impulse responses (BRIRs) corresponding to the adjusted location information; and

processing the two or more input audio signals based on the obtained set of coefficients.

4. The method according to claim 3, wherein obtaining the set of coefficients comprises:

in response to finding a predefined set of coefficients for the HRTFs or for the BRIRs, selecting the predefined set of coefficients as the obtained set of coefficients; and

in response to missing the predefined set of coefficients, determining the obtained set of coefficients by interpolating predefined sets of coefficients for further HRTFs, or for further BRIRs, corresponding to further location information.

5. The method according claim 1, wherein performing the surround sound virtualization further comprises:

determining an effective frequency range for the two or more input audio signals based on the virtualization amount; and

performing the surround sound virtualization on the two or more input audio signals in the determined effective frequency range.

6. The method according to, claim 1 wherein the predefined audio content category includes a category of music, and

wherein determining the virtualization amount comprises:

determining the virtualization amount as a decreasing function of a probability of the set belonging to the category of music.

7. The method according to claim 1 wherein the predefined audio content category includes a category of non-music, and

wherein determining the virtualization amount comprises:

determining the virtualization amount as an increasing function of a probability of the set belonging to the category of non-music.

8. The method according to claim 1 wherein the category of non-music includes at least two of a subcategory of speech, a subcategory of background sound, or a subcategory of noise, and

wherein determining the probability of the set of input audio signals belonging to the predefined audio content category comprises:

determining a probability of the set belonging to each of the at least two subcategories; and

determining the probability of the set belonging to the category of non-music based on the determined probabilities for the at least two subcategories.

9. A computer program product for virtualizing surround sound, comprising a computer program tangibly embodied on a non-transitory medium, the computer program comprising program code for performing the method according to claim 1.

10. A computer system comprising a central processing unit configured to perform the method according to claim 1.