Immersive audio rendering system

• US 9,088,858 B2
• Filed: 01/03/2012
• Issued: 07/21/2015
• Est. Priority Date: 01/04/2011
• Status: Active Grant

• Current Assignee: DTS, Inc. (Adeia Inc.)
• Original Assignee: DTS, Inc. (Adeia Inc.)
• Inventors: Kraemer, Alan D., Tracey, James, Katsianos, Themis
• Primary Examiner(s): Lee, Ping
• Application Number:
  US13/342,743
• Publication Number:
  US 20120170756A1
• Time in Patent Office: 1,295 Days
• Field of Search: 381/17, 381/18, 381/306, 381/307, 381/310, 381/303
• US Class Current: 1/1
• CPC Class Codes: H04S 1/002 Non-adaptive circuits, e.g. manually adjustable or static, for enhancing the sound image or the spatial distribution control circuits for electronic adaptation of the sound field H04S7/30
• CPC Class Codes: H04S 2400/01 Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
• CPC Class Codes: H04S 2400/07 Generation or adaptation of the Low Frequency Effect [LFE] channel, e.g. distribution or signal processing
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 7/30 Control circuits for electronic adaptation of the sound field
• CPC Class Codes: H04S 7/302 Electronic adaptation of stereophonic sound system to listener position or orientation H04S7/301 takes precedence

Abstract

A depth processing system can employ stereo speakers to achieve immersive effects. The depth processing system can advantageously manipulate phase and/or amplitude information to render audio along a listener'"'"'s median plane, thereby rendering audio along varying depths. In one embodiment, the depth processing system analyzes left and right stereo input signals to infer depth, which may change over time. The depth processing system can then vary the phase and/or amplitude decorrelation between the audio signals over time to enhance the sense of depth already present in the audio signals, thereby creating an immersive depth effect.

Claims

1. A method of rendering depth in an audio output signal, the method comprising:

receiving a plurality of audio signals;

identifying first depth steering information from the audio signals at a first time, the first depth steering information responsive to a first decorrelation of the audio signals;

applying nonlinear processing to the first depth steering information to produce second depth steering information, the nonlinear processing configured to accentuate a magnitude of the first depth steering information with a greater than linear increase that nonlinearly emphasizes the first decorrelation of the audio signals such that relatively faster changes in the magnitude of the first decorrelation are emphasized more than relatively slower changes in the magnitude of the first decorrelation;

identifying subsequent depth steering information from the audio signals at a second time;

decorrelating, by one or more processors, the plurality of audio signals by a first amount that depends at least partly on the second depth steering information to produce first decorrelated audio signals, wherein said decorrelating comprises applying greater decorrelation responsive to the second depth steering information being relatively higher and applying less decorrelation responsive to the second depth steering information being relatively lower;

outputting the first decorrelated audio signals for playback to a listener;

subsequent to said outputting, decorrelating the plurality of audio signals by a second amount different from the first amount, the second amount depending at least partly on the subsequent depth steering information to produce second decorrelated audio signals; and

outputting the second decorrelated audio signals for playback to the listener.

2. The method of claim 1, wherein said decorrelating the plurality of audio signals by a first amount comprises dynamically adjusting one or both of a delay and a gain applied to the plurality of audio signals.

3. The method of claim 1, further comprising processing the first and second decorrelated audio signals with a surround enhancement to widen a sound image of the first and second decorrelated audio signals.

4. The method of claim 3, further comprising modulating an amount of the surround enhancement applied to the first and second decorrelated audio signals based at least in part on the second and subsequent depth steering information.

5. The method of claim 4, further comprising reducing backwave crosstalk in the first and second decorrelated audio signals.

6. A method of rendering depth in an audio output signal, the method comprising:

receiving a plurality of audio signals;

identifying first depth steering information associated with the audio signals, the first depth steering information changing over time;

applying nonlinear processing to the first depth steering information to produce second depth steering information, the nonlinear processing configured to accentuate a magnitude of the first depth steering information with a greater than linear increase that nonlinearly emphasizes the first decorrelation of the audio signals such that relatively faster changes in the magnitude of the first decorrelation are emphasized more than relatively slower changes in the magnitude of the first decorrelation;

decorrelating the plurality of audio signals dynamically over time by an amount that depends on the second depth steering information, such that a greater existing depth in the audio signals is emphasized relatively more and a lower existing depth in the audio signals is emphasized relatively less, to produce a plurality of decorrelated audio signals; and

outputting the plurality of decorrelated audio signals for playback to a listener;

wherein at least said decorrelating is performed at least by electronic hardware.

7. The method of claim 6, wherein the plurality of audio signals comprise a left audio signal and a right audio signal.

8. The method of claim 7, wherein said identifying the first depth steering information comprises estimating the depth in the audio signals based at least partly on difference information between the left and right audio signals.

9. The method of claim 6, wherein said identifying the first depth steering information comprises estimating the depth in the audio signals based at least partly on video information associated with a video corresponding to the plurality of audio signals.

10. The method of claim 6, wherein the audio signals comprise object metadata comprising position information associated with audio objects.

11. The method of claim 10, wherein said identifying the first depth steering information comprises converting the position information of the audio objects into the depth steering information.

12. The method of claim 6, wherein said decorrelating the audio signals comprises introducing a dynamically changing delay into one or more of the audio signals, wherein the delay changes based on the first depth steering information.

13. The method of claim 12, wherein said decorrelating comprises increasing a first delay of a first one of the audio signals while simultaneously decreasing a second delay of a second one of the audio signals.

14. The method of claim 6, wherein said decorrelating the audio signals comprises applying a dynamically changing gain to one or more of the audio signals, wherein the gain changes based on the first depth steering signal.

15. The method of claim 14, wherein said decorrelating comprises increasing a first gain of a first one of the audio signals while simultaneously decreasing a second gain of a second one of the audio signals.

16. A system for rendering depth in an audio output signal, the system comprising:

a depth estimator configured to;

receive two or more audio signals and to identify depth information associated with the two or more audio signals, and

apply nonlinear processing to the depth information to produce nonlinear depth information, the nonlinear processing configured to accentuate a magnitude of the depth information with a greater than linear increase; and

a depth renderer comprising one or more processors, the depth renderer configured to decorrelate the two or more audio signals by an amount that depends on the nonlinear depth information, such that a greater existing depth in the two or more audio signals is emphasized relatively more and a lower existing depth in the two or more audio signals is emphasized relatively less, to produce a plurality of decorrelated audio signals, and output the plurality of decorrelated audio signals.

17. The system of claim 16, wherein the depth estimator is further configured to identify depth information from normalized difference information associated with the two or more audio signals.

18. The system of claim 16, wherein the depth estimator is further configured to identify depth information based at least partly on determining which of the two or more audio signals is dominant.

19. The system of claim 16, wherein the two or more audio signals comprise a front left audio signal, a front right audio signal, a left surround audio signal, and a right surround audio signal.

20. The system of claim 19, wherein the depth renderer produces the plurality of decorrelated audio signals by at least decorrelating the front left audio signal and the front right audio signal and separately decorrelating the left surround audio signal and the right surround audio signal.

21. The system of claim 16, wherein the depth renderer applies a depth rendering filter to the two or more audio signals, the depth rendering filter comprising a feed-forward component and a feedback component, and wherein the feedback component is configured to reduce a comb filter effect generated by the feed-forward component.

22. The system of claim 21, wherein the feedback component is further configured to eliminate the comb filter effect generated by the feed-forward component.

23. A method of rendering depth in an audio output signal, the method comprising:

receiving input audio comprising two or more audio signals;

estimating depth information associated with the input audio, the depth information changing over time, said estimating the depth information comprising calculating an amount of existing decorrelation between the two or more audio signals;

emphasizing the depth information to produce nonlinear depth information by at least nonlinearly accentuating a magnitude of the first depth stcering information with a greater than linear increase;

enhancing the audio dynamically based on the nonlinear depth information, by one or more processors, said enhancing varying dynamically based on variations in the nonlinear depth information over time,

said enhancing comprising emphasizing the existing decorrelation between the two or more audio signals based in part on the amount of existing decorrelation; and

outputting the enhanced audio.