Scalable compressed audio bit stream and codec using a hierarchical filterbank and multichannel joint coding

• US 20070063877A1
• Filed: 06/12/2006
• Published: 03/22/2007
• Est. Priority Date: 06/17/2005
• Status: Active Grant

• Current Assignee: DTS, Inc. (Adeia Inc.)
• Original Assignee: DTS Canada ULC
• Inventors: Beaton, Richard, Shmunk, Dmitry
• Granted Patent: US 7,548,853 B2
• Time in Patent Office: Days
• US Class Current: 341/50
• CPC Class Codes: G10L 19/0204 using subband decomposition
• CPC Class Codes: G10L 19/0212 using orthogonal transformation
• CPC Class Codes: G10L 19/022 Blocking, i.e. grouping of samples in time; Choice of analysis windows; Overlap factoring
• CPC Class Codes: G10L 19/035 Scalar quantisation
• CPC Class Codes: G10L 19/24 Variable rate codecs, e.g. for generating different qualities using a scalable representation such as hierarchical encoding or layered encoding
• CPC Class Codes: G10L 25/18 the extracted parameters being spectral information of each sub-band

Abstract

A method for compressing audio input signals to form a master bit stream that can be scaled to form a scaled bit stream having an arbitrarily prescribed data rate. A hierarchical filterbank decomposes the input signal into a multi-resolution time/frequency representation from which the encoder can efficiently extract both tonal and residual components. The components are ranked and then quantized with reference to the same masking function or different psychoacoustic criteria. The selected tonal components are suitably encoded using differential coding extended to multichannel audio. The time-sample and scale factor components that make up the residual components are encoded using joint channel coding (JCC) extended to multichannel audio. A decoder uses an inverse hierarchical filterbank to reconstruct the audio signals from the tonal and residual components in the scaled bit stream.

Claims

1. A method of encoding an input signal, comprising:

using a hierarchical filterbank (HFB) to decompose an input signal into a multi-resolution time/frequency representation;

extracting tonal components at multiple frequency resolutions from the time/frequency representation;

extracting residual components from the time/frequency representation;

ranking the components based on their relative contribution to decoded signal quality;

quantizing and encoding the components; and

eliminating a sufficient number of the lowest ranked encoded components to form a scaled bit stream having a data rate less than or approximately equal to a desired data rate.

2. The method of claim 1, wherein the components are ranked by first grouping the tonal components into at least one frequency sub-domain at different frequency resolutions and grouping the residual components into at least one residual sub-domain at different time scales and/or frequency resolutions, ranking the sub-domains based on their relative contribution to decoded signal quality and ranking the components within each sub-domain based on their relative contribution to decoded signal quality.

3. The method of claim 2, further comprising:

forming a master bit stream in which the sub-domains and components within each sub-domain are ordered based on their ranking, said low ranking components being eliminated by starting with the lowest ranking component in the lowest ranking sub-domain and eliminating components in order until the desired data rate is achieved.

4. The method of claim 1, further comprising:

forming a master bit stream including the ranked quantized components, wherein the master bit stream is scaled by eliminating a sufficient number of low ranking components to form the scaled bit stream.

5. The method of claim 4, wherein the scaled bit stream is recorded on or transmitted over a channel having the desired data rate as a constraint.

6. The method of claim 5, wherein the scaled bit stream is one of a multiple of scaled bit streams and the data rate of each individual bit stream is controlled independently, with the constraint that the sum of individual data rates must not exceed a maximum total data rate, each said data rate being dynamically controlled in time in accordance with decoded signal quality across all bit streams.

7. The method of claim 1, wherein the residual components are derived from a residual signal between the input signal and the tonal components, whereby tonal components that are eliminated to form the scaled bit stream are also removed from the residual signal.

8. The method of claim 1, wherein the residual components include time-sample components and scale factor components that modify the time-sample components at different time scales and/or frequency resolutions.

9. The method of claim 8, wherein the time-sample components are represented by a grid G and the scale factor components comprise a series of one or more grids G0, G1 at multiple time scales and frequency resolutions that are applied to the time-sample components by dividing the grid G by grid elements of G0, G1 in the time/frequency plane, each grid G0, G1 having a different number of scale factors in time and/or frequency.

10. The method of claim 8, wherein the scale factors are encoded by applying a two-dimensional transform to the scale factor components and quantizing the transform coefficients.

11. The method of claim 10, wherein the transform is a two-dimensional Discrete Cosine Transform.

12. A method of claim 1, wherein the HFB decomposes the input signal into transform coefficients at successively lower frequency resolution levels at successive iterations, wherein said tonal and residual components are extracted by:

extracting tonal components from the transform coefficients at each iteration, quantizing and storing the extracted tonal components in a tone list;

removing the tonal components from the input signal to pass a residual signal to the next iteration of the HFB; and

applying a final inverse transform with relatively lower frequency resolution than the final iteration of the IfFB to the residual signal to extract the residual components.

13. The method of claim 12, further comprising:

removing some of the tonal components from the tone list after the final iteration; and

locally decoding and inverse quantizing the removed quantized tonal components, and combining them with the residual signal at the final iteration.

14. The method of claim 13, wherein at least some of the relatively strong tonal components removed from the list are not locally decoded and recombined.

15. The method of claim 12, wherein the tonal components at each frequency resolution are extracted by:

identifying the desired tonal components through application of a perceptual model;

selecting the most perceptually significant of the transform coefficients;

storing parameters of each selected transform coefficient as the tonal component, said parameters including the amplitude, frequency, phase, and position in the frame of the corresponding transform coefficient; and

quantizing and encoding the parameters for each tonal component in the tone list for insertion into the bit stream.

16. The method of claim 12, wherein the residual components include time-sample components represented as a Grid G, the extraction of the residual components further comprises:

constructing one or more scale-factor grids of different time/frequency resolutions, elements of which represent maximum signal values or signal energies in a time/frequency region;

dividing the elements of time-sample grid G by corresponding elements of the scale-factor grids to produce a scaled time sample grid G; and

quantizing and encoding the scaled time-sample grid G and scale-factor grids for insertion into the encoded bit stream.

17. A method of claim 1, wherein the input signal is decomposed and the tonal and residual components are extracted by,

(a) buffering samples of the input signal into frames of N samples;

(b) multiplying the N samples in each frame by an N-sample window function;

(c) applying an N-point transform to produce N/2 original transform coefficients;

(d) extracting tonal components from the N/2 original transform coefficients, quantizing and storing the extracted tonal components in a tone list;

(e) subtracting the tonal components by inverse quantizing and subtracting the resulting tonal transform coefficients from the original transform coefficients to give N/2 residual transform coefficients;

(f) dividing the N/2 residual transform coefficients into P groups of M_i coefficients, such that the sum of the M_i coefficients is

$N/2<br><br>\left(\underset{i=1}{\overset{P}{\sum <br><br>}}<br><br>M_i=N/2;<br><br>\right)$

(g) for each of P groups, applying a (2*M_i)-point inverse transform to the residual transform coefficients to produce (2*M_i) sub-band samples from each group;

(h) in each sub-band, multiplying the 2*M_i sub-band samples by a 2*M_i point window function;

(i) in each sub-band, overlapping with M_i previous samples and adding corresponding values to produce M_i new samples for each sub-band;

(j) repeating steps (a)-(i) on one or more of the sub-bands of M_i new samples using successively smaller transform sizes N until the desired time/transform resolution is attained; and

(k) Applying a final inverse transform with relatively lower frequency resolution N to the M_i new samples for each sub-band output at the final iteration to produce subbands of time samples in a grid G of sub-bands and multiple time samples in each sub-band.

18. The method of claim 1, wherein the input signal is a multichannel input signal, each said tonal component being jointly encoded by forming groups of said channels and for each said group,

Selecting a primary channel and at least one secondary channel, which are identified through a bitmask with each bit identifying the presence of a secondary channel,

Quantizing and encoding the primary channel;

and

Quantizing and encoding the difference between the primary and each secondary channel.

19. The method of claim 18, wherein a joint channel mode for encoding each channel group is selected based on a metric that indicates which mode provides the least perceived distortion for the desired data rate in the decoded output signal.

20. The method of claim 1, wherein the input signal is a multichannel signal, further comprising:

subtracting the extracted tonal components from the input signal for each channel to form residual signals;

forming the channels of the residual signal into groups determined by perceptual criteria and coding efficiency;

determining primary and secondary channels for each said residual signal group;

calculating a partial grid to encode relative spatial information between each primary/secondary channel pairing in each residual signal group;

quantizing and encoding residual components for the primary channel in each group as respective grids G;

quantizing and encoding the partial grid to reduce the required data rate; and

inserting the encoded partial grid and the grid G for each group into the scaled bit stream.

21. The method of claim 20, wherein the secondary channels are constructed from linear combinations of one or more channels.

22. A method of encoding an audio input signal, comprising:

decomposing an audio input signal into a multi-resolution time/frequency representation;

extracting tonal components at each frequency resolution;

removing the tonal components from the time/frequency representation to form a residual signal;

extracting residual components from the residual signal;

grouping the tonal components into at least one frequency sub-domain;

grouping the residual components into at least one residual sub-domain;

ranking the sub-domains based on psychoacoustic importance;

ranking the components within each sub-domain based on psychoacoustic importance;

quantizing and encoding the components within each sub-domain; and

eliminating a sufficient number of the low ranking components from the lowest ranked sub-domains to form a scaled bit stream having a data rate less than or approximately equal to a desired data rate.

23. The method of claim 22, wherein the tonal components are grouped into a plurality of frequency sub-domains at different frequency resolutions and said residual components include grids that are grouped into a plurality of residual sub-domains at different frequency and/or time resolutions.

24. The method of claim 22, further comprising:

forming a master bit stream in which the sub-domains and components within each sub-domain are ordered based on their ranking, said low ranking components being eliminated by starting with the lowest ranking component in the lowest ranking sub-domain and eliminating components in order until the desired data rate is achieved.

25. A scalable bit stream encoder for encoding an input audio signal and forming a scalable bit stream, comprising:

a hierarchical filterbank (HFB) that decomposes the input audio signal into transform coefficients at successively lower frequency resolution levels and back into time-domain sub-band samples at successively finer time scales at successive iterations;

a tone encoder that (a) extracts tonal components from the transform coefficients at each iteration, quantizes and stores them in a tone list, (b) removes the tonal components from the input audio signal to pass a residual signal to the next iteration of the HFB and (c) ranks all of the extracted tonal components based on their relative contribution to decoded signal quality;

a residual encoder that applies a final inverse transform with relatively lower frequency resolution than the final iteration of the HFB to the final residual signal to extract the residual components and ranks the residual components based on their relative contribution to decoded signal quality;

a bit stream formatter that assembles the tonal and residual components on a frame-by-frame bases to form a master bit stream; and

a scaler that eliminates a sufficient number of the lowest ranked encoded components from each frame of the master bit stream to form a scaled bit stream having a data rate less than or approximately equal to a desired data rate.

26. The encoder of claim 25, wherein the tone encoder groups the tonal components into frequency sub-domains at different frequency resolutions and ranks the components with each sub-domain, the residual encoder groups the residual components into residual sub-domains at different time scales and/or frequency resolutions and ranks the components with each sub-domain, and said bit stream formatter ranks the sub-domains based on their relative contribution to decoded signal quality.

27. The encoder of claim 26, wherein the bit stream formatter orders the sub-domains and the components within each sub-domain based on their ranking, said scaler eliminating said low ranking components being by starting with the lowest ranking component in the lowest ranking sub-domain and eliminating components in order until the desired data rate is achieved.

28. The encoder of claim 25, wherein the input audio signal is a multichannel input audio signal, said tone encoder jointly encoded each said tonal components by forming groups of said channels and for each said group,

selecting a primary channel and at least one secondary channel, which are identified through a bitmask with each bit identifying the presence of a secondary channel;

quantizing and encoding the primary channel; and

quantizing and encoding the difference between the primary and each secondary channel.

29. The encoder of claim 25, wherein the input signal is a multichannel audio signal, said residual encoder,

forming the channels of the residual signal into groups determined by perceptual criteria and coding efficiency;

determining primary and secondary channels for each said residual signal group;

calculating a partial grid to encode relative spatial information between each primary/secondary channel pairing in each residual signal group;

quantizing and encoding residual components for the primary channel in each group as respective grids G;

quantizing and encoding the partial grid to reduce the required data rate; and

inserting the encoded partial grid and the grid G for each group into the scaled bit stream.

30. The encoder of claim 25, wherein the residual encoder extracts time-sample components represented by a grid G and a series of one or more scale factor grids G0, G1 at multiple time and frequency resolutions that are applied to the time-sample components by dividing the grid G by grid elements of G0, G1 in the time/frequency plane, each grid G0, G1 having a different number of scale factors in time and/or frequency.

31. A method of reconstructing a time-domain output signal from an encoded bit stream, comprising:

receiving a scaled bit stream having a predetermined data rate within a given range as a sequence of frames, each frame containing at least one of the following (a) a plurality of quantized tonal components representing frequency domain content at different frequency resolutions of the input signal, b) quantized residual time-sample components representing the time-domain residual formed from the difference between the reconstructed tonal components and the input signal, and c) scale factor grids representing signal energies of the residual signal, which at least partially span a frequency range of the input signal;

receiving information for each frame about the position of the quantized components and/or grids within the frequency range;

parsing the frames of the scaled bit stream into the components and grids;

decoding any tonal components to form transform coefficients;

decoding any time-sample components and any grids;

multiplying the time-sample components by grid elements to form time-domain samples; and

applying an inverse hierarchical filterbank to the transform coefficients and time-domain samples to reconstruct a time-domain output signal.

32. The method of claim 31, wherein the time-domain samples are formed by,

parsing the bit stream into a scale factor Grid Gland the time-sample components;

decoding and inverse quantizing grid G1 scale factor grid to produce a G0 scale factor grid; and

decoding and inverse quantizing the time-sample components, multiplying those time-sample values by G0 scale factor grid values to produce reconstructed time-samples,

33. The method of claim 32, wherein the signal is a multichannel signal in which the residual channels have been grouped and encoded, each said frame also containing d) partial grids representing the signal energy ratios of the residual signal channels within channel groups further comprising:

parsing the bit stream into the partial grids;

decoding and inverse quantizing the partial grids; and

multiplying the reconstructed time-samples by the partial grid applied to each secondary channel in a channel group to produce the reconstructed time-domain samples.

34. The method of claim 31, wherein the input signal is multichannel in which tonal components groups containing a primary and one or more secondary channels, each said frame also containing e) a bitmask associated with the primary channel in each group in which each bit identifies the presence of a secondary channel that has been jointly encoded with the primary channel, parsing the bit stream into the bitmasks;

decoding the tonal components for the primary channel in each group;

decoding the jointly encoded tonal components in each group;

for each group, using the bitmask to reconstruct the tonal components for each said secondary channel from the tonal components of primary channel and the jointly encoded tonal components.

35. The method of claim 34, wherein the secondary channel tonal components are decoded by decoding the difference information between the primary and secondary frequencies, amplitudes and phases being entropy-coded and stored for each secondary channel in which the tonal component is present.

36. The method of claim 31, wherein the inverse hierarchical filterbank reconstructs the output signal by transforming the time-domain samples into residual transform coefficients, combining them with the transform coefficients for a set of tonal components at a low frequency resolution and inverse transforming the combined transform coefficients to form a partially reconstructed output signal, and repeating the steps on this partially reconstructed output signal with the transform coefficients for another set of tonal components at the next highest frequency resolution until the output signal is reconstructed.

37. The method of claim 36, wherein the time-domain samples are represented as sub-bands, said inverse hierarchical filterbank reconstructing the time-domain output signal by:

a) windowing the signal(s) in each of the time-domain sub-bands of the input frame to form windowed time-domain sub-bands;

b) applying a time-to-frequency domain transform to each of the windowed time-domain sub-bands to form transform coefficients;

c) concatenating the resulting transform coefficients to form larger set(s) of the residual transform coefficients;

d) synthesizing the transform coefficients from the set of tonal components;

e) combining the transform coefficients reconstructed from the tonal and time-domain components into a single set of combined transform coefficients;

t) applying an inverse transform to the combined transform coefficients, windowing and overlap adding with the previous frame to reconstruct a partially reconstructed time domain signal; and

g) applying successive iterations of steps (a) to (f) on the partially reconstructed time domain signal(s) using the next set of tonal components until the time-domain output signal is reconstructed.

38. The method of claim 36, in which each input frame contains M_i time samples in each of P sub-bands, said inverse hierarchical filterbank performing the following steps:

a) in each sub-band i, buffering and concatenated the M_i previous samples with the current M_i samples to produce 2*M_i new samples;

b) in each sub-band i, multiplying the 2*M_i sub-band samples by a 2*M_i point window function;

c) applying a (2*M_i)-point transform to the sub-band samples to produce M1 transform coefficients for each sub-band i;

d) concatenating the M1 transform coefficients for each sub-band i to form a single set of N/2 coefficients;

e) synthesizing tonal transform coefficients from the decoded and inverse quantized set of tonal components and combining them with the concatenated coefficients of the previous step to form a single set of combined concatenated coefficients;

f) applying an N-point inverse transform to the combined concatenated coefficients to produce N samples;

g) multiplying each Frame of N samples by an N-sample window function to produce N windowed samples;

h) overlap adding the resulting windowed samples to produce N/2 new output samples at the given sub-band level as the partially reconstructed output signal; and

i) repeating steps (a)-(h) on the N/2 new output samples using the next set of tonal components until all sub-bands have been processed and the N original time samples are reconstructed as the output signal.

39. A decoder for reconstructing a time-domain output audio signal from an encoded bit stream, comprising:

a bit stream parser for parsing each frame of a scaled bit stream into its audio components, each frame containing at least one of the following (a) a plurality of quantized tonal components representing frequency domain content at different frequency resolutions of the input signal, b) quantized residual time-sample components representing the time-domain residual formed from the difference between the reconstructed tonal components and the input signal, and c) scale factor grids representing the signal energies of the residual signal;

a residual decoder for decoding any time-sample components and any grids to reconstruct time samples;

a tonal decoder for decoding any tonal components to form transform coefficients; and

an inverse hierarchical filterbank that reconstructs the output signal by transforming the time samples into residual transform coefficients, combining them with the transform coefficients for a set of the tonal components at a low frequency resolution and inverse transforming the combined transform coefficients to form a partially reconstructed output signal, and repeating the steps on this partially reconstructed output signal with the transform coefficients for another set of tonal components at the next highest frequency resolution until the output audio signal is reconstructed.

40. The decoder of claim 39, wherein each input frame contains M_i time samples in each of P sub-bands, said inverse hierarchical filterbank performing the following steps:

a) in each sub-band i, buffering and concatenated the M_i previous samples with the current M_i samples to produce 2*M_i new samples;

b) in each sub-band i, multiplying the 2*M_i sub-band samples by a 2*M_i point window function;

c) applying a (2*M_i)-point transform to the sub-band samples to produce M residual transform coefficients for each sub-band i;

d) concatenating the M_i residual transform coefficients for each sub-band i to form a single set of N/2 coefficients;

e) synthesizing tonal transform coefficients from the decoded and inverse quantized set of tonal components and combining them with the concatenated residual transform coefficients to form a single set of combined concatenated coefficients;

f) applying an N-point inverse transform to the combined concatenated coefficients to produce N samples;

g) multiplying each Frame of N samples by an N-sample window function to produce N windowed samples;

h) overlap adding the resulting windowed samples to produce N/2 new output samples at the given sub-band level as the partially reconstructed output signal; and

i) repeating steps (a)-(h) on the N/2 new output samples using the next set of tonal components until all sub-bands have been processed and the N original time samples are reconstructed as the output signal.

41. A method of hierarchically filtering an input signal to achieve a nearly arbitrary time/frequency decomposition, comprising the steps of:

(a) buffering samples of the input signal into frames of N samples;

(b) multiplying the N samples in each frame by an N-sample window function;

(c) applying an N-point transform to produce N/2 transform coefficients;

(d) dividing the N/2 residual transform coefficients into P groups of M_i coefficients, such that the sum of the M_i coefficients is

$N/2<br><br>\left(\underset{i=1}{\overset{P}{\sum <br><br>}}<br><br>M_i=N/2;<br><br>\right)$

(e) for each of P groups, applying a (2*M_i)-point inverse transform to the transform coefficients to produce (2*M_i) sub-band samples from each group;

(f) in each sub-band i, multiplying the (2*M_i) sub-band samples by a (2*M_i)-point window function;

(g) in each sub-band i, overlapping with M_i previous samples and adding corresponding values to produce M_i new samples for each sub-band; and

(h) repeating steps (a)-(g) on one or more of the sub-bands of M_i new samples using successively smaller transform sizes N until the desired time/transform resolution is achieved.

42. The method of claim 41, wherein the transform is an MDCT transform.

43. The method of claim 41, wherein steps (a)-(g) are repeated on all of the sub-bands of M_i.

44. The method of claim 41, wherein steps (a)-(g) are repeated on only a defined set of low frequency sub-bands of M_i.

45. A method of hierarchically reconstructing time samples of an input signal, in which each input frame contains M_i time samples in each of P sub-bands, comprising performing the following steps:

a) in each sub-band i, buffering and concatenating the M_i previous samples with the current M_i samples to produce 2*M_i new samples;

b) in each sub-band i, multiplying the 2*M1 sub-band samples by a 2*M_i point window function;

c) applying a (2*M_i)-point transform to the windowed sub-band samples to produce M_i transform coefficients for each sub-band i;

d) concatenating the M_i transform coefficients for each sub-band i to form a single group of N/2 coefficients;

e) applying an N-point inverse transform to the concatenated coefficients to produce a frame of N samples;

f) multiplying each frame of N samples by an N-sample window function to produce N windowed samples;

g) overlap adding the resulting windowed samples to produce N/2 new output samples at the given sub-band level; and

h) repeating steps (a) through (g) until all sub-bands have been processed and the N original time samples are reconstructed.