Metadata driven dynamic range control
1. A method for dynamic range control of an audio signal, the method comprising:
- a. receiving a bitstream representing a piece of sound program content having a plurality of channels, wherein the bitstream has inserted therein Dynamic Range Control, DRC, gain metadata for each DRC group of a set of DRC groups, wherein each channel of the sound program content is grouped into a single DRC group from the set of DRC groups; and
b. using the DRC gain metadata for each DRC group to apply corresponding DRC gain values to frames in the DRC group for playback of the piece of sound program content by an audio playback device,wherein the DRC gain metadata for each DRC group includes;
a1. a first data value representing a selected coding mode for an initial DRC gain value;
b1. a second data value representing the initial DRC gain value; and
c1. a third data value representing differences applied to the initial DRC gain value to generate the DRC gain values for each frame of the DRC group.
A system for encoding and applying Dynamic Range Control/Compression (DRC) gain values to a piece of sound program content is described. In particular, a set of DRC gain values representing a DRC gain curve for the piece of content may be divided into frames corresponding to frames of the piece of content. A set of fields may be included with an audio signal representing the piece of content. The additional fields may represent the DRC gain values using linear or spline interpolation. The additional fields may include 1) an initial gain value for each DRC frame, 2) a set of slope values at particular points in the DRC curve, 3) a set of time delta values for each consecutive pair of slope values, and/or 4) one or more gain delta values representing changes of DRC gain values in the DRC gain curve between points of the slope values.
|TRANSCODING OF AUDIO METADATA|
Patent #US 20100083344A1
Current AssigneeDolby International AB
Sponsoring EntityDolby International AB
|APPARATUS FOR PROCESSING MIX SIGNAL AND METHOD THEREOF|
Patent #US 20100040135A1
Current AssigneeLG Electronics Inc.
Sponsoring EntityLG Electronics Inc.
|Apparatus and Method for Transmitting a Sequence of Data Packets and Decoder and Apparatus for Decoding a Sequence of Data Packets|
Patent #US 20100284281A1
Current AssigneeFraunhofer-Gesellschaft zur Forderung der angewandten Forschung e.V.
Sponsoring EntityFraunhofer-Gesellschaft zur Forderung der angewandten Forschung e.V.
|Method for Correcting Metadata Affecting the Playback Loudness of Audio Information|
Patent #US 20100250258A1
Current AssigneeDolby Laboratories Incorporated
Sponsoring EntityDolby Laboratories Incorporated
|APPARATUS FOR PROCESSING AN AUDIO SIGNAL AND METHOD THEREOF|
Patent #US 20100310085A1
Current AssigneeLG Electronics Inc.
Sponsoring EntityLG Electronics Inc.
|Selectively using multiple entropy models in adaptive coding and decoding|
Patent #US 20070016418A1
Current AssigneeMicrosoft Technology Licensing LLC
Sponsoring EntityMicrosoft Corporation
|Method and device for gain quantization in variable bit rate wideband speech coding|
Patent #US 20050251387A1
Current AssigneeNokia Technologies Oy
Sponsoring EntityNokia Corporation
|Method for independent dynamic range control|
Patent #US 6,782,366 B1
Current AssigneeAvago Technologies International Sales Pte Limited
Sponsoring EntityLSI Logic Corporation
|Channel reconfiguration with side information|
Patent #US 8,280,743 B2
Current AssigneeDolby Laboratories Incorporated
Sponsoring EntityDolby Laboratories Incorporated
|System and Method for Non-destructively Normalizing Loudness of Audio Signals Within Portable Devices|
Patent #US 20120310654A1
Current AssigneeDolby International AB, Dolby Laboratories Incorporated
Sponsoring EntityDolby International AB, Dolby Laboratories Incorporated
|METHODS AND SYSTEMS FOR DETERMINING AUDIO LOUDNESS LEVELS IN PROGRAMMING|
Patent #US 20130156229A1
Current AssigneeTime Warner Cable Enterprises LLC
Sponsoring EntityTime Warner Cable Enterprises LLC
|Metadata driven dynamic range control|
Patent #US 9,607,624 B2
Current AssigneeApple Inc.
Sponsoring EntityApple Inc.
- 1. A method for dynamic range control of an audio signal, the method comprising:
a. receiving a bitstream representing a piece of sound program content having a plurality of channels, wherein the bitstream has inserted therein Dynamic Range Control, DRC, gain metadata for each DRC group of a set of DRC groups, wherein each channel of the sound program content is grouped into a single DRC group from the set of DRC groups; and b. using the DRC gain metadata for each DRC group to apply corresponding DRC gain values to frames in the DRC group for playback of the piece of sound program content by an audio playback device, wherein the DRC gain metadata for each DRC group includes; a1. a first data value representing a selected coding mode for an initial DRC gain value; b1. a second data value representing the initial DRC gain value; and c1. a third data value representing differences applied to the initial DRC gain value to generate the DRC gain values for each frame of the DRC group.
- View Dependent Claims (2, 3, 4, 5, 6, 7, 8)
- 9. An audio system for dynamic range control of an audio signal, comprising:
a processor; and memory having stored therein instructions that when executed by the processor a. receive a bitstream representing a piece of sound program content having a plurality of channels, wherein the bitstream has inserted therein Dynamic Range Control, DRC, gain metadata for each DRC group of a set of DRC groups, wherein each channel of the sound program content is grouped into a single DRC group from the set of DRC groups, and wherein the DRC gain metadata for each DRC group includes; a1. a first data value representing a selected coding mode for an initial DRC gain value; b1. a second data value representing the initial DRC gain value; and
- View Dependent Claims (10, 11, 12, 13, 14, 15, 16, 17)
This application is a divisional of co-pending U.S. application Ser. No. 14/226,596 filed on Mar. 26, 2014, which claims the benefit of the earlier filing dates of U.S. provisional application No. 61/806,628, filed Mar. 29, 2013; U.S. provisional application No. 61/857,966 filed Jul. 24, 2013; and U.S. provisional application No. 61/891,687 filed Oct. 16, 2013.
An embodiment of the invention generally relates to a system and method for encoding and applying Dynamic Range Control/Compression (DRC) to an audio signal. Furthermore, the system and method described herein takes into account the DRC requirements of new codecs under development in MPEG-H (3D Audio). Other embodiments are also described.
Dynamic Range Control/Compression (DRC) reduces the dynamic range of an audio signal by some amount by (1) making soft parts in the audio signal louder; (2) making loud parts in the audio signal softer; or (3) both making soft parts louder and making loud parts softer. A reduced dynamic range may be desirable in several situations, including for audio playback systems that can only reproduce a small dynamic range while maintaining low distortions, listening environments with distracting sounds, and in situations where the listener does not want to distract others.
Although, DRC is an important feature for today'"'"'s audio codecs, several recent audio codecs do not support DRC. For example, DRC is absent in the Unified Speech and Audio Coding (USAC) standard by the Moving Picture Experts Group (MPEG). Advanced Audio Coding (AAC) incorporates a DRC tool, but this DRC tool has drawbacks, including limited time resolution and aliasing distortions.
A system and method for encoding and applying Dynamic Range Control/Compression (DRC) gain values to a piece of sound program content is described. In one embodiment, a set of DRC gain values representing a DRC gain curve for the piece of sound program content may be divided into frames corresponding to frames of the piece of sound program content. An additional field or set of fields may be included with an audio signal representing the piece of sound program content. The additional fields may represent the DRC gain values using linear or spline interpolation. In one embodiment, the additional fields may include 1) an initial gain value for each DRC frame, 2) a set of slope values at particular points in the DRC curve, 3) a set of time delta values for each consecutive pair of slope values, and 4) one or more gain delta values representing changes of DRC gain values in the DRC gain curve between points corresponding to the slope values. As described, the system and method herein provides an efficient technique for encoding and applying DRC gain values for a piece of sound program content.
The above summary does not include an exhaustive list of all aspects of the present invention. It is contemplated that the invention includes all systems and methods that can be practiced from all suitable combinations of the various aspects summarized above, as well as those disclosed in the Detailed Description below and particularly pointed out in the claims filed with the application. Such combinations have particular advantages not specifically recited in the above summary.
The embodiments of the invention are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings in which like references indicate similar elements. It should be noted that references to “an” or “one” embodiment of the invention in this disclosure are not necessarily to the same embodiment, and they mean at least one.
Several embodiments of the invention with reference to the appended drawings are now explained. Whenever the shapes, relative positions and other aspects of the parts described in the embodiments are not clearly defined, the scope of the invention is not limited only to the parts shown, which are meant merely for the purpose of illustration. Also, while numerous details are set forth, it is understood that some embodiments of the invention may be practiced without these details. In other instances, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the understanding of this description.
Metadata systems which incorporate Dynamic Range Control/Compression (DRC) metadata into the bitstream/format provide several advantages over systems that determine DRC gain values at the listener'"'"'s end (i.e., at playback). These advantages include (1) lower complexity at playback of the audio signal; (2) complexity of DRC is less of an issue during playback, which allows more complex DRC procedures to be implemented; and (3) an audio playback device at the listener'"'"'s end may decide whether to apply DRC. Although using DRC metadata systems offer several advantages, traditional DRC metadata systems, such as those provided by the Advanced Television Systems Committee (ATSC) and the Moving Picture Experts Group'"'"'s (MPEG) also provide several disadvantages.
Traditional DRC metadata systems (e.g., those defined by ATSC and MPEG standards) support light and heavy compression as shown in Table 1. In most cases, the rate of DRC gain value updates is one per frame. At a sample rate of 48 kHz, this is equivalent to an update interval between 21 and 43 ms. AC-3 in light compression mode has a six-times faster rate, at approximately 5 ms at 48 kHz. Moreover, the DRC gain values in these traditional DRC metadata systems are updated at lower rates for lower audio sample rates.
Actual DRC tuning suggests that gain changes should be much faster for certain audio signals than what can be achieved with current standards.
Another problem with current DRC standards and systems, such as MPEG-AAC and ATSC, arises from the fact that the DRC gain is applied in the frequency domain before an inverse MDCT filter bank is applied. The MDCT filter bank is a transform based on time-domain aliasing cancelation. Aliasing cancelation cannot be achieved if different gain values are applied to consecutive overlapping blocks. A gain change can result in audible distortions such as pre-echoes. This can easily be shown for a castanet recording.
In one embodiment, MDCT artifacts may be avoided if the DRC gain is applied in the time domain after the audio signal is reconstructed by the decoder. In the frequency domain the gain may be modified at most once per long or short block. In contrast, the time-domain approach described herein supports the desired higher time resolution.
Time domain approaches currently do not support multi-band DRC (available by MPEG light compression), but embodiments discussed herein may be enhanced to support multi-band DRC. The proposed scheme may slightly increase decoder complexity due to interpolation and application of DRC gains. However, these drawbacks appear irrelevant given that unnecessary distortions can be avoided especially with content that may come at a high bit rate and may be reproduced with a high quality playback system.
The DRC tool described herein is based on a unified DRC gain encoding that may be applied to a time domain or sub-band domain audio signal such as the sub-bands of the QMF filter bank of an HE-AAC decoder. The following description covers the time-domain application first. For the sub-band domain application only modifications to the time-domain approach are described.
This section describes how the dynamic compression tool is applied to a time-domain audio signal after decoding as shown in
Since the encoder provides only sparsely sampled gain values, the decoder may apply interpolation to achieve a smooth gain transition between the samples. The sample rate of the interpolated gain is the audio sample rate. The interpolation technique used may be based on splines. The interpolated values of one segment between two subsequent gain samples are derived from the two gain samples at both ends of the segment and their slope (derivative). Hence, when transitioning from one segment to the next, the first derivative is continuous as both segments have the same slope at the transition point.
In one embodiment, the smallest possible time interval for sampling the gain curve is a fixed value between 0.5 and 1.0 ms and the largest possible time interval is one gain sample per DRC frame.
In addition to the spline mode described above, a “simple” mode may be used to transmit just one DRC gain value per DRC frame without timing and slope parameters. This mode is most suitable for frames with virtually constant DRC gain and consumes the lowest number of bits.
For applications of the DRC tool in tandem with an audio codec, the following parameters are provided to adjust the DRC frame size and time resolution so that codec and DRC processing can be done most efficiently in terms of complexity and delay. The parameters are:
- DRC frame size in units of the audio sample interval
- delta_t_min in units of the audio sample interval
- delay mode
These parameters have default values, but a codec specification may overwrite the defaults.
Modifying the DRC Characteristic
The DRC tool supports modifications of the decoded DRC gain by several means:
- Boost factor
- Compress factor
- Custom DRC characteristic
The boost factor is a value between 0 and 1 that is applied to positive gain values in dB to reduce amplification. The compress factor is a value between 0 and 1 applied to negative gain values to reduce attenuation.
The DRC configuration that includes the encoder DRC may be referred to hereinafter as the “Sample Description”. For example, the first six static DRC characteristics are shown in
An example for the DRC metadata generation at the transmitter is shown in
The receiver can modify the static DRC characteristic based on the transmitter'"'"'s DRC characteristic as conveyed in the Sample Description and based on a custom target DRC characteristic. Starting with the received DRC gain value (gainQuant), the receiver can apply the inverse transmitter DRC characteristic and then apply a new target DRC characteristic as shown in Table 2:
The inverse of the transmitter characteristics 1 to 6 may be computed according to Table 3 and Table 4. Please note that characteristic 2 does not have a useful inverse because the gain is always 0 dB.
Decoder DRC target characteristics are not considered to be standardized. They can be optionally defined by an implementer to achieve customized compression characteristics. Sections below explain how the gain mapping is applied in more detail.
The Sample Description may include a total of 11 encoder DRC characteristics. For compatibility with existing systems the Sample Description contains in addition to the first 6 characteristics described above and shown in
Interpolation of the DRC gain in the decoder is based on pairs of gain samples. Each pair has gain coordinates (time and value in dB) and slope information. The decoder will choose one of three available types of interpolation as illustrated in
The DRC gain information is organized in DRC frames. Each DRC frame contains DRC data to generate the DRC gain for the duration of a DRC frame. The DRC frame duration is constant for a given audio item and it is a multiple of the audio sample interval. DRC frames do not overlap. In practice, whenever suitable, the DRC frame size is recommended to be identical to the codec'"'"'s frame size to minimize delay and complexity. This may be the default setting.
The DRC tool uses a uniform time grid to generate a sparse representation of the DRC gain. The spacing of this grid defines the highest available time resolution delta_t_min. The unit of delta_t_min is one sample interval at the audio sample rate. For complexity reasons, delta_t_min is chosen to be an integer multiple of the audio sample interval with a corresponding duration between [0.5 . . . 1.0] ms. Preferably, delta_t_min is an integer power of 2, so that sample rates can be efficiently converted between audio and DRC. The default values are computed based on the following equation:
delta_t_min=2M where fs0.0005 s<delta_t_min≤fs0.001 s (1)
In the above equation, the audio sample rate fs is in Hz, and the exponent M is a non-negative integer.
Look-Ahead in Decoder
The DRC tool decoder can be operated in one of two delay modes. The low-delay mode immediately applies the decoded DRC gain while the default mode applies the DRC gain with a delay of one DRC frame. The default mode supports gain sample interpolation from any position of the current DRC frame to any position of the next DRC frame. The low-delay mode requires that a gain value sample is located at the end of the DRC frame.
For common perceptual codecs the default delay mode B will not require additional decoder delay. The delay is already required due to the overlap add operation.
The low-delay mode may be suitable for decoders that don'"'"'t have inherent delay such as a delay due to overlap add. For instance, this is the case for some lossless codecs.
The decoding process of the gain coordinates and slopes consists of the following sequence of tasks:
- Gather the DRC configuration information
- Parse the DRC bitstream
- Apply the code tables including Huffman decoding to decode the quantized values
- Undo the differential encoding
The DRC configuration information may be part of the Sample Description. The DRC configuration information may include the following parameters relevant for decoding:
- The number of gain sequences: nDrcGainSequences
- The assignment of a gain sequence to each channel. Channels using the same sequence are referred to as channel groups. The total number of groups is nDrcChannelGroups
- The number of DRC bands in a group: nDrcBands
Given these parameters, the DRC bitstream can be parsed according to Table 20 and Table 21. In the following, the pseudo code is limited to one gain sequence for clarity. For the general case, an outer loop may be added to process each gain sequence in Table 6 and Table 9.
The coded values are decoded by applying Table 22 and Table 25. This operation is expressed in Table 6 by the pseudo-functions decode_initial_gain( ) decode_delta_gain( ), decode_time_delta( ), and decode_slope( ). Differentially encoded values are then converted into absolute values according to Table 6. The decoded result is represented by the gain values gDRC[g][b][k], the time values tDRC[g][b][k], and the slope values sDRC[g][b][k] where g is the channel group index, b is the band index, and k is the spline node index. Time values are integer numbers relative to the beginning of the DRC frame in units of delta_t_min. The audio sample that coincides with the beginning of the DRC frame has a time value of tDRC=0.
Gain Modifications and Interpolation
As mentioned above under the heading “Modifying the DRC Characteristic”, there are several ways to adapt the DRC characteristics in the DRC tool decoder. These adjustments are applied to the decoded gain samples in the dB domain.
The function toLinear( ) is introduced in Table 7 to include all necessary steps to generate a linear gain sample from the logarithmic value in dB (see Table 7). This function contains an optional mapping function mapGain( ) (see Table 2) that supports modifications of the DRC gain values with the purpose of achieving a different compression characteristic than the one used in the encoder. The mapping is controlled by the index characteristicIndex that will select one of the custom decoder DRC characteristics if it is larger than 0. Otherwise, the encoder characteristic will not be replaced. A modified characteristic can be generated based on the encoder compression characteristic that is conveyed in the Sample Description. Moreover, a compression and boost factor is supported to scale negative and positive gains, respectively. These factors have a value of 1.0, unless values in the range [0,1] are supplied by the user. Finally, the loudness normalization gain is applied.
Before the gain can be applied to the audio signal, the audio signal must be converted to the linear domain and gain values between gain samples must be interpolated. To achieve lower complexity, the dB to linear conversion may be done before the interpolation. Hence, the interpolation process is entirely done in the linear domain. Both, the gain modification and conversion to the linear domain are done using the pseudo code of Table 7. The input variables are the gain samples and slopes in the dB domain. The output consists of the gain samples and slopes in the linear domain. For loudness normalization, a loudness normalization gain value in dB (loudnessNormalizationGainsDb) can be supplied to the decoder by a loudness control tool or other means. If not supplied, a default value of 0.0 is used. In one embodiment, the normalization gain is calculated as the difference between the target loudness and the content loudness in dB FS. The target loudness is the desired output loudness level. The content loudness is equal to the program loudness or anchor loudness as defined in ISO/MPEG, “14496-12 PDAM 3—Enhanced Audio (File Format)”, 106th MPEG meeting Geneva, Switzerland, October 2013. If both program loudness and anchor loudness are not supplied, a default value may be used for content loudness.
The gain interpolation is implemented by the pseudo code in Table 8. The input variables are:
- the time difference between the two gain samples in units of the target sample rate interval tGainStep
- a pair of subsequent gain samples gain0 and gain1 in dB
- a pair of corresponding slope steepness values slope0 and slope1 in the dB domain.
This function uses toLinear( ) to convert the variables to the linear domain. The result is a smooth sequence of gain values at the target sample rate located between the pair of gain samples. The target sample rate is the sample rate of the compressed audio signal.
Applying the Compression
The interpolated gain values of each spline segment are concatenated to generate a complete gain vector gain[g][b][t] for the entire DRC frame. Finally, the gain vector is applied as shown in Table 9. The function channelInDrcGroup( ) returns TRUE if the current channel c belongs to the current DRC channel group as specified in the Sample Description. Please note that the scheduling of the spline segments depends on the delay mode (see the section above labeled “Look-ahead in Decoder”) as indicated in Table 9.
Table 9 is based on the following assumptions:
- splineSegment is a vector that contains the gain values of one spline segment.
- duration is an integer number describing the duration of the spline segment in units of audio sample intervals.
- nNodes is the number of gain values in the current DRC frame.
- drcFrameSize is the number of audio sample intervals in a DRC frame.
- Initialize the following variables if delayMode==DELAY_MODE_DEFAULT:
- gDRCprev[g][b]=0.0, sDRCprev[g][b]=0.0;
- tDRCprev[g][b]=drcFrameSize; nNodesPrev[g][b]=1.
Multi-Band DRC Filter Bank
When the DRC gains are applied in the time domain and a multi-band DRC is used, the time-domain audio signal must be split into sub-bands before the DRC gain is applied to the bands. The filter configuration parameters may be conveyed by the DRCInstructions( ) defined in the MPEG File Format. The MPEG File Format may provide the bitstream syntax for the number of bands and the crossover frequency indices between bands.
The time-domain audio signal is split into the specified number of bands by Linkwitz-Riley (LR) filters with a topology shown in
As shown in
Each BW filter and each all-pass filter is implemented as second order section with the following transfer function.
Based on the crossover frequency indices in Table 26, the decoder can look up the normalized crossover frequencies fc,Norm or the filter coefficient parameters γ and δ. The filter coefficients are then computed using Table 10 for the BW filters and Table 11 for the all-pass filters. The crossover frequencies fc in Hz are computed by:
In case of multi-rate decoder configurations such as dual-rate HE-AAC, fs is the sample rate of the final output signal.
The all-pass filters in
After the DRC gains are applied to the individual bands, the final audio signal is computed by adding all bands.
DRC Applied to Decoder'"'"'s Sub-Band Domain
While the application of DRC gains in the time-domain is mandatory for AAC, other MPEG codecs use sub-band domain DRC. The concept of sub-band domain DRC means that the existing sub-band signals of the decoder are subject to the DRC gain application. Therefore, it is not necessary to add time-domain band splitting for a multi-band DRC and it is possible to apply DRC gains before rendering and/or downmixing in the frequency domain. Table 12 contains a non-exhaustive list of codecs and the domain where the DRC gain is applied. The domain may depend on the decoder configuration and not on the bitstream. For instance, if MPEG-Surround is decoded with a plain AAC decoder, the DRC gains are applied in the time domain. Furthermore, the sub-band domain may not be the MDCT domain of a core codec. Instead, the sub-band domain is usually the QMF domain.
To achieve multi-band compression, the compressor bands are mapped to groups of decoder sub-bands. There is no need to do additional filtering. The DRC crossover frequencies are mapped to the closest decoder sub-band crossover frequency available. Given the normalized sub-band crossover frequencies fc,Norm,SB(s) for sub-band s, the mapped crossover frequency of fc,Norm(b) is:
The DRC gains may be decoded as described above. The DRC gains may be interpolated using the same technique as described in Table 8 and Table 9, however, the sampling rate of the interpolation result is lowered to match the sample rate of the sub-band signals. This can be achieved by sub-sampling the interpolated time-domain DRC gains by a factor of L or by directly interpolating using the sub-band sample rate as target.
To avoid sharp spectral transitions between the DRC bands, there may be a “cross-fade” between the gains of neighboring DRC bands. This operation is called overlapping. The overlapping is controlled by weight coefficients w, one for each sub-band. The weight coefficients w may be computed according to Table 14 and determine the contribution of the DRC gain of the current band and that of the next band.
After the overlap, the DRC gains of each compressor band are applied to each sub-band group corresponding to the compressor band. A small time delay D to account for the filter bank delay is added to the DRC gains to achieve proper time alignment with the audio signal. The down-sampling and delay operations can be expressed by the first part of pseudo code in Table 13. The values of the two parameters are discussed as codec-specific values below. The meaning of variables and functions of the pseudo code in Table 13 is explained in Table 15. The description assumes that the sample rates in all sub-bands are equal. If this is not the case, the down-sampling factor L may be adjusted for the different sub-band sample rates.
DRC Configuration for Legacy Streaming Scenarios
The DRC configuration information may be conveyed by the MPEG File Format syntax. However, if a legacy streaming format such as ADTS is used to carry an MPEG Audio stream that doesn'"'"'t support MPEG File Format, the configuration information may be embedded in the audio stream. This can be achieved by adding the AudioSampleEntry( ) syntax (or a compressed version of it) of the File Format to the uni_drc_info( ) syntax. Since the sample entry information is only required at a lower rate than the frame rate, a presence flag can be used that indicates when this information is available. The extended syntax is given in Table 16.
For this case, the DRC information can only be decoded after the Sample Entry has been received by the decoder. The repetition rate of the Sample Entry information determines the decoding delay.
If a bitstream contains the proposed DRC metadata and other DRC metadata such MPEG light or heavy compression, the proposed metadata will take precedence unless the decoder is instructed to apply the other DRC metadata.
Decoder Specific Information
Advanced Audio Coding (AAC)
DRC Metadata Extension for AAC
For AAC a new extension payload with a new ID to carry the proposed DRC metadata in a Fill Element may be used. The IDs are encoded using a 4 bit code and currently only 7 are defined. Embedding this DRC information in a new extension payload guarantees backwards compatibility with existing decoders that will skip the new payload. The proposed new extension_type is given in Table 17. It contains uni_drc_info( ) as given in Table 20.
Delay Mode for AAC
AAC uses the default delay mode.
DRC Frame Size and Time Resolution for AAC
The DRC frame size has the default size (i.e., it has the same time duration as the AAC frame size).
The value of delta_t_min in number of samples at the audio rate is calculated as specified in section above labeled “Time resolution”. Specific values are provided here for convenience based on the following formula and Table 18:
The applicable exponent M may be found by looking up the audio sample rate range that fulfills:
Given the codec frame size NCodec, the DRC frame size in units of DRC samples at a rate of delta_t_min is:
DRC Metadata Extension
In USAC the new extension payload can be carried in the extension payload element UsacExtElement. For that purpose a new extension element type is defined according to Table 19. The default codec dependent DRC settings apply.
MPEG-4 HE-AAC, HE-AACv2, MPEG-D Surround, MPEG-D SAOC
DRC Metadata Extension
The DRC metadata may be carried with the AAC core stream as described above.
MPEG-4 HE-AAC, HE-AACv2, MPEG-D Surround, and MPEG-D SAOC are composed of a core decoder such as AAC-LC and one or more additional layers on top of this core decoder. The additional layers enhance the audio bandwidth or the number of audio channels compared to the core. For these decoders, the DRC gains should be applied to the sub-bands of the highest layer immediately before the synthesis filter bank but not after the rendering/mixing stage.
DRC Gain Applications in Sub-Bands
For the DRC gain application in the QMF domain, the time-domain DRC gains may be delayed by D time-domain sample intervals and down-sampled by a factor of L. The values of D and L depend on the configuration, such as single rate versus dual rate HE-AAC. Proper time alignment between DRC gains and the audio signal must be achieved for all configurations.
The DRC bitstream is defined in Table 20 and Table 21. Typically the DRC bitstream time_domain_drc_info( ) is carried in an extension payload field of the host codec.
Linear Interpolation DRC Encoding
Although described above in relation to encoding and decoding of gain values based using spline interpolation, in some embodiments gain values may be encoded and applied with the use of linear interpolation. For example, in one embodiment, DRC values may be encoded for a piece of sound program content using spline nodes as described above. In this embodiment, slope values between each of the spline nodes may be omitted from the bitstream. Instead, linear interpolation may be performed between spline nodes instead of spline interpolation. In this fashion, encoding of the DRC gain values may be simplified by avoiding generation of slope values.
Window-Based Overlap-Add DRC Gain Interpolation
In one embodiment, a window-based overlap-add gain interpolation method may be used for decoding DRC gain values. In this method, gain values are encoded and received in a similar fashion as described above. However, each gain value is used as a multiplier for a window (e.g., a vector of window coefficients) during decoding. The interpolated gain curve may thereafter be obtained by using an overlap-add method. For example, the interpolated DRC gain curve for a window may be the product of a gain value multiplied by a window. One reason for using windows is that an identical gain curve may be generated compared to a gain curve that results from standardized decoders, which apply gain values in sub-bands. Further, the window-based overlap-add gain interpolation method does not generate aliasing distortions. A more in-depth description of this window-based overlap-add gain interpolation method is described below by way of example.
The equation below illustrates how a DRC window is computed from a long AAC synthesis window with the AAC frame size N:
wDRC,long(n)=wAAC,long2(n) for n=[0,2N−1] (8)
Short and transitional windows may be computed in a similar fashion. Equation (9) below shows the overlap-add process of subsequent DRC windows weighted by the DRC gain values derived from the bitstream. The time and frame indices are denoted n and k, respectively. The time index of 0 is located at the beginning of the current synthesis window (the first output sample of the current frame).
g(n)=gDRC(k−1)wDRC(k−1,N+n)+gDRC(k)wDRC(k,n) for n=[0,N−1] (9)
The DRC gain may then be applied to the decoder output signal xAAC according to equation (10) below to generate the final compressed audio output xDRC. The DRC gain is not applied in the MDCT domain.
xDRC(n)=g(n)xAAC(n) for n=[0,N−1] (10)
When using light compression in MPEG, the multi-band DRC metadata may be used to apply independent DRC gain values to individual or grouped short blocks. In contrast to the label “Multi-band”, each DRC gain may be encoded such that it is applied to the entire MDCT spectrum of a short block. Hence each DRC gain operates as a single-band DRC. If this is the case, the DRC operation can instead be done in the time-domain similarly as described above.
For example, if 5 DRC gain values are given for the 8 short blocks as shown in the top graph of
In general, the bitstream syntax allows the independent choice of single-band or multi-band DRC for individual frames. With the time-domain DRC implementation described above, the decoder will switch to MDCT domain DRC processing whenever there is true multi-band DRC gain information (“true multi-band” meaning that there are unequal DRC gain values for different sub-bands).
This proposal includes a modified MPEG-AAC DRC implementation that avoids aliasing distortions in a backward compatible way for single band DRCs. Although described in relation to MPEG-AAC DRC, in other embodiments any type of bitstream audio may be used that includes frequency domain DRC gain values.
In the embodiment described above, the decoder is modified to apply DRC in the time domain. In another embodiment, an additional field may be added to the bitstream to increase the variability of DRC gain value application to the audio signal in the time domain. The new field for the DRC gain values may be defined in various locations in the bitstream syntax. For MPEG standards, one option is the definition of an additional extension payload carried in a Fill Element as shown in Table 27, In this embodiment, the audio channels of the program content may be grouped into DRC groups, where each group has an independent set of DRC information, i.e., a separate independent DRC is applied to each group of channels. An audio channel may only belong to one DRC group or none. The grouping information may be added to the Sample Description, which occurs once at the beginning of a track. In this embodiment, the number of DRC groups is called nDrcChannelGroups.
When observing the time-varying gain of practical implementations, it can be seen that the gain may change very slowly at times, while it can exhibit dramatic changes when the audio signal exhibits attacks. The necessary bit rate to encode the DRC gain values is reduced by supporting an individually selectable time resolution for each so-called drcGainInfoBlock. An audio frame is uniformly divided into up to 8 of these info blocks as shown in Table 28 and each of the blocks can contain up to 16 gain values.
The bit rate increase associated with a larger time resolution of the gain values is further mitigated by using an adaptive scheme with entropy coding of the gain changes. The DRC gain values can be transmitted in each audio frame using the syntax defined in Table 27.
The entry drcGainCodingMode determines the number of gain values for an info block as given in Table 29. There may be at least one gain value per frame to support random break-in. The first gain value is encoded according to Table 30. The remaining gain values are differentially encoded using Table 31 or Table 32 (depending on the drcGainCodingMode selected).
The non-uniform resolution of the difference values is motivated by psychoacoustics, such as the observation that deviations in the gain change are less audible the larger the gain change. Vice versa, if the gain is almost constant (and the audio envelope, too), deviations in gain changes are more audible. The asymmetric range is adapted to the fast acting DRC gain reduction for sudden attacks in the audio signal. Gain increases are usually slower.
A typical audio decoder reconstructs the audio signal using an overlap-add method with 50% overlap of subsequent blocks. Each of the blocks is weighted by a window that tapers off at either end. For instance, a typical frame size of MPEG-AAC is 1024 samples. For each new frame, the decoder reconstructs 2048 samples, the first 1024 of which are added to the last 1024 samples of the previous block and the result is the decoder output. The info blocks that come with frame k are scheduled uniformly during the second half of the reconstructed block. The gain values within each info block are distributed uniformly across the info block'"'"'s duration. This scheme ensures that all necessary DRC gain values are available when decoding starts and ends, as well as for interpolation.
An example is shown in
The computation of the gain value timing is given in Table 34. The result tGain[g][k] indicates the sample location in units of sample intervals starting with 0.0 at the first sample of the current output frame. The frame size is denoted Nframe in samples.
Given the gain values and their timing, a smooth gain curve for all samples of the current output frame may be constructed by linear interpolation of the linear gain values as shown in Table 35. gDRCprev is the last DRC gain value of the previous frame. In this embodiment, the first gain value of the next frame is needed to interpolate the gain values of the frame for output. Due to the overlap-add process, that gain value is available without extra reading ahead in the bitstream. The function toLinear( ) is introduced to include all necessary steps to generate a linear gain value from the logarithmic value in dB.
Finally, the interpolated DRC gains are applied as shown in Table 36.
This embodiment described above includes improved DRC metadata encoding and processing for audio standards such as MPEG-Audio. Shortcomings of current standards, such as the generation of aliasing distortions and insufficient time resolution of the DRC metadata have been addressed.
As discussed above, multiple techniques may be used for encoding and applying DRC gain values for a piece of sound program content. In some embodiments, a method for applying frequency domain Dynamic Range Control (DRC) gain values to an audio signal in the time domain, comprises: receiving a bitstream, wherein the bitstream includes an encoded audio signal and frequency domain DRC gain values; decoding, by a decoder in a playback device, the encoded audio signal to produce a decoded audio signal in the time domain; determining, by the decoder, DRC window weights for applying the frequency DRC gain values to the decoded audio signal in the time domain; determining time domain DRC gain values based on the frequency domain DRC gain values and the DRC window weights; and applying the time domain DRC gain values for corresponding frames of the decoded audio signal in the time domain.
In one embodiment, the DRC window weights are determined based on a synthesis window of the decoder. In one embodiment, the DRC window weights are computed as the square of the decoder synthesis window with the same timing as the synthesis window of the decoder. In one embodiment, the DRC window weights are determined based on the product of the synthesis window of the decoder and a window of the encoder. In one embodiment, the time domain DRC gain values for a current frame of the decoded audio signal are determined based on the frequency domain DRC gain values for the current frame with a corresponding DRC window weight applied and the frequency domain DRC gain values for the previous frame with a corresponding DRC window weight applied. In one embodiment, applying the time domain DRC gain values to produce the DRC audio signal in the time domain is based on the product of the time domain DRC gain values and corresponding time divisions of the decoded audio signal. In one embodiment, one or more of the time domain DRC gain values are applied to an entire DRC window for the decoded audio signal. In one embodiment, the encoded audio signal is a Moving Picture Experts Group-Advanced Audio Coding (MPEG-AAC) DRC audio signal. In one embodiment, the encoded audio signal is an Advanced Television Systems Committee'"'"'s (ATSC) DRC audio signal.
In another embodiment, a method of encoding Dynamic Range Control (DRC) gain values in a bitstream representing a piece of sound program content comprises: grouping each audio channel of the sound program content into a single DRC group from a set of DRC groups; and inserting DRC gain metadata into the bitstream for each DRC group, wherein the DRC gain metadata for each DRC group is used to variably apply corresponding DRC gain values to each frame in the DRC group. In one embodiment, the DRC gain metadata for each DRC group includes: a first data value representing a selected coding mode for an initial DRC gain value; a second data value representing the initial DRC gain value; and a third data value representing differences applied to the initial DRC gain value to generate the DRC gain values for each frame of the DRC group. In one embodiment, the first data value represents the number of gain values to apply to each frame of the DRC group based on the initial DRC gain value. In one embodiment, the selected coding mode represented by the first data value is chosen from a predefined set of coding modes. In one embodiment, the DRC gain values are applied using interpolation. In one embodiment, the interpolation is a linear interpolation in the linear domain. In one embodiment, multiple channels are assigned to a single DRC group. In one embodiment, a non-uniform time resolution is used for update rates of DRC gain values based on the variance of gain generated by encoder DRC to minimize the bit rate of the bitstream. In one embodiment, the first data value representing the initial gain value is encoded using a non-uniform quantization scale based on psychoacoustics to minimize the bit rate of the bitstream. In one embodiment, the first data value representing the initial gain value is encoded using a variable-length code to minimize the bit rate of the bitstream. In one embodiment, the third data value representing differences applied to the initial DRC gain value for each frame of the DRC group is encoded to minimize the bit rate of the bitstream. In one embodiment, the third data value representing differences applied to the initial DRC gain value are encoded variable-length codes to minimize the bit rate of the bitstream.
As explained above, an embodiment of the invention may be a machine-readable medium such as one or more solid state memory devices having stored thereon instructions which program one or more data processing components (generically referred to here as “a processor” or a “computer system”) to perform some of the operations described above. In other embodiments, some of these operations might be performed by specific hardware components that contain hardwired logic. Those operations might alternatively be performed by any combination of programmed data processing components and fixed hardwired circuit components.
While certain embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, and that the invention is not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those of ordinary skill in the art. The description is thus to be regarded as illustrative instead of limiting.