Multi-channel audio encoding and decoding

• US 20040049379A1
• Filed: 08/15/2003
• Published: 03/11/2004
• Est. Priority Date: 09/04/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Thumpudi, Naveen, Chen, Wei-Ge
• Granted Patent: US 7,502,743 B2
• Time in Patent Office: Days
• US Class Current: 704/205
• CPC Class Codes: G10L 19/00 Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis in musical instruments G10H
• CPC Class Codes: G10L 19/008 Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
• CPC Class Codes: G10L 19/0212 using orthogonal transformation

Abstract

An audio encoder and decoder use architectures and techniques that improve the efficiency of multi-channel audio coding and decoding. The described strategies include various techniques and tools, which can be used in combination or independently. For example, an audio encoder performs a pre-processing multi-channel transform on multi-channel audio data, varying the transform so as to control quality. The encoder groups multiple windows from different channels into one or more tiles and outputs tile configuration information, which allows the encoder to isolate transients that appear in a particular channel with small windows, but use large windows in other channels. Using a variety of techniques, the encoder performs flexible multi-channel transforms that effectively take advantage of inter-channel correlation. An audio decoder performs corresponding processing and decoding. In addition, the decoder performs a post-processing multi-channel transform for any of multiple different purposes.

Claims

1. In an audio encoder, a computer-implemented method of encoding comprising:

receiving multi-channel audio data; and

performing a pre-processing multi-channel transform on the audio data, wherein the encoder varies the transform during the encoding so as to control quality.

2. The method of claim 1 wherein the multi-channel audio data is in two channels.

3. The method of claim 1 wherein the multi-channel audio data is in more than two channels.

4. The method of claim 1 wherein the transform is performed in the time domain.

5. The method of claim 1 further comprising performing a second multi-channel transform on the audio data, wherein the second multi-channel transform is performed in the frequency domain.

6. The method of claim 1 wherein the encoder varies the transform to reduce complexity of the audio data by increasing inter-channel correlation when quality is low anyway.

7. The method of claim 1 wherein the encoder varies the transform based at least in part on quality measurements during the encoding.

8. The method of claim 7 wherein the transform uses a matrix with at least one element that varies in proportion to the quality measurements.

9. The method of claim 1 wherein the encoder blends plural matrices for the transform across a transition.

10. The method of claim 1 wherein the encoder varies the transform by using an identity matrix or performing no transform for some of the audio data.

11. The method of claim 1 wherein the encoder varies the transform on a frame-by-frame basis.

12. A computer-readable medium storing computer-executable instructions for causing a computer programmed thereby to perform the method of claim 1.

13. In an audio encoder, a computer-implemented method comprising:

receiving multi-channel audio data;

performing a first multi-channel transform on the audio data; and

outputting information indicating a second multi-channel transform so as to enable an audio decoder to construct one or more phantom channels.

14. The method of claim 13 wherein the multi-channel audio data is in two channels.

15. The method of claim 13 wherein the multi-channel audio data is in more than two channels.

16. The method of claim 13 wherein the first transform is a pre-processing multi-channel transform performed in the time domain, and wherein the second transform is a post-processing multi-channel transform performed in the time domain.

17. The method of claim 13 wherein the encoder varies the first transform on a frame-by-frame basis.

18. A computer-readable medium storing computer-executable instructions for causing a computer programmed thereby to perform the method of claim 13.

19. In an audio decoder, a computer-implemented method comprising:

receiving encoded multi-channel audio data;

decoding the audio data, producing decoded time domain audio data; and

performing a post-processing multi-channel transform on the decoded audio data, wherein the decoder uses the transform for any of plural different purposes.

20. The method of claim 19 wherein the multi-channel audio data is in two channels.

21. The method of claim 19 wherein the multi-channel audio data is in more than two channels.

22. The method of claim 19 wherein the decoder constructs one or more phantom channels with the transform.

23. The method of claim 22 wherein the one or more phantom channels include a phantom center channel.

24. The method of claim 19 wherein the decoder performs spatial effects with the transform.

25. The method of claim 19 wherein the decoder folds down decoded channels into fewer output channels with the transform.

26. The method of claim 19 wherein the decoder varies the transform within an audio sequence.

27. The method of claim 26 wherein the decoder varies the transform by selecting between an identity matrix and one or more other matrices.

28. The method of claim 26 wherein the decoder varies the transform on a frame-by-frame basis.

29. The method of claim 26 wherein the decoder blends plural matrices for the transform across a transition.

30. The method of claim 26 further comprising, for each of plural frames:

receiving information indicating whether to perform the transform;

if the transform is to be performed, receiving information indicating whether to perform the transform using a pre-defined matrix or a custom matrix; and

if the transform is to be performed with the custom matrix, receiving elements of the custom matrix.

31. A computer-readable medium storing computer-executable instructions for causing a computer programmed thereby to perform the method of claim 19.

32. In an audio encoder, a computer-implemented method comprising:

receiving multi-channel audio data;

grouping plural windows from different channels into one or more tiles; and

outputting tile configuration information for the one or more tiles.

33. The method of claim 32 wherein the multi-channel audio data is in two channels.

34. The method of claim 32 wherein the multi-channel audio data is in more than two channels.

35. The method of claim 32 wherein the encoder groups windows that have the same start time and same stop time into a single tile.

36. The method of claim 32 wherein the different channels include first, second, and third channels, wherein the first channel includes a first window and a second window, wherein the second channel includes a window co-located in time with the first window of the first channel, wherein the third channel includes a window co-located in time with the second window of the first channel, wherein the first window of the first channel is in a first tile along with the window of the second channel, and wherein the second window of the first channel is in a second tile along with the window of the third channel.

37. The method of claim 32 wherein the tile configuration information includes tile size and channel member information.

38. The method of claim 32 wherein the outputting comprises sending a signal to indicate whether the different channels all have an identical window configuration.

39. The method of claim 38 wherein the outputting further comprises, if the different channels all have an identical window configuration, sending plural tile sizes.

40. The method of claim 38 wherein the outputting further comprises, if the different channels do not all have an identical window configuration, sending one or more channel masks and plural tile sizes.

41. A computer-readable medium storing computer-executable instructions for causing a computer programmed thereby to perform the method of claim 32.

42. In an audio encoder, a computer-implemented method comprising:

receiving audio data in plural channels, wherein the plural channels include first, second, and third channels;

partitioning the audio data into plural windows;

grouping the plural windows into plural groups, wherein the plural groups include first and second groups, wherein the first and second channels but not the third channel are members of the first group, and wherein the first and third channels but not the second channel are members of the second group; and

outputting configuration information for the plural groups.

43. The method of claim 42 wherein the encoder independently partitions the audio data in each of the plural channels.

44. The method of claim 42 wherein the encoder groups windows that have the same start time and same stop time into a single one of the plural groups.

45. The method of claim 42 wherein a third group of the plural groups includes windows from three or more channels.

46. The method of claim 42 wherein the encoder completes the partitioning for a given frame of the audio data before beginning the grouping for the given frame.

47. The method of claim 42 wherein the encoder performs the partitioning and the grouping concurrently for a given frame of the audio data.

48. A computer-readable medium storing computer-executable instructions for causing a computer programmed thereby to perform the method of claim 42.

49. In an audio decoder, a computer-implemented method comprising:

receiving encoded audio data in plural channels;

retrieving tile configuration information for one or more tiles; and

decoding the audio data based at least in part upon the retrieved tile configuration information.

50. The method of claim 49 wherein the plural channels consist of two channels.

51. The method of claim 49 wherein the plural channels consist of more than two channels.

52. The method of claim 49 wherein each of the one or more tiles includes one or more windows that have the same start time and same stop time.

53. The method of claim 49 wherein the tile configuration information includes tile size and channel member information.

54. The method of claim 49 wherein the retrieving comprises getting a signal to indicate whether the plural channels all have an identical window configuration.

55. The method of claim 54 wherein the retrieving further comprises, if the plural channels all have an identical window configuration, getting plural tile sizes.

56. The method of claim 54 wherein the retrieving further comprises, if the plural channels do not all have an identical window configuration, getting one or more channel masks and plural tile sizes.

57. A computer-readable medium storing computer-executable instructions for causing a computer programmed thereby to perform the method of claim 49.

58. In an audio encoder, a computer-implemented method comprising:

receiving multi-channel audio data;

weighting the audio data so as to shape noise according to quantization bands;

after the weighting, performing a multi-channel transform on the weighted audio data; and

after the multi-channel transform, quantizing the audio data.

59. The method of claim 58 wherein the multi-channel audio data is in two channels.

60. The method of claim 58 wherein the multi-channel audio data is in more than two channels.

61. The method of claim 58 further comprising, before the multi-channel transform, applying per-channel weights to the audio data.

62. A computer-readable medium storing computer-executable instructions for causing a computer programmed thereby to perform the method of claim 58.

63. In an audio decoder, a computer-implemented method comprising:

receiving encoded multi-channel audio data;

performing an inverse multi-channel transform on the audio data; and

after the inverse multi-channel transform, performing inverse weighting and inverse quantization in a combined step.

64. The method of claim 63 wherein the multi-channel audio data is in two channels.

65. The method of claim 63 wherein the multi-channel audio data is in more than two channels.

66. The method of claim 63 wherein for each of plural coefficients the combined step includes a single multiplication by a total quantization amount.

67. The method of claim 63 wherein the combined step further factors in per-channel weights.

68. A computer-readable medium storing computer-executable instructions for causing a computer programmed thereby to perform the method of claim 63.

69. In an audio decoder, a computer-implemented method comprising:

receiving encoded multi-channel audio data;

performing an inverse multi-channel transform on the audio data; and

after the inverse multi-channel transform, performing inverse weighting, inverse quantization, and inverse frequency transformations;

wherein one or more channels are dropped from the multi-channel audio data.

70. The method of claim 69 wherein the multi-channel audio data is in more than two channels.

71. The method of claim 69 wherein the one or more channels are dropped to reduce computational complexity.

72. The method of claim 69 wherein an encoder drops the one or more channels.

73. The method of claim 69 wherein the decoder drops the one or more channels after performing the inverse frequency transformations.

74. The method of claim 69 wherein the decoder drops the one or more channels after performing the inverse multi-channel transform but before performing the inverse frequency transformations.

75. The method of claim 74 wherein the decoder applies per-channel quantization step modifiers.

76. A computer-readable medium storing computer-executable instructions for causing a computer programmed thereby to perform the method of claim 69.

77. In an audio encoder, a computer-implemented method comprising:

receiving multi-channel audio data;

selectively grouping plural channels of the multi-channel audio data into plural channel groups for multi-channel transforms, wherein the encoder groups the plural channels differently at different times in an audio sequence; and

performing a multi-channel transform on the audio data for each of one or more of the plural channel groups.

78. The method of claim 77 wherein the multi-channel audio data is in two channels.

79. The method of claim 77 wherein the multi-channel audio data is in more than two channels.

80. The method of claim 77 wherein each of the plural channel groups includes one or more channels.

81. The method of claim 77 wherein at least one of the plural channel groups includes three or more channels.

82. The method of claim 77 wherein a tile includes one or more of the plural channel groups.

83. The method of claim 77 wherein each of the plural channel groups has an associated multi-channel transform.

84. The method of claim 77 wherein the encoder selectively groups the plural channels based at least in part upon channel correlations.

85. The method of claim 84 wherein the encoder computes the channel correlations overall and at specific frequency bands.

86. The method of claim 77 further comprising outputting one or more channel masks.

87. A computer-readable medium storing computer-executable instructions for causing a computer programmed thereby to perform the method of claim 77.

88. In an audio decoder, a computer-implemented method comprising:

receiving encoded audio data in plural channels;

retrieving information for plural channel groups of the plural channels for inverse multi-channel transforms, wherein the plural channels are grouped differently at different times in an audio sequence; and

performing an inverse multi-channel transform on the audio data for each of one or more of the plural channel groups.

89. The method of claim 88 wherein the plural channels consist of two channels.

90. The method of claim 88 wherein the plural channels consist of more than two channels.

91. The method of claim 88 wherein each of the plural channel groups includes one or more channels.

92. The method of claim 88 wherein at least one of the plural channel groups includes three or more channels.

93. The method of claim 88 wherein a tile includes one or more of the plural channel groups.

94. The method of claim 88 wherein the retrieved information includes one or more channel masks.

95. A computer-readable medium storing computer-executable instructions for causing a computer programmed thereby to perform the method of claim 88.

96. In an audio encoder, a computer-implemented method comprising:

receiving multi-channel audio data;

transforming the audio data according to a hierarchy of plural multi-channel transforms in plural stages; and

outputting information for the hierarchy of plural multi-channel transforms.

97. The method of claim 96 wherein the multi-channel audio data is in two channels.

98. The method of claim 96 wherein the multi-channel audio data is in more than two channels.

99. The method of claim 96 wherein each of the plural stages includes one or more of the transforms.

100. The method of claim 96 wherein the encoder selects the transforms.

101. The method of claim 96 wherein channel groups are the same in at least two of the plural stages.

102. The method of claim 96 wherein channel groups are different in at least two of the plural stages.

103. The method of claim 96 wherein the output information comprises channel group information and transform information.

104. The method of claim 96 wherein the plural stages consist of two stages.

105. The method of claim 96 wherein the hierarchy emulates another transform while reducing computational complexity compared to the other transform.

106. The method of claim 96 wherein at least one of the transforms is an identity transform, and wherein at least one of the transforms is a general unitary transform factored into plural matrices.

107. A computer-readable medium storing computer-executable instructions for causing a computer programmed thereby to perform the method of claim 96.

108. In an audio decoder, a computer-implemented method comprising:

receiving encoded multi-channel audio data;

retrieving information for a hierarchy of plural inverse multi-channel transforms in plural stages; and

decoding the audio data, including transforming the audio data according to the hierarchy of the plural inverse multi-channel transforms.

109. The method of claim 108 wherein the multi-channel audio data is in two channels.

110. The method of claim 108 wherein the multi-channel audio data is in more than two channels.

111. The method of claim 108 wherein channel groups are the same in at least two of the plural stages.

112. The method of claim 108 wherein channel groups are different in at least two of the plural stages.

113. The method of claim 108 wherein the plural stages consist of first and second stages, and wherein the retrieving comprises:

(a) getting a bit;

(b) if the bit indicates there are no more transforms in the first stage, continuing to step (c), otherwise, getting transform information for a transform in the first stage, getting a new bit, and repeating step (b) with the new bit; and

(c) getting transform information for one or more transforms in the second stage.

114. The method of claim 113 further comprising getting channel group information along with at least some of the transform information.

115. A computer-readable medium storing computer-executable instructions for causing a computer programmed thereby to perform the method of claim 108.

116. In an audio encoder, a computer-implemented method comprising:

receiving multi-channel audio data;

selecting a multi-channel transform from among plural available types of multi-channel transforms;

selectively turning the selected transform on/off at plural frequency bands; and

performing the selected transform on the audio data at one or more of the plural frequency bands at which the selected transform is on, wherein the encoder performs no transform or an identity transform on the audio data at zero or more of the plural frequency bands at which the selected transform is off.

117. The method of claim 116 wherein the multi-channel audio data is in two channels.

118. The method of claim 116 wherein the multi-channel audio data is in more than two channels.

119. The method of claim 116 further comprising outputting a mask including one bit for each of the plural frequency bands.

120. The method of claim 116 further comprising outputting a single bit and, if the selected transform is not turned on at all of the plural frequency bands, a mask including one bit for each of the plural frequency bands.

121. The method of claim 116 wherein the encoder selectively turns the selected transform on/off based at least in part upon channel correlation measurements at the plural frequency bands.

122. A computer-readable medium storing computer-executable instructions for causing a computer programmed thereby to perform the method of claim 116.

123. In an audio decoder, a computer-implemented method comprising:

receiving encoded multi-channel audio data;

selecting an inverse multi-channel transform from among plural available types of inverse multi-channel transforms;

retrieving information for frequency band on/off selections for plural frequency bands; and

performing the selected transform on the audio data at one or more of the plural frequency bands at which the selected transform is on, wherein the encoder performs no transform or an identity transform on the audio data at zero or more of the plural frequency bands at which the selected transform is off.

124. The method of claim 123 wherein the multi-channel audio data is in two channels.

125. The method of claim 123 wherein the multi-channel audio data is in more than two channels.

126. The method of claim 123 wherein the retrieved information comprises a mask including one bit for each of the plural frequency bands.

127. The method of claim 123 wherein the retrieved information comprises a single bit and, if the selected transform is not turned on at all of the plural frequency bands, a mask including one bit for each of the plural frequency bands.

128. A computer-readable medium storing computer-executable instructions for causing a computer programmed thereby to perform the method of claim 123.

129. In an audio encoder, a computer-implemented method comprising:

receiving multi-channel audio data;

selecting a multi-channel transform from among plural available types of multi-channel transforms, wherein the plural available types include three or more pre-defined transforms; and

performing the selected transform on the audio data.

130. The method of claim 129 wherein the multi-channel audio data is in two channels.

131. The method of claim 129 wherein the multi-channel audio data is in more than two channels.

132. The method of claim 129 wherein the pre-defined transforms include an identity transform and one or more of a DCT variant and a Hadamard transform.

133. The method of claim 129 wherein the plural available types further include a general unitary transform.

134. The method of claim 129 further comprising outputting information indicating the selected transform.

135. A computer-readable medium storing computer-executable instructions for causing a computer programmed thereby to perform the method of claim 129.

136. In an audio encoder, a computer-implemented method comprising:

receiving multi-channel audio data;

selecting a multi-channel transform from among plural available types of multi-channel transforms, wherein the plural available types include plural pre-defined transforms and at least one custom transform; and

performing the selected transform on the audio data.

137. The method of claim 136 wherein the multi-channel audio data is in two channels.

138. The method of claim 136 wherein the multi-channel audio data is in more than two channels.

139. The method of claim 136 further comprising outputting information indicating the selected transform.

140. The method of claim 139 wherein the output information includes information for individual elements of the selected transform.

141. The method of claim 136 wherein the encoder selects one of the plural pre-defined transforms if performance of the selected pre-defined transform is suitably close to performance of the custom transform in terms of redundancy removal.

142. A computer-readable medium storing computer-executable instructions for causing a computer programmed thereby to perform the method of claim 136.

143. In an audio decoder, a computer-implemented method comprising:

receiving encoded multi-channel audio data;

selecting an inverse multi-channel transform from among plural available types of inverse multi-channel transforms, wherein the plural available types include three or more pre-defined transforms; and

performing the selected transform on the audio data.

144. The method of claim 143 wherein the multi-channel audio data is in two channels.

145. The method of claim 143 wherein the multi-channel audio data is in more than two channels.

146. The method of claim 143 wherein the pre-defined transforms include an identity transform and one or more of a DCT variant and a Hadamard transform.

147. The method of claim 143 further comprising, before the selecting, retrieving information indicating the selected transform.

148. The method of claim 147 wherein the plural available types further include a custom transform, wherein the retrieved information includes one or more signals to select the custom transform, and wherein the retrieved information further includes information for individual elements of the custom transform.

149. A computer-readable medium storing computer-executable instructions for causing a computer programmed thereby to perform the method of claim 143.

150. In an audio decoder, a computer-implemented method comprising:

receiving encoded multi-channel audio data;

selecting an inverse multi-channel transform from among plural available types of inverse multi-channel transforms, wherein the plural available types include plural pre-defined transforms and at least one custom transform; and

performing the selected transform on the audio data.

151. The method of claim 150 wherein the multi-channel audio data is in two channels.

152. The method of claim 150 wherein the multi-channel audio data is in more than two channels.

153. The method of claim 150 further comprising, before the selecting, retrieving information indicating the selected transform.

154. The method of claim 153 wherein the retrieved information includes one or more signals to select the custom transform, and wherein the retrieved information further includes information for individual elements of the custom transform.

155. A computer-readable medium storing computer-executable instructions for causing a computer programmed thereby to perform the method of claim 150.

156. In an audio encoder, a computer-implemented method comprising:

receiving multi-channel audio data;

computing an arbitrary unitary transform matrix for a multi-channel transform;

factorizing the arbitrary unitary transform matrix into plural rotation matrices and a sign matrix;

performing the factorized transform on the audio data; and

outputting information for the factorized transform.

157. The method of claim 156 wherein the multi-channel audio data is in two channels.

158. The method of claim 156 wherein the multi-channel audio data is in more than two channels.

159. The method of claim 156 wherein the output information includes angles for the plural rotation matrices and signs for the sign matrix.

160. The method of claim 159 further comprising quantizing the angles to 6-bit precision.

161. A computer-readable medium storing computer-executable instructions for causing a computer programmed thereby to perform the method of claim 156.

162. In an audio decoder, a computer-implemented method comprising:

receiving encoded multi-channel audio data;

retrieving information for a factorized transform of an arbitrary unitary inverse transform matrix; and

performing the factorized transform on the audio data.

163. The method of claim 162 wherein the multi-channel audio data is in two channels.

164. The method of claim 162 wherein the multi-channel audio data is in more than two channels.

165. The method of claim 162 wherein the retrieved information includes angles for plural rotation matrices and signs for a sign matrix.

166. The method of claim 165 wherein the angles are quantized to 6-bit precision.

167. A computer-readable medium storing computer-executable instructions for causing a computer programmed thereby to perform the method of claim 162.