Method and apparatus for processing audio signals using motion of a sound source, reverberation property, or semantic object

• US 9,294,862 B2
• Filed: 04/16/2009
• Issued: 03/22/2016
• Est. Priority Date: 04/17/2008
• Status: Expired due to Fees

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Kim, Hyun-Wook, Lee, Chul-Woo, Jeong, Jong-Hoon, Lee, Nam-Suk, Moon, Han-Gil, Lee, Sang-Hoon
• Primary Examiner(s): Shah, Paras D
• Application Number:
  US12/988,430
• Publication Number:
  US 20110060599A1
• Time in Patent Office: 2,532 Days
• Field of Search: 704/201, 704/500, 704/503
• US Class Current: 1/1
• CPC Class Codes: G10L 19/00 Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis in musical instruments G10H
• CPC Class Codes: H04R 5/04 Circuit arrangements, e.g. for selective connection of amplifier inputs/outputs to loudspeakers, for loudspeaker detection, or for adaptation of settings to personal preferences or hearing impairments combinations of amplifiers H03F3/68; stereophonic systems H04S
• CPC Class Codes: H04S 2400/03 Aspects of down-mixing multi-channel audio to configurations with lower numbers of playback channels, e.g. 7.1 -> 5.1 H04S2400/01 takes precedence
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 7/30 Control circuits for electronic adaptation of the sound field

Abstract

Methods and apparatuses for encoding and decoding an audio signal are provided, a method of encoding an audio signal including: receiving the audio signal including information about a moving sound source; receiving position information about the moving sound source; generating dynamic track information indicating motion of the moving sound source by using the position information; and encoding the audio signal and the dynamic track information.

Claims

1. A method of encoding an audio signal, the method comprising:

receiving an audio signal comprising information about a moving sound source;

receiving position information about the moving sound source;

generating dynamic track information indicating motion of the moving sound source by using the position information; and

encoding the audio signal and the dynamic track information,

wherein the dynamic track information comprises control points which express a dynamic track of the moving sound source and the number of frames to which the dynamic track expressed by the control points is applied.

2. The method of claim 1, wherein the dynamic track information comprises a plurality of points for expressing the dynamic track.

3. The method of claim 2, wherein the dynamic track is a Bé

zier curve using the plurality of points as control points.

4. The method of claim 2, wherein:

when the dynamic track is applied to a first frame and a second frame, the encoding the audio signal and the dynamic track information comprises inserting the dynamic track information into the first frame and not the second frame.

93. A non-transitory computer readable recording medium having recorded thereon a program executed by a computer for performing the method of claim 1.

5. A method of decoding an audio signal, the method comprising:

receiving a signal comprising an encoded audio signal and encoded dynamic track information, the audio signal comprising information about a moving sound source and the dynamic track information indicating motion of a position of the moving sound source; and

decoding the encoded audio signal and the encoded dynamic track information from the received signal,

wherein the dynamic track information comprises control points which express a dynamic track of the moving sound source and the number of frames to which the dynamic track expressed by the control points is applied.

6. The method of claim 5, further comprising distributing output to a plurality of speakers so as to correspond to the dynamic track information.

7. The method of claim 5, further comprising changing a frame rate of the audio signal by using the dynamic track information.

8. The method of claim 5, further comprising changing a number of channels of the audio signal by using the dynamic track information.

9. The method of claim 5, further comprising searching the audio signal for a period corresponding to a predetermined motion property of the moving sound source by using the dynamic track information.

10. The method of claim 9, wherein:

the dynamic track information comprises a plurality of points for expressing the dynamic track; and

the searching is performed by using the plurality of points.

11. The method of claim 10, wherein:

the searching is performed by using the number of the frames comprised in the dynamic track information.

12. The method of claim 5, wherein:

the dynamic track information comprises a plurality of points for expressing the dynamic track; and

when the dynamic track is applied to a first frame and a second frame, the dynamic track information is comprised in the first frame and not the second frame.

94. A non-transitory computer readable recording medium having recorded thereon a program executed by a computer for performing the method of claim 5.

13. A method of encoding an audio signal, the method comprising:

receiving a reverberation property of an audio signal separately from receiving the audio signal, the reverberation property being initially separately recorded from the audio signal;

obtaining the audio signal based on the reverberation property; and

encoding, by an encoder comprising a processor, the obtained audio signal and the reverberation property.

14. The method of claim 13, wherein:

the audio signal is recorded in a predetermined space; and

the reverberation property is of the predetermined space.

15. The method of claim 13, wherein the reverberation property is indicated by an impulse response.

16. The method of claim 15, wherein the encoding comprises encoding the audio signal so that an initial reverberation period of the impulse response is expressed in a type of a high-degree infinite impulse response (IIR) filter, and a latter reverberation period of the impulse response is expressed in a type of a low-degree infinite impulse response filter.

95. A non-transitory computer readable recording medium having recorded thereon a program executed by a computer for performing the method of claim 13.

17. A method of decoding an audio signal, the method comprising:

receiving a signal comprising an encoded first reverberation property and an encoded audio signal comprising the first reverberation property, the encoded first reverberation property being initially separately recorded from the encoded audio signal;

decoding, by a decoder comprising a processor, the encoded audio signal from the received signal; and

generating the decoded audio signal based on the encoded audio signal and the first reverberation property.

18. The method of claim 17, further comprising:

decoding the first reverberation property from the received signal;

calculating a reversed function of the first reverberation property; and

obtaining an audio signal from which the first reverberation property is removed by applying the reversed function to the audio signal comprising the first reverberation property.

19. The method of claim 18, further comprising:

receiving a second reverberation property; and

generating an audio signal comprising the second reverberation property by applying the second reverberation property to the audio signal from which the first reverberation property is removed.

20. The method of claim 19, wherein the receiving the second reverberation property comprises receiving the second reverberation property input by a user from an input device, or receiving the second reverberation property that is previously stored in a memory, from the memory.

21. The method of claim 17, wherein:

the audio signal is recorded in a predetermined space; and

the first reverberation property is of the predetermined space.

96. A non-transitory computer readable recording medium having recorded thereon a program executed by a computer for performing the method of claim 17.

22. A method of encoding an audio signal, the method comprising:

receiving an audio signal recorded in a predetermined space;

receiving a reverberation property of the predetermined space, the reverberation property being initially separately recorded from the audio signal;

calculating a reversed function of the reverberation property;

obtaining an audio signal from which the reverberation property is removed by applying the reversed function to the received audio signal; and

encoding the reverberation property and the audio signal from which the reverberation property is removed.

97. A non-transitory computer readable recording medium having recorded thereon a program executed by a computer for performing the method of claim 22.

23. A method of decoding an audio signal, the method comprising:

receiving a signal comprising an encoded audio signal and an encoded reverberation property, the encoded audio signal being initially separately recorded from the encoded reverberation property;

decoding the encoded audio signal from the received signal;

decoding the encoded reverberation property from the received signal; and

obtaining an audio signal comprising the reverberation property by applying the decoded reverberation property to the decoded audio signal.

98. A non-transitory computer readable recording medium having recorded thereon a program executed by a computer for performing the method of claim 23.

24. A method of decoding an audio signal, the method comprising:

receiving a signal comprising an encoded audio signal and an encoded first reverberation property, the encoded audio signal being initially separately recorded from the encoded first reverberation property;

decoding the encoded audio signal from the received signal;

receiving a second reverberation property;

generating an audio signal comprising the second reverberation property by applying the received second reverberation property to the decoded audio signal, and

generating another audio signal comprising the first reverberation property by applying the received first reverberation property to the decoded audio signal.

99. A non-transitory computer readable recording medium having recorded thereon a program executed by a computer for performing the method of claim 24.

25. A method of encoding an audio signal, the method comprising:

receiving, for each of a plurality of semantic objects of the audio signal, at least one parameter indicating at least one property of the semantic object of the audio signal; and

encoding, for each of the plurality of the semantic objects of the audio signal, by an encoder comprising a processor, the at least one parameter,

wherein, for each of the plurality of the semantic objects of the audio signal, the at least one parameter comprises a physical model comprising a transfer function to express a repeated creation and/or extinction of a sound source and indicates a physical property of thea sound source corresponding to the semantic object.

26. The method of claim 25, wherein the at least one parameter further comprises at least one of:

a note list which indicates pitch and beat of the semantic object; and

an actuating signal which actuates the semantic object.

27. The method of claim 26, wherein the transfer function is a ratio between an output signal and the actuating signal in a frequency domain.

28. The method of claim 26, wherein the encoding comprises encoding a coefficient in a frequency domain of the actuating signal.

29. The method of claim 26, wherein the encoding comprises encoding coordinates of a plurality of points in a time domain of the actuating signal.

30. The method of claim 25, wherein the at least one parameter comprises position information indicating a position of the semantic object.

31. The method of claim 25, wherein the at least one parameter comprises spatial information indicating a reverberation property of a space where the audio signal of the semantic object is generated.

32. The method of claim 25, further comprising:

receiving spatial information indicating a reverberation property of a space where the audio signal is generated,

wherein the encoding comprises encoding the at least one parameter comprising the spatial information.

33. The method of claim 31, wherein the spatial information comprises an impulse response exhibiting the reverberation property.

100. A non-transitory computer readable recording medium having recorded thereon a program executed by a computer for performing the method of claim 25.

34. A method of decoding an audio signal, the method comprising:

receiving, for each of a plurality of semantic objects of the audio signal, an input signal comprising at least one encoded parameter indicating at least one property of the semantic object of the audio signal; and

decoding, for each of the plurality of the semantic objects of the audio signal, by a decoder comprising a processor, the at least one encoded parameter from the input signal,

wherein, for each of the plurality of the semantic objects of the audio signal, the at least one encoded parameter comprises a physical model comprising a transfer function to express a repeated creation and/or extinction of a sound source and indicates a physical property of the sound source corresponding to the semantic object.

35. The method of claim 34, further comprising restoring the audio signal by using the at least one parameter.

36. The method of claim 34, wherein the at least one parameter further comprises at least one of:

a note list which indicates pitch and beat of the semantic object; and

an actuating signal which actuates the semantic object.

37. The method of claim 34, wherein the at least one parameter further comprises position information indicating a position of the semantic object.

38. The method of claim 37, further comprising distributing output to a plurality of speakers so as to correspond to the position information.

39. The method of claim 34, wherein the at least one parameter comprises spatial information indicating a reverberation property of a space where the audio signal of the semantic object is generated.

40. The method of claim 34, further comprising decoding spatial information from the input signal,

wherein the input signal further comprises the spatial information indicating a reverberation property of a space where the audio signal is generated.

41. The method of claim 40, further comprising restoring the audio signal by using the at least one parameter and the spatial information.

42. The method of claim 34, further comprising processing the at least one parameter.

43. The method of claim 42, wherein the processing comprises searching for a parameter corresponding to a predetermined audio property from among the at least one parameter.

44. The method of claim 42, wherein the processing comprises editing a parameter of the at least one parameter.

45. The method of claim 44, further comprising generating an edited audio signal by using the edited parameter.

46. The method of claim 44, wherein the editing the parameter comprises at least one of deleting the semantic object from the audio signal, inserting a new semantic object into the audio signal, and replacing the semantic object of the audio signal with the new semantic object.

47. The method of claim 44, wherein the editing the parameter comprises at least one of deleting the parameter, inserting a previously presented parameter into the audio signal, and replacing the parameter with the new parameter.

101. A non-transitory computer readable recording medium having recorded thereon a program executed by a computer for performing the method of claim 34.

48. An apparatus for encoding an audio signal, the apparatus comprising:

a processor;

a receiver which receives an audio signal comprising information about a moving sound source and position information about the moving sound source;

a dynamic track information generator which generates dynamic track information indicating motion of the moving sound source by using the position information; and

an encoder which uses the processor which encodes the audio signal and the dynamic track information,

wherein the dynamic track information comprises control points which express a dynamic track of the moving sound source and the number of frames to which the dynamic track expressed by the control points is applied.

49. The apparatus of claim 48, wherein the dynamic track information comprises a plurality of points for expressing the dynamic track.

50. The apparatus of claim 49, wherein the dynamic track is a Bé

zier curve using the plurality of points as control points.

51. An apparatus for decoding an audio signal, the apparatus comprising:

a processor;

a receiver which receives a signal comprising an encoded audio signal and encoded dynamic track information, the audio signal comprising information about a moving sound source and the dynamic track information indicating motion of a position of the moving sound source; and

a decoder which uses the processor which decodes the audio signal and the dynamic track information from the received signal,

wherein the dynamic track information comprises control points which express a dynamic track of the moving sound source and the number of frames to which the dynamic track expressed by the control points is applied.

52. The apparatus of claim 51, further comprising an output distributor which distributes output to a plurality of speakers so as to correspond to the dynamic track information.

53. The apparatus of claim 51, wherein the decoder changes a frame rate of the audio signal by using the dynamic track information.

54. The apparatus of claim 51, wherein the decoder changes a number of channels of the audio signal by using the dynamic track information.

55. The apparatus of claim 51, wherein the decoder searches the audio signal for a period corresponding to a predetermined motion property of the moving sound source by using the dynamic track information.

56. The apparatus of claim 55, wherein:

the dynamic track information comprises a plurality of points for expressing the dynamic track; and

the decoder searches the audio signal by using the plurality of points.

57. The apparatus of claim 56, wherein:

the decoder searches the audio signal by using the number of the frames comprised in the dynamic track information.

58. An apparatus for encoding an audio signal, the apparatus comprising:

a processor;

a receiver which separately receives an audio signal and a reverberation property of the audio signal, the reverberation property being initially separately recorded from the audio signal;

an obtainer which obtains the audio signal based on the reverberation property; and

an encoder which uses the processor which encodes the obtained audio signal and the reverberation property.

59. The apparatus of claim 58, wherein:

the audio signal is recorded in a predetermined space; and

the reverberation property is of the predetermined space.

60. The apparatus of claim 58, wherein the reverberation property is indicated by an impulse response.

61. The apparatus of claim 60, wherein the encoder encodes the audio signal so that an initial reverberation period of the impulse response is expressed in a type of a high-degree infinite impulse response (IIR) filter, and a latter reverberation period of the impulse response is expressed in a type of a low-degree infinite impulse response filter.

62. An apparatus for decoding an audio signal, the apparatus comprising:

a processor;

a receiver which receives a signal comprising an encoded first reverberation property and an encoded audio signal comprising the first reverberation property, the encoded first reverberation property being initially separately recorded from the encoded audio signal;

a decoder which uses the processor which decodes the audio signal from the received signal; and

a generator which generates the decoded audio signal based on the encoded audio signal and the first reverberation property.

63. The apparatus of claim 62, further comprising a reverberation remover which decodes the first reverberation property from the received signal, calculates a reversed function of the first reverberation property, and obtains an audio signal from which the first reverberation property is removed by applying the reversed function to the audio signal comprising the first reverberation property.

64. The apparatus of claim 63, further comprising a reverberation applier which receives a second reverberation property, and generates an audio signal comprising the second reverberation property by applying the received second reverberation property to the audio signal from which the first reverberation property is removed.

65. The apparatus of claim 64, wherein the receiver receives the second reverberation property input by a user from an input device, or receives the second reverberation property that is previously stored in a memory, from the memory.

66. The apparatus of claim 62, wherein:

the audio signal is recorded in a predetermined space; and

the first reverberation property is of the predetermined space.

67. An apparatus for encoding an audio signal, the apparatus comprising:

a processor;

a receiver which receives an audio signal recorded in a predetermined space, and a reverberation property of the predetermined space, the reverberation property being initially separately recorded from the audio signal;

a reverberation remover which calculates a reversed function of the reverberation property, and obtains an audio signal from which the reverberation property is removed by applying the reversed function to the received audio signal; and

an encoder which uses the processor which encodes the reverberation property and the audio signal from which the reverberation property is removed.

68. An apparatus for decoding an audio signal, the apparatus comprising:

a processor;

a receiver which receives a signal comprising an encoded audio signal and an encoded reverberation property, the encoded audio signal being initially separately recorded from the encoded audio signal;

a decoder which uses the processor which decodes the audio signal and the reverberation property from the received signal; and

a reverberation restorer which obtains an audio signal comprising the reverberation property by applying the decoded reverberation property to the decoded audio signal.

69. An apparatus for decoding an audio signal, the apparatus comprising:

a processor;

a receiver which receives a second reverberation property and a signal comprising an encoded audio signal and an encoded first reverberation property, the encoded audio signal being initially separately recorded from the encoded first reverberation property;

a decoder which uses the processor which decodes the audio signal from the received signal; and

a reverberation applier which generates an audio signal comprising the second reverberation property by applying the second reverberation property to the audio signal and generates another audio signal comprising the first reverberation property by applying the first reverberation property to the audio signal.

70. An apparatus for encoding an audio signal, the apparatus comprising:

a processor;

a receiver which, for each of a plurality of semantic objects of an audio signal, receives at least one parameter indicating at least one property of a semantic object of the audio signal; and

an encoder which uses the processor which, for each of the plurality of semantic objects of the audio signal, encodes the at least one parameter,

wherein, for each of the plurality of semantic objects of the audio signal, the at least one parameter comprises a physical model comprising a transfer function to express a repeated creation and/or extinction of a sound source and indicates a physical property of the sound source corresponding to the semantic object.

71. The apparatus of claim 70, wherein the at least one parameter further comprises at least one of:

a note list which indicates pitch and beat of the semantic object; and

an actuating signal which actuates the semantic object.

72. The apparatus of claim 71, wherein the transfer function is a ratio between an output signal and the actuating signal in a frequency domain, with regard to the semantic object.

73. The apparatus of claim 71, wherein the encoder encodes a coefficient in a frequency domain of the actuating signal.

74. The apparatus of claim 71, wherein the encoder encodes coordinates of a plurality of points in a time domain of the actuating signal.

75. The apparatus of claim 70, wherein the at least one parameter comprises position information indicating a position of the semantic object.

76. The apparatus of claim 70, wherein the at least one parameter comprises spatial information indicating a reverberation property of a space where the audio signal of the semantic object is generated.

77. The apparatus of claim 70, wherein:

the receiver receives spatial information indicating a reverberation property of a space where the audio signal is generated; and

the encoder encodes the at least one parameter comprising the spatial information.

78. The apparatus of claim 76, wherein the spatial information comprises an impulse response exhibiting the reverberation property.

79. An apparatus for decoding an audio signal, the apparatus comprising:

a processor;

a receiver which, for each of a plurality of semantic objects of the audio signal, receives an input signal comprising at least one encoded parameter indicating at least one property of the semantic object of the audio signal; and

a decoder which uses the processor which, for each of the plurality of the semantic objects of the audio signal, decodes the at least one encoded parameter from the input signal,

wherein, for each of the plurality of the semantic objects of the audio signal, the at least one encoded parameter comprises a physical model comprising a transfer function to express a repeated creation and/or extinction of a sound source and which indicates a physical property of the sound source corresponding to the semantic object.

80. The apparatus of claim 79, further comprising a restorer which restores the audio signal by using the at least one parameter.

81. The apparatus of claim 79, wherein the at least one parameter further comprises at least one of:

a note list which indicates pitch and beat of the semantic object; and

an actuating signal which actuates the semantic object.

82. The apparatus of claim 79, wherein the at least one parameter further comprises position information indicating a position of the semantic object.

83. The apparatus of claim 82, further comprising an output distributor which distributes output to a plurality of speakers so as to correspond to the dynamic track information.

84. The apparatus of claim 79, wherein the at least one parameter further comprises spatial information indicating a reverberation property of a space where the audio signal of the semantic object is generated.

85. The apparatus of claim 79, wherein:

the input signal further comprises encoded spatial information indicating a reverberation property of a space where the audio signal is generated; and

the decoder decodes the spatial information from the input signal.

86. The apparatus of claim 85, further comprising a restorer which restores the audio signal by using the at least one parameter and the spatial information.

87. The apparatus of claim 79, further comprising a processor which processes the at least one parameter.

88. The apparatus of claim 87, wherein the processor comprises a searcher which searches for a parameter corresponding to a predetermined audio property from among the at least one parameter.

89. The apparatus of claim 87, wherein the processor comprises an editor which edits the at least one parameter.

90. The apparatus of claim 89, further comprising a generator which generates an edited audio signal by using the edited parameter.

91. The apparatus of claim 89, wherein the editor deletes the semantic object from the audio signal, inserts a new semantic object into the audio signal, or replaces the semantic object of the audio signal with the new semantic object.

92. The apparatus of claim 89, wherein the editor deletes the at least one parameter, inserts a new parameter into the audio signal, or replaces the at least one parameter with the new parameter.