Object oriented audio coding

• US 6,108,626 A
• Filed: 05/14/1998
• Issued: 08/22/2000
• Est. Priority Date: 10/27/1995
• Status: Expired due to Term

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation), Robert Bosch GmbH, CSELT Centro Studi e Laboratori Telecomunicazioni SpA (Telecom Italia S.p.A.)
• Original Assignee: Robert Bosch GmbH, CSELT Centro Studi e Laboratori Telecomunicazioni SpA (Telecom Italia S.p.A.)
• Inventors: Muller, Jorg, Cellario, Luca, Festa, Michele, Sereno, Daniele
• Primary Examiner(s): Knepper, David D.
• Application Number:
  US09/068,136
• Time in Patent Office: 831 Days
• Field of Search: 704/200, 704/201, 704/205, 704/206, 704/219-223, 704/229, 704/230, 704/254, 704/255, 704/270
• US Class Current: 704/230
• CPC Class Codes: H04B 1/667 using a division in frequency subbands for TV signals H04N19/63
• CPC Class Codes: H04B 14/046 Systems or methods for reducing noise or bandwidth

Abstract

Audio sources are coded by recognizing different classes of audio such as speech and music. The classes are used to select between coding algorithms and to provide object definitions. Objects have abstract and concrete classes which may further rely on parameters produced by linear prediction and subband filters to provide a frame-based bit stream of information. Each object in the bit stream has layers of information such as basic bit rate, coding parameters and enhancement parameters. The layers of information in each object allow altering selected parameters to manipulate audio signals.

Claims

1. A method of processing digitized audio signals, comprising at least:

a coding phase in which a signal to be processed, organized into a sequence of frames comprising a predetermined number of samples, is split into a plurality of frequency bands which can be independently coded, and a coded signal is generated including information relevant to signals in at least selected ones of said frequency bands, the coding taking place according to an embedded coding technique such that the coded signal comprises a basic layer, containing the minimum amount of information ("core information") needed for decoding and corresponding to a minimum bit rate, a total layer, containing the whole of the coded information and corresponding to a maximum bit rate, and a plurality of intermediate layers which contribute to the coded signal by respective information blocks ("enhancement information") coding respective signal portions that cannot be represented by the only core information, and which cause an increase of the bit rate of the coded signal by successive steps from the basic layer to the total layer, the basic layer being generated by a first coding step and each block of enhancement information being generated by a respective second coding step; and

a decoding phase, in which the information relevant to the different frequency bands included in the coded signal is independently decoded, in such a manner that for a frequency band for which both enhancement information blocks and the core information are to be decoded, the coded signals are submitted to a set of first decoding steps, the number of which is the same as that of the second coding steps performed for that band and in each of which one enhancement information block is decoded, and to a second decoding step in which the core information is decoded, whereas for a frequency band for which only the core information is to be decoded, the coded signals are submitted to the second decoding step only; and

the decoded signals relevant to the different bands are recombined to build a reconstructed signal with bandwidth characteristics corresponding to those of the original signal;

characterized in that, during said coding phase, a two-stage classification is performed by which each audio signal to be coded in a given frame is allotted to one out of a plurality of abstract and to one out of a plurality of concrete classes of said one abstract class, the concrete classes being related with the characteristics of a signal portion and identifying elementary audio objects present in the frame and the abstract classes being related with the nature of an audio signal and identifying macro-objects resulting from a combination of elementary audio objects;

in that said first coding step for a given audio object is performed by means of a first coding algorithm selected out of a plurality of first coding algorithms and any second coding step for that given audio object is performed by means of a respective second coding algorithm selected out of a plurality of second coding algorithms, the choice amongst the plurality of said first and respectively second coding algorithms depending at least on the results of said two step classification;

the coding phase generating, for each object, an object bit stream, containing all information relevant to a same concrete class for that audio signal in that frame, and a macro-object bit stream combining bit streams of different objects of a same to abstract class or different abstract classes and having bit-rate and bandwidth characteristics which depend on the choices made for said first and said second algorithms and on configuration information passed from a user equipment (US) to coding devices (AC) and/or on control information passed from a transmission system (SY) to the coding device;

in that the method further comprises, between the coding and decoding phases, a phase of manipulation of the bit stream generated by said coding phase, for the scaling of the coded bit stream in dependence of information about the abstract and concrete classes, included in the coded bit stream, and of said configuration and control information;

and in that in said decoding phase, said first decoding step is performed by means of a respective algorithm complementary to the second algorithm selected in the coding phase to generate the enhancement information block to be decoded in that step, and the second decoding step is performed according to an algorithm complementary to the first algorithm selected in the first coding step;

each of said first and second decoding algorithms being selected out of a plurality of first and second decoding algorithms, complementary each to one of said second and first coding algorithms, respectively, according to information provided with the abstract and concrete class and/or configuration information provided in a set up phase.

2. A method as claimed in claim 1, characterized in that the first and second algorithms are independently selected for different bands.

3. A method as claimed in claim 1 characterized in that the second coding algorithms code a quantization error obtained as a result of the application of the first coding algorithm.

4. A method as claimed in claim 1 characterized in that the first algorithms include linear prediction analysis coding algorithms at least for signals of a lower set of frequency bands, and shape/gain vector quantization coding algorithms for signals of higher frequency bands and for signals where linear prediction is not exploited.

5. A method as claimed in claim 1, characterized in that the second algorithms include shape/gain vector quantization coding algorithms.

6. A method as claimed in claim 1, characterized in that the selection of the first and second algorithm is carried out in dependence of configuration information passed from a user equipment (US) to coding devices (AC) and/or of control information passed from a transmission system (SY) to the coding devices.

7. A method as claimed in claim 1, wherein, before coding, each frame is divided into a plurality of subframes, characterized in that the length of said subframes is selectable out of a plurality of length values, so as to obtain a plurality of possible values of the coding delay.

8. A method as claimed in claim 1, characterized in that said frequency bands have a predetermined bandwidth, independently of a sampling frequency of the signal to be coded.

9. A method as claimed in claim 1, characterized in that the signal to be coded is sampled at any arbitrary input sampling frequency from 8 to 64 kHz and, before coding, it is upsampled to an internal sampling frequency which is the power of 2 immediately higher than the input sampling frequency;

and in that this upsampling is disabled for input sampling frequencies of 8, 16 and 32 kHz.

10. A method as claimed in claim 1, characterized in that, for speech signal processing, the coding phase comprises the steps of:

selecting a standard-defined speech coding algorithm as first coding algorithm for a whole set of frequency bands;

building the basic layer with the core information generated by submitting the speech signal to the standard-defined algorithm; and

building a coded signal corresponding to one of said intermediate layers or to the total layer, so as to obtain a coded signal upgraded with respect to the standard-defined coded signal;

and in that the decoding phase comprises the steps of a) decoding the only basic layer, or b) decoding the whole of the coded signal, depending on the availability of decoding algorithms and/or the quality to be attained for the decoded signal.

11. A method as claimed in claim 1, characterized in that the selection of the frequency bands to be submitted to at least the first coding step, the selection of the bands for which also second coding steps are to be performed and the number of second coding steps for a given frequency band are determined in dependency of the bandwidth and bit rate desired for the coded signal and on requirements of a user equipment (US) and of a system (SY) in which the coded signal is exploited, independently of the bandwidth and sampling frequency of the signal to be coded, on a frame per frame basis.

12. A method as claimed in claim 1, characterized in that the selection of the frequency bands to be submitted to the first coding step is carried out by the following operations:

a) determining a total bandwidth allocable to the coded signal for the available bit rate;

b) determining the energy associated to each band included in said bandwidth, and comparing said energy with a respective first energy threshold;

c) enabling insertion of core information for all bands of which the energy exceeds the respective threshold.

13. A method as claimed in claim 12, characterized in that the thresholds are constant thresholds.

14. A method as claimed in claim 12, characterized in that the thresholds are thresholds determined by exploiting a perceptual model.

15. A method as claimed in claim 1, characterized in that the selection of the bands to which said second coding steps are to be applied and the determination of the number of second coding steps for a given band comprise the following operations:

d) applying the whole of the second coding steps to each band;

e) determining the increase in signal quality afforded by each second step with respect to a previous one or to the first coding step in the same band;

f) enabling insertion into the coded signal of enhancement information blocks provided by second coding steps which allow the highest quality increases, until the available bit rate is completely exploited.

16. A method as claimed in claim 15, characterized in that said evaluation of the quality increase is carried out by exploiting a perceptual model.

17. A method as claimed in claim 1, characterized in that said audio signals are the audio component of multimedia signals.

18. A method as claimed in claim 1, characterized in that the audio signals to be coded are submitted to a said two-stage classification by which the signal in a given frame is allotted to one out of a plurality of abstract classes and to one out of a plurality of concrete classes of said one abstract class, the classification being used as control parameter for said splitting into frequency bands, said selection of the frequency bands to which a first coding algorithm and possibly a second coding algorithm are to be applied, and said determination of the number of second coding steps for a given frequency band.

19. A method according to claim 1, characterized in that said two-stage classification is made automatically.

20. A method according to claim 1, characterized in that said two-stage classification is driven by information passed from a user equipment (US).

21. A method according to claim 1, characterized in that said object bit streams are made up by packets of bits produced by individual coding steps and said macro-object bit stream (OB11 . . . 0821_--) comprises:

a first group of overhead bits (OVH1, OVH2) containing information regarding the classification results and the frequency bands being submitted to at least the first coding step;

the packets of the core information; and

, if second coding steps have been performed, a second group of overhead bit (OVH3) containing information regarding the number of coding steps performed for the different frequency bands having been submitted to at least the first coding step, and the packets of the enhancement information blocks; and

in that bit streams of different macro-objects (OB11 . . .

     0821) coded in the frame are transmitted in sequence, the transmission being preceded by a configuration phase in which a further group of overhead bits (OVHO) is transmitted, which group contains all service information necessary for the configuration of a decoder (AD).

22. A method as claimed in claim 21, characterized in that the bit packets within a macro-object bit stream are ordered by rank, the rank being related to the energy values of the different bands, in case of packets belonging to core information, and to the quality increase, in case of packets belonging to enhancement information.

23. A method as claimed in claim 22, characterized in that said scaling comprises the following steps:

a1) determining a bandwidth allocable in the frame to the or each macro-object for a desired bit rate;

b1) eliminating bit packets relevant to frequency bands which cause an exceeding of said bandwidth;

c1) if the residual bit rate exceeds the desired bit rate, eliminating one block of enhancement information for each band, starting from the band with the highest frequency, until the desired bit rate is attained or the core information only is left, the elimination being cyclically repeated, if necessary;

d1) if the residual bit rate at the end of step c1) still exceeds the desired bit rate, eliminating core packets of one or more frequency bands, starting from the highest frequency one.

24. A method as claimed in claim 1, characterized in that it further comprises a bit stream manipulation phase, carried out at one or a plurality of locations along a transmission path (1) between a coder (AC) and a decoder (AD), for stepwise bit rate and bandwidth scaling of the coded signal, each scaling step corresponding to suppression of one enhancement information block or, when no enhancement information block is present in the coded signal, to suppression of the core information of a frequency band.

25. A method as claimed in claim 1, characterized in that said bit stream manipulation phase is carried out at one or a plurality of locations upstream of a decoder (AD), and provides for a stepwise bit rate or bandwidth scaling of the coded signal, each scaling step corresponding to suppression of one enhancement information block or, when no enhancement information block is present in the coded signal, to suppression of the core information of a frequency band.

26. A method as claimed in claim 24, characterized in that said scaling comprises the following steps:

a2) determining a total bandwidth allocable in the frame to the or each macro-object for a desired bit rate;

b2) eliminating bit packets relevant to frequency bands which cause an exceeding of said bandwidth;

c2) if the residual bit rate exceeds the desired bit rate, eliminating one packet of enhancement information at a time, starting from the packet with the lowest rank, until the desired bit rate is attained or the core information packets only are left;

d2) if the residual bit rate at the end of step c2) still exceeds the desired bit rate, eliminating core information of one or more frequency bands, starting from the lowest rank band.

27. A method as claimed in claim 24, characterized in that said bit stream manipulation phase comprises the step of altering the value of predetermined coded parameters in individual macro-objects.

28. A method as claimed in claim 24, characterized in that said bit stream manipulation phase is performed simultaneously on a plurality of concurrent macro-objects, in such a way as to allot different bit rates and/or bandwidths to different macro-objects.

29. A method as claimed in claim 28, characterized in that said bit stream manipulation phase comprises the step of building a single bit stream at a predetermined bit rate including contributions from a plurality of concurrent macro-objects of which the overall bit rate exceeds the predetermined bit rate.

30. A method as claimed in claim 28, characterized in that said bit stream manipulation phase comprises the step of performing a level manipulation on the individual macro-objects, and the different bit rates and/or bandwidths allotted to the individual macro-objects are selected on the ground of that level manipulation.

31. A method as claimed in claim 24, characterized in that said bit stream manipulation phase is carried out on individually selected frames.

32. Apparatus for processing digitized audio signals, comprising:

an encoder (AC) arranged to receive frames of samples of an audio signal to be coded, having given bandwidth characteristics, and comprising;

filtering means (FB1, FB2, FB3) for splitting the signal to be coded into a plurality of frequency bands, coding units (LCC, HCC, LEC, HEC) associated to each frequency band for the embedded coding of the signals of that band and comprising, for each band, a first coding unit (LCC, HCC), enabled for at least selected ones of the frequency bands and generating at each frame a core information for the respective band, and a set of second coding units (LCC, HCC), intended to generate a succession of enhancement information blocks for that band, the core information being the minimum amount of information needed for signal decoding; and

means (BCU) for combining coded signals of the different frequency bands into a single embedded coded signal which comprises a basic layer, containing the core information of said selected frequency bands and corresponding to a minimum bit rate, a total layer, containing the whole of the coded information and corresponding to a maximum bit rate, and a plurality of intermediate layers which contribute to the coded signal by respective enhancement information blocks and cause an increase of the bit rate of the coded signal by successive steps from the basic layer to the total layer, and

a decoder (AD) comprising;

decoding units (LED, HED, LCD, HCD) for independently decoding the coded signal of the different frequency bands, and comprising, for each frequency band, a set of first decoding units (LED, HED), in one to one correspondence with the coding units of said second set (LEC, HEC) and intended each to decode an enhancement information block, and a second decoding unit (LCD, HCD) intended to decode the core information; and

synthesis filtering means (FB4, FB5, FB6) for recombining the decoded signals of the different frequency bands and reconstructing a decoded signal with bandwidth characteristics corresponding to that of the original audio signal;

characterized in that

the first coding unit (LCC, HCC) and each second coding unit (LEC, HEC) are configurable so as to apply to the signal being coded a first or respectively a second coding algorithm selected out of a plurality of first and second coding algorithms and each first decoding unit (LED, HED) and the second decoding unit (LCD, HCD) are configurable so as to apply to the signal being decoded a first or respectively a second decoding algorithm complementary to the second and the first coding algorithm, respectively, applied by the second and first coding units (LEC, HEC, LCC, HCC);

and in that it, further comprises;

a classification unit (CR) for submitting the audio signal to be coded to a two stage classification by which the signal is a given frame is allotted to one out of a plurality of abstract classes and to one out of a plurality of concrete classes of said one abstract class, the concrete classes being related with the characteristics of a signal portion and identifying elementary audio objects present in the frame and the abstract classes being related with the nature of to an audio signal and identifying macro-objects resulting from a combination of elementary audio objects;

the classification unit (CR) providing the information on the classification to the filtering means (FB1 . . . FB3) and to said first and second coding units (LCC, HCC, LEC, HEC) as control parameter for said splitting into frequency bands, the enabling of selected first and second coding units (LCC, HCC, LEC, HEC) and the selection of a proper coding algorithm by the or each coding unit, and to said combining means (BCU) for insertion into the coded bit stream; and

at least one bit stream manipulation unit (BMU), located upstream the decoder (AD), for bit rate or bandwidth scaling of the coded signal relevant to individual macro-objects and/or objects.

33. An apparatus as claimed in claim 32, characterized in that said encoder (AC) receives a signal sampled at any arbitrary input sampling frequency from 8 to 64 kHz, and further comprises means (MXU) for upsampling said signal at an internal sampling frequency which is the power of 2 immediately higher than the input sampling frequency;

said upsampling means (MXU) being disabled for input sampling frequencies of 8, 16 and 32 kHz.

34. An apparatus as claimed in claim 32, characterized in that said filtering means (FB1 . . . FB3) in the encoder (AC) are configurable so as to generate frequency bands of predetermined bandwidth, independently of the sampling frequency of the signal to be coded.

35. An apparatus as claimed in claim 32, wherein the encoder (AC) comprises means (SF1, SF2) for dividing each frame into a plurality of subframes of which the duration is a submultiple of the frame duration, characterized in that the means (SF1, SF2) for dividing the frames are configurable so as to generate subframes of which the duration is selectable out of a plurality of duration values, so as to obtain a plurality of possible values of the coding delay.

36. An apparatus as claimed in claim 32, characterized in that the first and second coding units (LCC, HCC, LEC, HEC) of each band are configurable independently of the coding units of the other bands.

37. An apparatus as claimed in claim 32, characterized in that the first coding units (LCC, HCC) are configurable so as to apply linear prediction analysis coding algorithms at least for signals of lower set of frequency bands, and shape/gain vector quantization coding algorithms for signals of higher frequency bands and for signals where linear prediction is not exploited.

38. An apparatus as claimed in claim 32, characterized in that the second coding units (LEC, HEC) associated with a frequency band code a quantization error obtained as a result of the application of the first coding algorithm to signals in the same frequency band.

39. An apparatus as claimed in claim 32, characterized in that the second coding units (LEC, HEC) are configurable so as to apply shape/gain vector quantization coding algorithms to all frequency bands.

40. An apparatus as claimed in claim 32, characterized in that the first and second coding units (LCC, HCC, LEC, HEC) are enabled and configured, on a frame per frame basis, in dependency of the bandwidth and bit rate desired for the coded signal and of requirements of a user (US) and of a transmission system (SY), independently of the bandwidth and sampling frequency of the signal to be coded.

41. An apparatus as claimed in claim 32, characterized in that, for speech signal processing:

a first coding unit (LCC) is configured so as to apply a standard-defined speech coding algorithm to a whole set of frequency bands;

said combining means (BCU) are arranged to build the basic layer with the information generated by the application of the standard-defined algorithm and to build a coded signal corresponding to one of said intermediate layers or to the total layer, so as to obtain a coded signal upgraded with respect to the standard-defined coded signal;

and in that the decoding units (LED, HED, LCD, HCD) are so configured as to decode either the only basic layer or the whole of the coded signal, depending on the availability of decoding algorithms and/or the quality to be attained for the decoded signal.

42. An apparatus as claimed in claim 32, characterized in that the enabling of said first and second coding units (LCC, HCC, LEC, HEC) is controlled by said combining means (BCU).

43. An apparatus as claimed in claim 32, characterized in that, for the enabling of the first coding units (LCC, HCC), said combining means (BCU) comprise means (BCL) which evaluate the energy of the signals in the different frequency bands and enable only first coding units (LCC, HCC) associated with bands where the energy exceeds a predetermined threshold.

44. An apparatus as claimed in claim 43, characterized in that said threshold is calculated by a processing unit (PMP) by exploiting a perceptual model.

45. An apparatus as claimed in claim 32, characterized in that, for the enabling of the second coding units (LEC, HEC), said combining means (BCU) comprise means (BCL) which evaluate an increase in the quality of the coded signal at the output of each second coding unit (LEC, HEC) with respect to the quality at the output of an immediately preceding second coding unit (LEC, HEC) or of the first coding unit (LCC, HCC) associated to the same band, only second coding units affording the highest quality increases being enabled.

46. An apparatus as claimed in claim 44 characterized in that said means (BCL) for the quality increase evaluation exploit information on a perceptual model provided by said processing unit (PMP).

47. An apparatus as claimed in claim 32, characterized in that said combining means (BCU) are arranged to combine, into an object bit stream, packets of bits produced by individual coding units (LCC, HCC, LEC, HEC) and containing all information relevant to a given concrete class for a single audio signal in one frame, and to combine the bit stream of the different objects of said audio signal in said frame into a macro-object bit stream (OB11 . . . OB21) which comprises:

a first group of overhead bits (OVH1, OVH2) containing information regarding the classification results and the frequency bands for which at least the first coding unit (LCC, HCC) is enabled;

the packets of the core information; and

, if second coding units (LEC, HEC) are enabled, a second group of overhead bits (OVH3) containing information regarding the number of coding units enabled for the different frequency bands and the packets of the enhancement information blocks; and

in that said combining means (BCU) are arranged to transmit, in a set up phase of a communication, all service information necessary for configuring the decoder (AD) in the same manner as the coder (AD), and then, during the communication, to transmit in sequence the bit streams of different macro-objects coded in the frame.

48. An apparatus as claimed in claim 47, characterized in that the combination means (BCU) are arranged to transmit the bit packets within a macro-object bit stream (OB11 . . . OB21) in an order of frequency band, starting with the lowest frequency band.

49. An apparatus as claimed in claim 48, characterized in that the combining means (BCU) are arranged to transmit the bit packets within a macro-object bit stream (OB11 . . . OB21) in an order or rank starting with those of the highest rank, the rank being related to the energy values of the different bands, in case of packets belonging to core information, and to the entity of the quality increase, in case of packets belonging to enhancement information.

50. An apparatus as claimed in claim 32, characterized in that said manipulation unit (BMU) is arranged to perform a stepwise scaling of the bit rate or the bandwidth, by suppressing a bit packet at each scaling step, starting from the packets provided by the second coding units (LEC, HEC) and continuing with packets provided by first coding units (LCC, HCC), when no packet provided by the second coding units (LEC, HEC) is present in the bit stream.

51. An apparatus as claimed in claim 50, characterized in that said bit stream manipulation unit (BMU) is arranged to manipulate the bit streams of individual objects or macro-objects in individually selected frames.

52. An apparatus as claimed in claim 50, characterized in that, for said scaling, said bit stream manipulation unit (BMU) is arranged to:

determine a bandwidth allocable in the frame to the or each macro-object for a desired bit rate;

eliminate bit packets relevant to frequency bands which lie beyond said bandwidth;

eliminate one bit packet provided by a second coding unit (LEC, HEC) for each of the frequency bands which have been kept, starting from the band with the highest frequency, until the desired bit rate is attained or the core information only is left, the elimination being cyclically repeated, if necessary;

eliminate the bit packet provided by a first coding unit (LCC, HCC), for one or more frequency bands, starting from the highest frequency one.

53. An apparatus as claimed in claim 50, characterized in that, for said scaling, said bit stream manipulation unit (BMU) is arranged to:

determine a bandwidth allocable in the frame to the or each macro-object for a desired bit rate;

eliminate bit packets relevant to frequency bands which lie beyond said bandwidth;

eliminate bit packets provided by second coding units (LEC, HEC), starting from the packet of lowest rank;

eliminate the bit packet provided by a first coding unit (LCC, HCC), for one or more frequency bands, starting from the lowest energy band.

54. An apparatus as claimed in claim 50, characterized in that said bit stream manipulation unit (BMU) is arranged to alter the values of predetermined coded parameters in individual macro-objects.

55. An apparatus as claimed in claim 50, characterized in that said bit stream manipulation unit (BMU) is arranged to build a single bit stream at a predetermined bit rate including contributions from a plurality of concurrent macro-objects of which the overall bit rate exceeds the predetermined bit rate.

56. An apparatus as claimed in claim 55, characterized in that said bit stream manipulation unit (BMU) is arranged to manipulate the level of the individual macro-objects before building the single bit stream, and to select a bit rate and/or a bandwidth for the contribution of the individual macro-objects on the ground of the manipulated level.