Scalable and perceptually ranked signal coding and decoding

• US 20020176353A1
• Filed: 08/22/2001
• Published: 11/28/2002
• Est. Priority Date: 05/03/2001
• Status: Active Grant

• Current Assignee: University of Washington, Washington University In St Louis
• Original Assignee: University of Washington
• Inventors: Atlas, Les E., Vinton, Mark S.
• Granted Patent: US 7,136,418 B2
• Time in Patent Office: Days
• US Class Current: 370/203
• CPC Class Codes: G10L 19/02 using spectral analysis, e.g. transform vocoders or subband vocoders

Abstract

A method and system for encoding and decoding an input signal in relation to the most perceptually relevant aspects of the input signal. A two-dimensional (2D) transform is applied to the input signal to produce a magnitude matrix and a phase matrix that can be inverse quantized by a decoder. A first column of coefficients of the magnitude matrix represents a mean spectral density (MSD) function of the input signal. Relevant aspects of the MSD function are encoded at a beginning of a data packet. The MSD function is also processed through a core perception model to determine bit allocation. The matrices are then quantized and priority ordered into a data packet, with the least perceptually relevant information at the end of the packet so that it may be ignored or truncated for scalability to the channel data rate capacity.

Claims

1. A method for encoding a signal for storage or transmission, comprising the steps of:

(a) implementing a two-dimensional transform of the signal, producing a transform matrix having modulation frequency as one dimension;

(b) reducing a dynamic range of the signal;

(c) quantizing and selecting coefficients included in the transform matrix; and

(d) producing data packets in which the coefficients that have been selected are encoded based upon a desired order of the coefficients, with coefficients that are more perceptually relevant being used first to fill each data packet and coefficients that are less perceptually relevant being handled in one of the following ways;

(i) discarded once an available space in each data packet that is to be stored or transmitted has been filled with the coefficients that are more perceptually relevant; and

(ii) disposed last within each data packet, so that the coefficients that are less perceptually relevant can subsequently be truncated from the data packet.

2. The method of claim 1, wherein the step of implementing the two-dimensional transform produces a phase matrix and a magnitude matrix, said magnitude matrix having the modulation frequency as one of its dimensions and including a mean spectral density function of the signal.

3. The method of claim 2, further comprising the step of determining weighting factors for use in reducing the dynamic range of the signal, based upon a perceptual model.

4. The method of claim 3, wherein the step of determining the weighting factors comprises the step of using the perceptual model to compute weighting factors from coefficients of the mean spectral density function.

5. The method of claim 3, further comprising the steps of converting the weighting factors to a logarithmic scale;

and thereafter, quantizing the weighting factors.

6. The method of claim 5, further comprising the step of inverse quantizing the weighting factors to produce inverse quantized weighting factors.

7. The method of claim 6, further comprising the steps of preparing the mean spectral density function for quantization using the inverse quantized weighting factors;

and quantizing the means spectral density function thus prepared, producing a quantized mean spectral density function.

8. The method of claim 7, wherein the quantized mean spectral density function is also encoded into the data packets.

9. The method of claim 7, further comprising the step of producing an inverse quantized mean spectral density function.

10. The method of claim 9, further comprising the step of processing the inverse quantized mean spectral density function with the perceptual model to produce bit allocations used for encoding the data packets.

11. The method of claim 10, further comprising the step of quantizing the phase matrix and the magnitude matrix using a number of bits determined by the perceptual model.

12. The method of claim 2, further comprising the step of quantizing the magnitude matrix to produce a quantized magnitude matrix, wherein the step of producing the data packets comprises the step of coding the quantized magnitude matrix with one of a fixed code and a variable length code.

13. The method of claim 1, wherein the step of producing the data packets includes the step of ordering the data corresponding to the signal with respect to their perceptual relevance so that data having lower modulation frequencies and lower base-transform frequencies are inserted into a data packet before data having higher modulation frequencies and higher base-transform frequencies.

14. The method of claim 1, wherein the step of implementing the two-dimensional transform includes the steps of:

(a) transforming even numbered window sequences by a discrete cosine transform to form an even transform sequence;

(b) transforming odd numbered window sequences by a discrete sine transform to form an odd transform sequence; and

(c) forming an orthogonal complex pair by combining the even transform sequence with the odd transform sequence.

15. The method of claim 14, further comprising the step of applying a second transform to the orthogonal complex pair.

17. The method of claim 16, wherein the step of applying the two-dimensional transform produces a phase matrix and a magnitude matrix, said magnitude matrix having the modulation frequency as one of its dimensions and including a mean spectral density function of the signal.

18. The method of claim 16, further comprising the step of processing the perceptual signal with a perceptual model before applying the two-dimensional transform.

19. The method of claim 16, further comprising the step of reducing a dynamic range of the perceptual signal that is encoded into the data packets.

20. The method of claim 16, wherein the step of determining the order comprises the step of ordering data for the perceptual signal so that lower modulation frequencies and lower base-transform frequencies are inserted closer to the beginning of the data packets than higher modulation frequencies and higher base-transform frequencies.

21. A machine readable medium on which are stored a plurality of machine readable instructions for carrying out the steps of claim 16.

16. A method for encoding data packets with data derived from a perceptual signal, said data packets being stored as originally encoded, or stored in a truncated form, or transmitted in a truncated form over a network at a data rate that may be less than required to transmit non-truncated data packets, comprising the steps of:

(a) applying a two-dimensional transform to the signal to produce a transform matrix having modulation frequency as one dimension;

(b) quantizing a mean spectral density derived from the transform matrix, to produce a quantized mean spectral density;

(c) determining an inverse quantized mean spectral density using the quantized mean spectral density;

(d) deriving bit allocations from the inverse quantized mean spectral density using a perceptual model;

(e) as a function of the bit allocations and the results of the two-dimensional transform, producing quantized components; and

(f) determining an order in which the perceptual data are loaded into each data packet, based upon the quantized components, wherein data that are perceptually more important are loaded closer to a beginning of the data packet, while data that are perceptually less important are handled in one of the following ways;

(i) loaded closer to an end of each data packet, if the entire data packet is to be stored in a non-truncated form; and

(ii) eliminated from the data packets, if said data packets are to be stored or transmitted over the network in the truncated form.

22. Apparatus for encoding data packets to include data derived from a perceptual signal, said data packets being, comprising:

(a) a memory in which a plurality of machine instructions are stored;

(b) a source of a perceptual signal to be encoded into data packets;

(c) a processor coupled in communication with the source of the perceptual signal, and the memory, said processor executing the machine instructions to carry out a plurality of functions, including;

(i) applying a two-dimensional transform to the perceptual signal, producing a transform matrix having modulation frequency as one dimension;

(ii) quantizing a mean spectral density of one component of the transform matrix, to produce a quantized mean spectral density;

(iii) determining an inverse quantized mean spectral density using the quantized mean spectral density;

(iv) deriving bit allocations from the inverse quantized mean spectral density using a perceptual model;

(v) as a function of the bit allocations and the transform matrix, producing quantized components; and

(vi) determining an order in which the perceptual data are loaded into each data packet, based upon the quantized components, so that data that are perceptually more important are loaded into a beginning of the data packet, while data that are perceptually less important are handled in one of the following ways;

(1) loaded closer to an end of each data packet; and

(2) eliminated from the data packets.

23. The apparatus of claim 22, wherein the two-dimensional transform produces a phase matrix and a magnitude matrix, said magnitude matrix having the modulation frequency as one of its dimensions and including a mean spectral density function of the perceptual signal.

24. The apparatus of claim 22, wherein the functions implemented by the processor when executing the machine instructions further comprise processing the perceptual signal with a perceptual model before applying the two-dimensional transform.

25. The apparatus of claim 22, wherein the functions implemented by the processor when executing the machine instructions further comprise reducing a dynamic range of the perceptual signal that is encoded into the data packets.

26. The apparatus of claim 22, wherein the function of determining the order arranges the data encoded into the data packet so that data having lower modulation frequencies and lower base-transform frequencies are inserted closer to the beginning of the data packets than data having higher modulation frequencies and higher base-transform frequencies.

27. The apparatus of claim 23, further comprising a network interface that is coupled to the processor and used to transmit the data packets over a network to a recipient device that receives at least portions of each data packet transmitted over the network, said recipient device including:

(a) a recipient memory in which a plurality of machine instructions are stored;

(b) a recipient network interface coupled to the network to receive encoded data packets; and

(c) a recipient processor that is coupled to the recipient network interface and to the recipient memory, said recipient processor executing the machine instructions stored in the recipient memory to carry out a plurality of functions for decoding each encoded data packet, including;

(i) decoding the mean spectral density and mean spectral density weights;

(ii) decoding template models from the encoded data packet;

(iii) decoding and reordering a magnitude content and a phase content from the encoded data packet;

(iv) inverse quantizing the magnitude matrix and the phase matrix;

(v) adding the template models to the inverse quantized magnitude matrix, said inverse quantized phase matrix and a result produced by thus adding comprising a two-dimensional transform;

(vi) inverting the two-dimensional transform; and

(vii) performing post processing to yield a pulse code modulated signal corresponding to the perceptual signal.

28. The apparatus of claim 27, wherein the recipient processor, when executing the machine instructions, also implements the following functions for each data packet that is received:

(a) converts the mean spectral density and mean spectral density weights to a decibel scale;

(b) produces a signal-to-mask ratio for each of a plurality of frequency bins as a function of the means spectral density and the mean spectral density weights; and

(c) computes a number of bits to be used in each frequency bin for a remaining magnitude matrix and a remaining phase matrix, such that a signal-to-noise ratio of the bits in the plurality of frequency bins is greater than the signal-to-mask ratio.

30. The method of claim 29, further comprising the step of enabling a bit resolution of the data packets that defines the capacity of the data packets to be selectively scaled as a function of the data transmission bandwidth of a network over which the data packets are transmitted.

31. The method of claim 29, further comprising the step of enabling a bit resolution of the data packets that defines the capacity of the data packets to be selectively scaled as a function of a capacity of the available storage in which the data packets are stored.

32. The method of claim 29, further comprising the step of decoding the data packets to recover the data to create a perceptible signal adapted to provide a perceptible experience by a user, loss of the perceptually less important data from the data packets being perceptually less noticeable in the perceptible signal as perceived by the user than would be a loss of perceptually more important data.

33. The method of claim 29, further comprising the step of storing data packets from which the perceptually less important data have been excluded in a storage.

34. The method of claim 29, further comprising the step of transmitting data packets from which the perceptually less important data have been excluded, over a network.

35. The method of claim 29, further comprising the step of truncating perceptually less important data from the data packets as necessary to accommodate an available data transmission rate for a network channel over which the data packets are to be transmitted.

36. The method of claim 35, wherein perceptually less important data are truncated from the data packet to achieve a data packet size sufficiently small so that the data packets are transmitted over the network channel in a continuous stream and so that the data conveyed by the data packets are adapted to be perceptually experienced by a user in real time as the data packets are received.

29. A method for perceptually ordering data within data packets that are sized as a function of either an available storage or an available data transmission bandwidth, comprising the steps of:

(a) determining a mean spectral density function of the data for inclusion in the data packets;

(b) determining a magnitude matrix and a phase matrix for the data;

(c) modeling the magnitude matrix;

(d) quantizing the magnitude matrix and the phase matrix for use in the data packets; and

(e) perceptually ordering the data included in the data packets, so that perceptually more important data are inserted first into each data packet, and perceptually less important data are inserted successively thereafter to ensure that an available capacity of the data packets is filled with perceptually more important data in preference to the perceptually less important data.