System and method for real-time jitter control and packet-loss concealment in an audio signal

• US 7,596,488 B2
• Filed: 09/15/2003
• Issued: 09/29/2009
• Est. Priority Date: 09/15/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: He, Li-Wei, Florencio, Dinei, Chou, Philip
• Primary Examiner(s): Lerner; Martin
• Application Number:
  US10/663,390
• Publication Number:
  US 20050058145A1
• Time in Patent Office: 2,206 Days
• Field of Search: 704/208, 704/211, 704/214, 704/218, 704/503, 704/219, 370/521, 370/516
• US Class Current: 704/208
• CPC Class Codes: G10L 19/005 Correction of errors induced by the transmission channel, if related to the coding algorithm
• CPC Class Codes: H04N 21/4392 involving audio buffer management
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/4396 by muting the audio signal
• CPC Class Codes: H04N 21/4398 involving reformatting operations of audio signals details of audio signal transcoding G10L19/173

Abstract

An “adaptive audio playback controller” operates by decoding and reading received packets of an audio signal into a signal buffer. Samples of the decoded audio signal are then played out of the signal buffer according to the needs of a player device. Jitter control and packet loss concealment are accomplished by continuously analyzing buffer content in real-time, and determining whether to provide unmodified playback from the buffer contents, whether to compress buffer content, stretch buffer content, or whether to provide for packet loss concealment for overly delayed or lost packets as a function of buffer content. Further, the adaptive audio playback controller also determines where to stretch or compress particular frames or signal segments in the signal buffer, and how much to stretch or compress such segments in order to optimize perceived playback quality.

Claims

1. A system for providing adaptive playback of an audio signal received across a packet based network, comprising:

storing data packets comprising a received audio data signal to a signal buffer;

outputting parts of the signal present in the signal buffer as needed for signal playback;

analyzing the data packets contained in the signal buffer to determine whether any data packets are missing, having not been received into the signal buffer by a predetermined packet late loss time;

specifying a maximum delay period, extending past the predetermined packet late loss time, for receiving any missing data packets;

following the predetermined packet late loss time, stretching at least part of the signal preceding the missing data packets present in the signal buffer, until any of receiving the missing data packets and exceeding the maximum delay period, when the analysis of the contents of the signal buffer indicates that the length of the signal in the signal buffer is less than a predetermined threshold; and

compressing at least part of the signal present in the signal buffer when the analysis of the contents of the signal buffer indicates that the length of the signal in the signal buffer is greater than a predetermined threshold.

2. The system of claim 1 wherein analyzing the contents of the signal buffer includes determining a type of the contents of the signal buffer from among a group including:

periodic content, quasi-periodic content, aperiodic content and mixed content.

3. The system of claim 2 wherein stretching at least part of the signal having any of periodic content and quasi-periodic content type comprises:

identifying at least one of the segment of the content of the signal buffer as a template;

searching for a matching segment in portions of the content of the signal buffer whose cross correlation peak exceeds a predetermined threshold; and

inserting the template into the content of the signal buffer, and aligning and merging the matching segments.

4. The system of claim 2 wherein stretching at least part of the signal having aperiodic content the type comprises automatically generating and inserting at least one synthetic segment into the buffered signal to increase the length of the content of the signal buffer.

5. The system of claim 4 wherein automatically generating the at least one synthetic segment comprises:

automatically computing the FFT of the at least part of the signal;

introducing a random rotation of the phase into the FFT coefficients; and

computing the inverse FFT for each segment, thereby creating the at least one synthetic segment.

6. The system of claim 4 wherein automatically generating the at least one synthetic segment comprises:

applying at least one LPC filter to the at least part of the signal to compute an LPC residual;

computing at least one FFT from the LPC residual;

introducing a random rotation of the phase into the coefficients of at least one of the computed FFTs;

computing inverse FFTs from the FFT coefficients to reconstruct the LPC residual; and

applying at least one inverse LPC filter to the LPC residual, thereby creating the at least one synthetic segment.

7. The system of claim 1 wherein the predetermined threshold for stretching and compressing at least part of the signal present in the signal buffer are optimized to compensate for clock drift between an encoder and a decoder.

8. A system for providing an adaptive playback of received frames of an audio signal transmitted across a packet-based network, comprising:

receiving and decoding data frames of an audio signal transmitted across a packet-based network;

storing the decoded data frames to a signal buffer;

analyzing the contents of the signal buffer to determine whether any data frames are missing due to corresponding data packets having not been received by a predetermined packet late loss time;

specifying a maximum delay period, extending past the predetermined packet late loss time, for receiving any missing data packets;

outputting one or more of the decoded frames present in the signal buffer when the analysis of the contents of the signal buffer indicates that the length of the signal in the signal buffer is between a predetermined minimum and a predetermined maximum buffer size;

following the predetermined packet late loss time, stretching and outputting one or more decoded frames preceding the missing data packets in the signal buffer, until any of receiving the missing data packets and exceeding the maximum delay period, when the analysis of the contents of the signal buffer indicates that the length of the decoded frames in the signal buffer is less than the predetermined minimum buffer size; and

compressing and outputting one or more decoded frames in the signal buffer when the analysis of the contents of the signal buffer indicates that the length of the decoded frames in the signal buffer is greater than the predetermined maximum buffer size.

9. The system of claim 8 wherein any frame output from the signal buffer is removed from the signal buffer as it is output.

10. The system of claim 8 further comprising packet loss concealment for signal packets declared to be late loss packets.

11. The system of claim 8 wherein stretching and outputting one or more decoded frames provides automatic jitter control as a function of buffer content.

12. The system of claim 11 wherein stretching one or more decoded frames further comprises automatically determining a content type of the stretched frames prior to stretching those frames.

13. The system of claim 12 wherein the content type includes any of voiced framed, unvoiced frames, and mixed frames.

14. The system of claim 13 wherein stretching any voiced frame comprises:

identifying at least one of the segment of the voiced frame as a template;

searching for a matching segment in adjacent frames whose cross correlation peak exceeds a predetermined threshold; and

aligning and merging the matching segments of the frame.

15. The system of claim 8 wherein stretching any unvoiced frame comprises automatically generating and inserting at least one synthetic segment into the current frame to increase a length of the current frame.

16. The system of claim 15 wherein automatically generating the at least one synthetic segment comprises:

automatically computing the FFT of the current frame;

introducing a random rotation of the phase into the FFT coefficients; and

computing the inverse FFT for each segment, thereby creating the at least one synthetic segment.

17. The system of claim 15 wherein automatically generating the at least one synthetic segment comprises:

applying at least one LPC filter to the current frame to compute an LPC residual;

computing at least one FFT from the LPC residual;

introducing a random rotation of the phase into the coefficients of at least one of the computed FFTs;

computing inverse FFTs from the FFT coefficients to reconstruct the LPC residual; and

applying at least one inverse LPC filter to the LPC residual, thereby creating the at least one synthetic segment.

18. The system of claim 8 wherein stretching any mixed frame comprises:

identifying at least one segment of the frame as a template;

searching for a matching segment whose cross correlation peak exceeds a predetermined threshold;

aligning and merging the matching segments of the frame to create an interim voiced segment;

automatically generating and inserting at least one synthetic segment into the current frame to create an interim unvoiced segment;

weighting each of the interim voiced segment and the interim unvoiced segment relative to a normalized cross correlation peak computed for the current segment; and

adding and windowing the interim voiced segment and the interim unvoiced segment to create a partially synthetic stretched segment.

19. The system of claim 8 wherein compressing any voiced frame comprises:

identifying at least one segment of the frame as a template;

searching for a matching segment whose cross correlation peak exceeds a predetermined threshold;

cutting out the signal between the template and the match; and

aligning and merging the matching segments of the frame.

20. The system of claim 8 wherein compressing any voiced frame comprises:

shifting a segment of the frame from a first position in the frame to a second position in the frame;

deleting the portion of the frame between the first position and the second position; and

adding the shifted segment of the frame to the signal representing the remainder of the frame by using a sine windowing function for blending the edges of the segment with the signal representing the remainder of the frame.

21. The system of claim 8 wherein both the predetermined minimum buffer size for stretching one or more decoded frames in the signal buffer and the predetermined maximum buffer size for compressing one or more decoded frames in the signal buffer are optimized to compensate for clock drift between an encoder and a decoder.