Use of frame caching to improve packet loss recovery

• US 8,634,413 B2
• Filed: 12/30/2004
• Issued: 01/21/2014
• Est. Priority Date: 12/30/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Lin, Chih-Lung, Xia, Minghui, Hsu, Pohsiang, Regunathan, Shankar, Holcomb, Thomas W.
• Primary Examiner(s): Pham, Chi
• Assistant Examiner(s): ADHAMI, MOHAMMAD SAJID
• Application Number:
  US11/027,294
• Publication Number:
  US 20060146830A1
• Time in Patent Office: 3,309 Days
• Field of Search: None
• US Class Current: 370/389
• CPC Class Codes: H04N 19/103 Selection of coding mode or of prediction mode
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/166 concerning the amount of transmission errors, e.g. bit error rate [BER]
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/188 the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/573 Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/89 involving methods or arrangements for detection of transmission errors at the decoder
• CPC Class Codes: H04N 19/91 Entropy coding, e.g. variable length coding [VLC] or arithmetic coding

Abstract

Various new and non-obvious apparatus and methods for using frame caching to improve packet loss recovery are disclosed. One of the disclosed embodiments is a method for using periodical and synchronized frame caching within an encoder and its corresponding decoder. When the decoder discovers packet loss, it informs the encoder which then generates a frame based on one of the shared frames stored at both the encoder and the decoder. When the decoder receives this generated frame it can decode it using its locally cached frame.

Claims

1. A method of decoding a signal transmitted over a network, comprising:

receiving, at a decoder, a compressed stream containing a sequence of encoded frames comprising key frames and predicted frames, the key frames having been intra-coded, the predicted frames having been encoded using motion-compensated predictive encoding with reference to respective one or more reference frames, at least some of the encoded frames having code elements designating said frames that have the code elements to be a cached frame corresponding to a frame cached at the encoder;

upon receiving any frame designated to be a cached frame, retaining temporarily said any designated frame in a frame cache, at the decoder;

with said any designated frame retained in the frame cache at the decoder, detecting packet loss affecting a subsequent one of the encoded frames sent to the decoder;

with the decoder, sending a packet loss detection signal to the encoder indicating that packet loss has been detected;

in response to the packet loss detection, the decoder receiving, from the encoder, a generated frame having been encoded by the encoder using motion-compensated inter-picture compression encoding with reference to said frame cached at the encoder as a substitute frame for the frame affected by the packet loss, the generated frame being marked to indicate that the generated frame can be decoded from said any designated frame retained in the frame cache at the decoder;

with the decoder, decoding the generated frame with reference to said any designated frame retained in the frame cache at the decoder which corresponds to the frame cached at the encoder; and

with the decoder, producing a reconstructed frame sequence with an uncompressed frame based at least in part on said decoding of the generated frame.

2. The method of claim 1, wherein the method further comprises periodically receiving and retaining frames designated to be a cached frame in the frame cache at the decoder.

3. The method of claim 2, wherein said periodically receiving occurs at a set interval.

4. The method of claim 1, wherein the packet loss has a time associated with it and wherein a timestamp representing the packet loss time is sent to the encoder.

5. The method of claim 1, wherein the encoder receives the packet loss detection signal, and upon receipt of the packet loss detection signal, encodes the generated frame using motion-compensated inter-picture compression encoding with reference to the frame cached at the encoder.

6. The method of claim 5, wherein the substitute frame further comprises a frame header and wherein the substitute frame is distinctively named in the frame header.

7. The method of claim 1, wherein the signal is an image signal, a video signal, an audio signal, or a speech signal.

12. The method of claim 1, wherein the packet loss detection signal is sent periodically by the decoder until the arrival of a new key frame at the decoder.

13. The method of claim 1, further comprising the decoder decoding any frames designated as generated frames and received from the encoder following said frame having packet loss using the frame retained in the frame cache at the decoder until receipt of a next frame designated to be a cached frame.

14. The method of claim 1, wherein the frame cache holds multiple frames.

17. The method of claim 1, wherein if the generated frame is not received by the decoder, then the decoder continues sending packet loss reports until arrival of a new key frame.

18. The method of claim 1, wherein the decoding the generated frame includes:

arithmetic decoding encoded transform coefficients and encoded side information;

inverse quantizing the transform coefficients to produce dequantized transform coefficients;

inverse frequency transforming the dequantized transform coefficients to produce residual values;

performing motion compensation to produce motion-compensated prediction values using motion information and the frame retained in the frame cache at the decoder;

combining the motion-compensated prediction values with the residual values; and

performing deblock filtering.

19. The method of claim 1, wherein the frame cache at the decoder is updated according to information in frame headers that indicates which of the frames is to be stored in the frame cache at the decoder, and wherein the frame cache at the decoder is also updated to store a current frame of the frames for use as a reference frame in decoding a next frame of the frames.

20. The method of claim 19, wherein information that indicates the current frame is to be stored in the frame cache for use as a reference frame is signaled in a frame header for the current frame as part of frame type.

21. The method of claim 1, wherein the compressed stream is received over a real-time transport protocol channel, and wherein the packet loss detection signal is transmitted over a real-time control protocol channel.

8. A computer-readable storage device or memory storing computer-executable instructions for causing a computer system to perform a method comprising:

receiving, at a decoder, a compressed stream containing a sequence of encoded frames comprising key frames and predicted frames, the key frames having been intra-coded, the predicted frames having been encoded using motion-compensated predictive encoding with reference to respective one or more reference frames, at least some of the encoded frames having code elements designating said frames that have the code elements to be a cached frame corresponding to a frame cached at the encoder;

upon receiving any frame designated to be a cached frame, retaining temporarily said any designated frame in a frame cache, at the decoder;

with said any designated frame retained in the frame cache at the decoder, detecting packet loss affecting a subsequent one of the encoded frames sent to the decoder;

with the decoder, sending a packet loss detection signal to the encoder indicating that packet loss has been detected;

in response to the packet loss detection, the decoder receiving, from the encoder, a generated frame having been encoded by the encoder using motion-compensated inter-picture compression encoding with reference to said frame cached at the encoder as a substitute frame for the frame affected by the packet loss, the generated frame being marked to indicate that the generated frame can be decoded from said any designated frame retained in the frame cache at the decoder;

with the decoder, decoding the generated frame with reference to said any designated frame retained in the frame cache at the decoder which corresponds to the frame cached at the encoder; and

with the decoder, producing a reconstructed frame sequence with an uncompressed frame based at least in part on said decoding of the generated frame.

22. The computer-readable storage device or memory of claim 8, wherein the method further comprises periodically receiving and retaining frames designated to be a cached frame in the frame cache at the decoder.

23. The computer-readable storage device or memory of claim 8, wherein the packet loss has a time associated with it and wherein a timestamp representing the packet loss time is sent to the encoder.

24. The computer-readable storage device or memory of claim 8, wherein the encoder receives the packet loss detection signal, and upon receipt of the packet loss detection signal, encodes the generated frame using motion-compensated inter-picture compression encoding with reference to the frame cached at the encoder.

25. The computer-readable storage device or memory of claim 8, wherein the packet loss detection signal is sent periodically by the decoder until the arrival of a new key frame at the decoder.

26. The computer-readable storage device or memory of claim 8, wherein the method further comprises the decoder decoding any frames designated as generated frames and received from the encoder following said frame having packet loss using the frame retained in the frame cache at the decoder until receipt of a next frame designated to be a cached frame.

27. The computer-readable storage device or memory of claim 8, wherein the frame cache holds multiple frames.

28. The computer-readable storage device or memory of claim 8, wherein the method further comprises:

if the generated frame is not received by the decoder, then the decoder continues sending packet loss reports until arrival of a new key frame.

29. The computer-readable storage device or memory of claim 8, wherein the decoding the generated frame includes:

arithmetic decoding encoded transform coefficients and encoded side information;

inverse quantizing the transform coefficients to produce dequantized transform coefficients;

inverse frequency transforming the dequantized transform coefficients to produce residual values;

performing motion compensation to produce motion-compensated prediction values using motion information and the frame retained in the frame cache at the decoder;

combining the motion-compensated prediction values with the residual values; and

performing deblock filtering.

30. The computer-readable storage device or memory of claim 8, wherein the frame cache at the decoder is updated according to information in frame headers that indicates which of the frames is to be stored in the frame cache at the decoder, and wherein the frame cache at the decoder is also updated to store a current frame of the frames for use as a reference frame in decoding a next frame of the frames.

9. A system of streaming video within a lossy network comprising:

a decoder operable to receive data packets for a compressed stream containing a sequence of encoded frames comprising key frames and predicted frames from an encoder, the key frames having been intra-coded, the predicted frames having been encoded using motion-compensated predictive encoding with reference to respective one or more reference frames, at least some of the encoded frames having code elements designating said frames that have the code elements to be a cached frame corresponding to a frame cached at the encoder;

a frame cache at the decoder;

the decoder operable upon receiving any frame designated to be a cached frame from the encoder to retain said any designated frame in the frame cache;

the decoder operable to decode the key frames received from the encoder;

the decoder operable to decode the motion-compensated predicted frames received from the encoder with reference to their respective reference frames;

the decoder being operable to detect packet loss that affects a subsequent one of the encoded frames; and

upon such detection, operable to signal said packet loss to the encoder and receive back from the encoder in response a generated frame having been encoded by the encoder using motion-compensated inter-picture compression encoding with respect to the frame cached at the encoder as replacement for the frame affected by the packet loss, the generated frame being marked to indicate that the generated frame can be decoded from said any designated frame retained in the frame cache at the decoder; and

upon receipt of said generated frame from the encoder, the decoder operable to decode the generated frame with reference to the frame retained in the frame cache at the decoder, and to reconstruct a sequence of frames decoded from the compressed stream having an uncompressed frame based at least in part on said decoding of the generated frame.

10. The system of claim 9, further comprising a synchronizer which synchronizes the frame cached at the encoder with the frame retained in the frame cache in the decoder.

15. The system of claim 9, wherein the decoder is operable to receive multiple generated frames from the encoder in response to detecting packet loss in frames, and decode said generated frames with reference to the frame retained in the frame cache in the decoder cache until another frame designated to be a cached frame is received.

16. The system of claim 9, wherein the decoder is operable to transmit location of the packet loss to the encoder using a timestamp of earliest detected packet loss.

11. A video processor, comprising:

a decoder with a frame cache, operable to retain a copy of a frame designated for caching in the frame cache and transmitted to the decoder from an encoder;

the decoder operable to receive data packets for a compressed stream containing a sequence of encoded frames comprising key frames and predicted frames from an encoder, the key frames having been intra-coded, the predicted frames having been encoded using motion-compensated predictive encoding with reference to respective one or more reference frames, at least some of the encoded frames having code elements designating said frames that have the code elements as corresponding to a frame cached at the encoder for caching in the frame cache;

the decoder operable to decode the key frames, and to decode the predicted frames with reference to their respective one or more reference frames;

the decoder operable to retain temporarily said any designated frame in a frame cache;

the decoder operable to detect packet loss that affects a subsequent frame during transmission from the encoder;

the decoder operable to transmit location of the packet loss to the encoder;

the decoder operable to receive from the encoder in response to the transmitted location of the packet loss a generated frame designated with a header code as having been encoded by the encoder using motion-compensated inter-picture compression encoding with reference to the frame cached at the encoder as a replacement for the frame affected by the packet loss;

the decoder operable to decode the generated frame designated with the header code as a replacement frame with reference to the copy of the frame retained in the frame cache of the decoder; and

the decoder operable to reconstruct a sequence of frames decoded from the compressed stream.

31. The video processor of claim 11, further comprising a synchronizer which synchronizes the frame cached at the encoder with the frame retained in the frame cache in the decoder.

32. The video processor of claim 11, wherein the decoder is operable to receive multiple generated frames from the encoder in response to detecting packet loss in frames, and decode said generated frames with reference to the frame retained in the frame cache in the decoder cache until another frame designated to be a cached frame is received.

33. The video processor of claim 11, wherein the decoder is operable to transmit location of the packet loss to the encoder using a timestamp of earliest detected packet loss.