Use of frame caching to improve packet loss recovery

• US 9,313,501 B2
• Filed: 09/13/2012
• Issued: 04/12/2016
• Est. Priority Date: 12/30/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Lin, Chih-Lung, Xia, Minghui, Hsu, Pohsiang, Regunathan, Shankar, Holcomb, Thomas W.
• Primary Examiner(s): Adhami, Mohammad
• Application Number:
  US13/614,453
• Publication Number:
  US 20130010861A1
• Time in Patent Office: 1,307 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: H04N 19/103 Selection of coding mode or of prediction mode
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/166 concerning the amount of transmission errors, e.g. bit error rate [BER]
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/188 the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/573 Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/89 involving methods or arrangements for detection of transmission errors at the decoder
• CPC Class Codes: H04N 19/91 Entropy coding, e.g. variable length coding [VLC] or arithmetic coding

Abstract

Various new and non-obvious apparatus and methods for using frame caching to improve packet loss recovery are disclosed. One of the disclosed embodiments is a method for using periodical and synchronized frame caching within an encoder and its corresponding decoder. When the decoder discovers packet loss, it informs the encoder which then generates a frame based on one of the shared frames stored at both the encoder and the decoder. When the decoder receives this generated frame it can decode it using its locally cached frame.

Claims

1. In a computing system that implements a video decoder, a method of decoding plural frames of a video sequence, the method comprising:

decoding a first frame of the plural frames using encoded data for the first frame, wherein the encoded data for the first frame includes a frame header for the first frame;

storing, for use as a first reference frame, the first frame in a first portion of a decoder-side frame cache that corresponds to a first portion of an encoder-side frame cache at a video encoder, wherein the frame header for the first frame includes information indicating that the first frame is to be stored in the first portion of the decoder-side frame cache;

decoding a second frame of the plural frames using encoded data for the second frame, the second frame following the first frame in the video sequence;

storing, for use as a second reference frame, the second frame in a second portion of the decoder-side frame cache different than the first portion of the decoder-side frame cache;

detecting loss of encoded data for one of the plural frames;

in response to the detecting loss of encoded data, transmitting a loss detection signal in order to cause the video encoder to generate a new frame relative to the first reference frame stored in the first portion of the encoder-side frame cache at the video encoder, the new frame including different data than the one of the plural frames for which loss of encoded data is detected; and

decoding the new frame using encoded data for the new frame and using the first reference frame stored in the first portion of the decoder-side frame cache, the encoded data for the new frame having been generated by the video encoder, in response to the loss detection signal, during encoding of the new frame relative to the first reference frame stored in the first portion of the encoder-side frame cache at the video encoder.

2. The method of claim 1 wherein the loss of encoded data is for a third frame of the plural frames, the third frame having been encoded relative to the second reference frame stored in a second portion of the encoder-side frame cache at the video encoder.

3. The method of claim 1 wherein the decoding the new frame includes:

arithmetic decoding encoded transform coefficients and encoded side information in the encoded data;

inverse quantizing the transform coefficients to produce dequantized transform coefficients;

inverse frequency transforming the dequantized transform coefficients to produce residual values;

performing motion compensation to produce motion-compensated prediction values using motion information and the first reference frame in the first portion of the decoder-side frame cache;

combining the motion-compensated prediction values with the residual values; and

performing deblock filtering.

4. The method of claim 1 wherein the first reference frame in the first portion of the decoder-side frame cache is updated according to information in frame headers that indicates which of the plural frames are to be stored in the first portion of the decoder-side frame cache, and wherein the second reference frame in the second portion of the decoder-side frame cache is updated to store a current frame of the plural frames for use in decoding a next frame of the plural frames.

5. The method of claim 1 wherein the encoded data is received over a real-time transport protocol channel, and wherein the loss detection signal is transmitted over a real-time control protocol channel.

6. The method of claim 1 wherein the detecting the loss of encoded data for one of the plural frames comprises discovering a gap in sequence numbers for plural packets due to packet loss or corruption of the encoded data.

7. The method of claim 1 further comprising repeating the transmitting of a loss detection signal until receipt of the encoded data for the new frame.

8. The method of claim 1 wherein the loss detection signal includes a timestamp and sequence number for a packet subject to the loss of encoded data.

9. The method of claim 1 wherein the loss detection signal includes a list of packets subject to the loss of encoded data.

10. The method of claim 1 wherein the loss detection signal identifies one or more slices subject to the loss of encoded data.

11. The method of claim 1 wherein the loss detection signal includes information that is usable by the video encoder to determine which stored reference frame to use among multiple reference frames stored at the video encoder, and wherein the encoded data for the new frame includes a frame header for the new frame, the frame header for the new frame including information that indicates to use the first reference frame in the decoding of the new frame.

12. In a computing system that implements a video encoder, a method of encoding plural frames of a video sequence, the method comprising:

encoding a first frame of the plural frames to produce encoded data for the first frame, wherein the encoded data for the first frame includes a frame header for the first frame;

storing, for use as a first reference frame, the first frame in a first portion of an encoder-side frame cache that corresponds to a first portion of a decoder-side frame cache at a video decoder, wherein the frame header for the first frame includes information indicating that the first frame is to be stored in the first portion of the decoder-side frame cache;

encoding a second frame of the plural frames to produce encoded data for the second frame, the second frame following the first frame in the video sequence;

storing, for use as a second reference frame, the second frame in a second portion of the encoder-side frame cache different than the first portion of the encoder-side frame cache;

receiving a loss detection signal that indicates loss of encoded data for one of the plural frames; and

in response to the loss detection signal, encoding a new frame relative to the first reference frame stored in the first portion of the encoder-side frame cache to produce encoded data for the new frame, the new frame including different data than the one of the plural frames for which the loss detection signal indicates loss of encoded data, the new frame being decodable by the video decoder using the encoded data for the new frame and the first reference frame stored in the first portion of the decoder-side frame cache.

13. The method of claim 12 wherein the loss of encoded data is for a third frame of the plural frames, the third frame having been encoded relative to the second reference frame stored in the second portion of the encoder-side frame cache.

14. The method of claim 12 wherein the encoding the new frame includes:

performing motion compensation to produce motion-compensated prediction values using motion information and the first reference frame in the first portion of the encoder-side frame cache;

determining residual values using the motion-compensated prediction values;

frequency transforming the residual values to produce transform coefficients;

quantizing the transform coefficients to produce quantized transform coefficients; and

arithmetic coding the quantized transform coefficients and side information.

15. The method of claim 12 wherein the first reference frame in the first portion of the encoder-side frame cache is updated according to information in frame headers that indicates which of the plural frames are to be stored in the first portion of the decoder-side frame cache, and wherein the second reference frame in the second portion of the encoder-side frame cache is updated to store a current frame of the plural frames for use in encoding a next frame of the plural frames.

16. The method of claim 12 wherein the encoded data is transmitted over a real-time transport protocol channel, and wherein the loss detection signal is received over a real-time control protocol channel.

17. The method of claim 12 further comprising setting an interval for updating the first reference frame stored in the first portion of the encoder-side frame cache that corresponds to the first portion of the decoder-side frame cache at the video decoder.

18. The method of claim 17 wherein the interval is a regular, periodic interval.

19. The method of claim 17 wherein the interval depends at least in part on one or more of network speed and loss history.

20. The method of claim 12 wherein the first frame is a predicted frame encoded relative to an earlier reference frame for a key frame, the second frame is a predicted frame encoded relative to the first reference frame, and the new frame is a predicted frame encoded relative to the first reference frame.

21. The method of claim 12 wherein the encoded data for the second frame includes a frame header for the second frame, the frame header for the second frame including information that indicates the second frame is to be stored in the second portion of the decoder-side frame cache, and wherein the information that indicates the second frame is to be stored in the second portion of the decoder-side frame cache is signaled in the frame header for the second frame as part of frame type.

22. One or more computer-readable storage devices or memory having stored thereon computer-executable instructions for causing a computer system programmed thereby to perform operations for decoding plural frames of a video sequence in a video decoder, the operations comprising:

decoding a first frame of the plural frames using encoded data for the first frame, wherein the encoded data for the first frame includes a frame header for the first frame;

storing, for use as a first reference frame, the first frame in a first portion of a decoder-side frame cache that corresponds to a first portion of an encoder-side frame cache at a video encoder, wherein the frame header for the first frame includes information indicating that the first frame is to be stored in the first portion of the decoder-side frame cache;

decoding a second frame of the plural frames using encoded data for the second frame, the second frame following the first frame in the video sequence;

storing, for use as a second reference frame, the second frame in a second portion of the decoder-side frame cache different than the first portion of the decoder-side frame cache;

detecting loss of encoded data for one of the plural frames;

in response to the detecting loss of encoded data, transmitting a loss detection signal in order to cause the video encoder to generate a new frame relative to the first reference frame stored in the first portion of the encoder-side frame cache at the video encoder; and

decoding the new frame using encoded data for the new frame and using the first reference frame stored in the first portion of the decoder-side frame cache, the encoded data for the new frame including different data than the one of the plural frames for which loss is detected, the encoded data for the new frame having been generated by the video encoder, in response to the loss detection signal, during encoding of the new frame relative to the first reference frame stored in the first portion of the encoder-side frame cache at the video encoder.

29. The computer-readable storage devices or memory of claim 22, wherein the loss of encoded data is for a third frame of the plural frames, the third frame having been encoded relative to the second reference frame stored in a second portion of the encoder-side frame cache at the video encoder.

30. The computer-readable storage devices or memory of claim 22, wherein the first reference frame in the first portion of the decoder-side frame cache is updated according to information in frame headers that indicates which of the plural frames are to be stored in the first portion of the decoder-side frame cache, and wherein the second reference frame in the second portion of the decoder-side frame cache is updated to store a current frame of the plural frames for use in decoding a next frame of the plural frames.

31. The computer-readable storage devices or memory of claim 22, wherein the encoded data is received over a real-time transport protocol channel, and wherein the loss detection signal is transmitted over a real-time control protocol channel.

40. The computer-readable storage devices or memory of claim 22 wherein the loss detection signal includes information that is usable by the video encoder to determine which stored reference frame to use among multiple reference frames stored at the video encoder, and wherein the encoded data for the new frame includes a frame header for the new frame, the frame header for the new frame including information that indicates to use the first reference frame in the decoding of the new frame.

23. A computer system that implements a video decoder, the computer system comprising a processor and memory, wherein the video decoder includes:

a jitter buffer for storing encoded data for plural frames of a video sequence;

a module configured for decoding the plural frames;

a decoder-side frame cache, the decoder-side frame cache comprising;

a first portion of the decoder-side frame cache for storing a first reference frame, wherein the first portion of the decoder-side frame cache corresponds to a first portion of an encoder-side frame cache at a video encoder, and wherein the first reference frame stored in the first portion of the decoder-side frame cache is updated according to information in frame headers for the plural frames that indicates which of the plural frames are to be cached in the first portion of the decoder-side frame cache after the decoding, and

a second portion of the decoder-side frame cache for storing a second reference frame, wherein the second portion of the decoder-side frame cache is different than the first portion of the decoder-side frame cache, and wherein the second reference frame stored in the second portion of the decoder-side frame cache is updated to store a current frame of the plural frames after the decoding, for use in decoding a next frame of the plural frames;

a module configured for detecting loss in the encoded data that affects decoding of a given frame of the plural frames, the given frame having been encoded relative to the second reference frame; and

a module configured for transmitting a loss detection signal;

the video decoder being configured to decode a new frame using the first reference frame in the first portion of the decoder-side frame cache, the new frame including different data than the encoded data for which loss is detected, the new frame having been encoded by the video encoder, in response to the loss detection signal, relative to the first reference frame stored in the first portion of the encoder-side frame cache at the video encoder.

24. The computer system of claim 23 wherein the video decoder is implemented at least in part with a field-programmable gate array, application-specific integrated circuit, or software that is executable on the processor.

25. The computer system of claim 23 wherein the module configured for detecting loss is adapted to discover gaps in sequence numbers for packets due to packet loss or corruption of the encoded data.

26. The computer system of claim 23 wherein the loss detection signal includes information that is usable by the video encoder to determine which reference frame to use among multiple reference frames cached at the video encoder.

41. The computer system of claim 23 wherein the loss detection signal module is further configured to repeat the transmitting of a loss detection signal until receipt of the encoded data for the new frame.

42. The computer system of claim 23 wherein the loss detection signal includes a timestamp and sequence number for a packet subject to the loss of encoded data.

43. The computer system of claim 23 wherein the loss detection signal includes a list of packets subject to the loss of encoded data.

44. The computer system of claim 23 wherein the loss detection signal identifies one or more slices subject to the loss of encoded data.

27. A computer system that implements a video encoder, the computer system comprising a processor and memory, wherein the video encoder includes:

a module configured for encoding plural frames of a video sequence;

an encoder-side frame cache, the encoder-side frame cache comprising;

a first portion of the encoder-side frame cache for storing a first reference frame, wherein the first portion of the encoder-side frame cache corresponds to a first portion of a decoder-side frame cache at a video decoder, and wherein the first reference frame stored in the first portion of the encoder-side frame cache is updated according to information in frame headers for the plural frames that indicates which of the plural frames are to be cached in the first portion of the decoder-side frame cache after decoding, and

a second portion of the encoder-side frame cache for storing a second reference frame, wherein the second portion of the encoder-side frame cache is different than the first portion of the encoder-side frame cache, and wherein the second reference frame stored in the second portion of the encoder-side frame cache is updated to store a current frame of the plural frames after encoding, for use in encoding a next frame of the plural frames; and

a module configured for receiving a loss detection signal that indicates loss of encoded data that affects decoding of a given frame of the plural frames, the given frame having been encoded relative to the second reference frame;

the video encoder being configured to encode, in response to the loss detection signal, a new frame relative to the first reference frame stored in the first portion of the encoder-side frame cache, the new frame including different data than the encoded data for which loss is detected, the new frame being decodable by the video decoder using encoded data for the new frame and the first reference frame stored in the first portion of the decoder-side frame cache.

28. The computer system of claim 27 wherein the encoded data for the new frame includes a frame header for the new frame, the frame header for the new frame including information that indicates which of multiple reference frames to use in the decoding of the new frame.

45. The computer system of claim 27 wherein the loss of encoded data is for a third frame of the plural frames, the third frame having been encoded relative to the second reference frame stored in the second portion of the encoder-side frame cache.

46. The computer system of claim 27 wherein the video encoder is further configured to encode the new frame by:

performing motion compensation to produce motion-compensated prediction values using motion information and the first reference frame in the first portion of the encoder-side frame cache;

determining residual values using the motion-compensated prediction values;

frequency transforming the residual values to produce transform coefficients;

quantizing the transform coefficients to produce quantized transform coefficients; and

arithmetic coding the quantized transform coefficients and side information.

47. The computer system of claim 27 wherein the first reference frame in the first portion of the encoder-side frame cache is updated according to information in frame headers that indicates which of the plural frames are to be stored in the first portion of the decoder-side frame cache, and wherein the second reference frame in the second portion of the encoder-side frame cache is updated to store a current frame of the plural frames for use in encoding a next frame of the plural frames.

48. The computer system of claim 27 wherein the encoded data is transmitted over a real-time transport protocol channel, and wherein the loss detection signal is received over a real-time control protocol channel.

49. The computer system of claim 27 wherein the video encoder is further configured to set an interval for updating the first reference frame stored in the first portion of the encoder-side frame cache that corresponds to the first portion of the decoder-side frame cache at the video decoder.

50. The computer system of claim 49 wherein the interval is a regular, periodic interval.

51. The computer system of claim 49 wherein the interval depends at least in part on one or more of network speed and loss history.

52. The computer system of claim 27 wherein the first frame is a predicted frame encoded relative to an earlier reference frame for a key frame, the second frame is a predicted frame encoded relative to the first reference frame, and the new frame is a predicted frame encoded relative to the first reference frame.

53. The computer system of claim 27 wherein the encoded data for the second frame includes a frame header for the second frame, the frame header for the second frame including information that indicates the second frame is to be stored in the second portion of the decoder-side frame cache, and wherein the information that indicates the second frame is to be stored in the second portion of the decoder-side frame cache is signaled in the frame header for the second frame as part of frame type.

32. One or more computer-readable storage devices or memory having stored thereon computer-executable instructions for causing a computer system programmed thereby to perform operations for encoding plural frames of a video sequence, the operations comprising:

encoding a first frame of the plural frames to produce encoded data for the first frame, wherein the encoded data for the first frame includes a frame header for the first frame;

storing, for use as a first reference frame, the first frame in a first portion of an encoder-side frame cache that corresponds to a first portion of a decoder-side frame cache at a video decoder, wherein the frame header for the first frame includes information indicating that the first frame is to be stored in the first portion of the decoder-side frame cache;

encoding a second frame of the plural frames to produce encoded data for the second frame, the second frame following the first frame in the video sequence;

storing, for use as a second reference frame, the second frame in a second portion of the encoder-side frame cache different than the first portion of the encoder-side frame cache;

receiving a loss detection signal that indicates loss of encoded data for one of the plural frames; and

in response to the loss detection signal, encoding a new frame relative to the first reference frame stored in the first portion of the encoder-side frame cache to produce encoded data for the new frame, the new frame including different data than the one of the plural frames for which loss is detected, the new frame being decodable by the video decoder using the encoded data for the new frame and the first reference frame stored in the first portion of the decoder-side frame cache.

33. The computer-readable storage devices or memory of claim 32, wherein the loss of encoded data is for a third frame of the plural frames, the third frame having been encoded relative to the second reference frame stored in the second portion of the encoder-side frame cache.

34. The computer-readable storage devices or memory of claim 32 wherein the first reference frame in the first portion of the encoder-side frame cache is updated according to information in frame headers that indicates which of the plural frames are to be stored in the first portion of the decoder-side frame cache, and wherein the second reference frame in the second portion of the encoder-side frame cache is updated to store a current frame of the plural frames for use in encoding a next frame of the plural frames.

35. The computer-readable storage devices or memory of claim 32 wherein the encoded data is transmitted over a real-time transport protocol channel, and wherein the loss detection signal is received over a real-time control protocol channel.

36. The computer-readable storage devices or memory of claim 32 wherein the method further comprises setting an interval for updating the first reference frame stored in the first portion of the encoder-side frame cache that corresponds to the first portion of the decoder-side frame cache at the video decoder.

37. The computer-readable storage devices or memory of claim 36 wherein the interval is a regular, periodic interval.

38. The computer-readable storage devices or memory of claim 36 wherein the interval depends at least in part on one or more of network speed and loss history.

39. The computer-readable storage devices or memory of claim 32 wherein the encoded data for the second frame includes a frame header for the second frame, the frame header for the second frame including information that indicates the second frame is to be stored in the second portion of the decoder-side frame cache, and wherein the information that indicates the second frame is to be stored in the second portion of the decoder-side frame cache is signaled in the frame header for the second frame as part of frame type.

54. The computer-readable storage devices or memory of claim 32 wherein the first frame is a predicted frame encoded relative to an earlier reference frame for a key frame, the second frame is a predicted frame encoded relative to the first reference frame, and the new frame is a predicted frame encoded relative to the first reference frame.