Use of frame caching to improve packet loss recovery

• US 9,866,871 B2
• Filed: 01/17/2014
• Issued: 01/09/2018
• Est. Priority Date: 12/30/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Lin, Chih-Lung, Xia, Minghui, Hsu, Pohsiang, Regunathan, Shankar, Holcomb, Thomas W.
• Primary Examiner(s): ADHAMI, MOHAMMAD SAJID
• Application Number:
  US14/158,720
• Publication Number:
  US 20140133583A1
• Time in Patent Office: 1,453 Days
• Field of Search: None
• CPC Class Codes: H04N 19/103 Selection of coding mode or of prediction mode
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/166 concerning the amount of transmission errors, e.g. bit error rate [BER]
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/188 the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/573 Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/89 involving methods or arrangements for detection of transmission errors at the decoder
• CPC Class Codes: H04N 19/91 Entropy coding, e.g. variable length coding [VLC] or arithmetic coding

Abstract

Various new and non-obvious apparatus and methods for using frame caching to improve packet loss recovery are disclosed. One of the disclosed embodiments is a method for using periodical and synchronized frame caching within an encoder and its corresponding decoder. When the decoder discovers packet loss, it informs the encoder which then generates a frame based on one of the shared frames stored at both the encoder and the decoder. When the decoder receives this generated frame it can decode it using its locally cached frame.

Claims

1. A method of transmitting a video signal over a network, the method comprising:

by a computing system that implements a video encoder having an encoder-side frame cache comprising a first portion and a second portion, the second portion being different than the first portion;

encoding at least a portion of the video signal into one or more frames, one or more of the frames including a frame header being marked to indicate that the marked frames are cached frames that can be used to decode another one of the frames and is to be cached in a decoder-side frame cache after being decoded by a video decoder;

storing the cached frames at the encoder in the first portion of the encoder-side frame cache, the first portion of the encoder-side frame cache corresponding to a first portion of the decoder-side frame cache at the video decoder, and wherein the first reference frame stored in the first portion of the encoder-side frame cache includes a frame header that indicates the first reference frame is to be cached in the first portion of a decoder-side frame cache after being decoded;

transmitting the frames;

receiving a loss detection signal sent in response to detecting loss of encoded video data for at least one frame transmitted by the video encoder; and

responsive to receiving the loss detection signal, encoding one or more new frames relative to the cached frames, the new frames being decodable by a video decoder using the encoded video data for the new frames and one or more decoder-side cached frames corresponding to the cached frames stored at the encoder, the new frames including different video data than the at least one frame for which loss of encoded video data is detected.

2. The method of claim 1, wherein the transmitting comprising sending the frames to a decoder over the network.

3. The method of claim 1, wherein at least one of the cached frames includes a portion of a frame but not an entire frame.

4. The method of claim 1, wherein the frame header indicates that a specific frame or portion of a specific frame is to be cached.

5. The method of claim 1, wherein at least one of the cached frames is a reference frame.

6. The method of claim 1, wherein at least one of the cached frames is not a reference frame.

7. The method of claim 1, wherein at least one of the cached frames is not displayed as part of the video signal.

8. The method of claim 1, further comprising storing the encoded new video frames in one or more computer-readable storage media.

9. A computer-readable storage device or memory storing computer-executable instructions for causing a computer system to perform a method of encoding video, the computer-executable instructions comprising:

instructions for encoding at least a portion of a video signal into one or more frames, one or more of the frames being cached frames, at least one of the cached frames including a header indicating that the at least one of the cached frames can be used to decode another one of the frames and is to be cached in a decoder-side frame cache after being decoded by a video decoder;

instructions for storing the cached frames at the encoder in an encoder-side frame cache, the encoder-side frame cache comprising a first portion and a second portion, the second portion being different than the first portion, the first portion of the encoder-side frame cache corresponding to a first portion of the decoder-side frame cache at the video decoder;

instructions for transmitting the frames;

instructions for receiving a loss detection signal sent in response to detecting loss of encoded video data for at least one frame transmitted by the video encoder; and

instructions for, responsive to receiving the loss detection signal, encoding one or more new frames relative to the cached frames, the new frames being decodable by a video decoder using the encoded video data for the new frames and one or more decoder-side cached frames corresponding to the cached frames stored at the first portion of the encoder, the new frames including different video data than the at least one frame for which loss of encoded video data is detected.

10. The computer-readable storage device or memory of claim 9, further comprising:

instructions for storing the cached frames in the encoder-sided frame cache periodically.

11. The computer-readable storage device or memory of claim 9, wherein at least one of the cached frames is a reference frame.

12. The computer-readable storage device or memory of claim 9, wherein:

a plurality of frames are stored in the encoder-side frame cache; and

the header contains sufficient information to determine which cached frame should be used.

13. The computer-readable storage device or memory of claim 9, wherein the cached frames are interframes.

14. The computer-readable storage device or memory of claim 9, wherein the cached frames include two cached frames indicated by a header in the encoded video signal and another cached frame that is the most recent interframe.

15. The computer-readable storage device or memory of claim 9, wherein at least one of the cached frames includes a portion of a frame but not an entire frame.

16. The computer-readable storage device or memory of claim 9, wherein the frame header indicates to a decoder that a specific frame or portion of a specific frame is to be cached.

17. The computer-readable storage device or memory of claim 16, wherein:

at least one other of the cached frames is cached because it is the most recent reference frame encoded.

18. A computer system that implements a video encoder, the computer system comprising a processor and memory, wherein the video encoder includes:

an encoder configured to receive a sequence of video frames and produce compressed video information as output;

a first portion of a encoder-side frame cache for storing a first reference frame, wherein the first portion of the encoder-side frame cache corresponds to a first portion of a decoder-side frame cache at a video decoder, and wherein the first reference frame stored in the first portion of the encoder-side frame cache includes a frame header that indicates the first reference frame is to be cached in the first portion of a decoder-side frame cache after being decoded;

a second portion of the encoder-side frame cache for storing a second reference frame, wherein the second portion of the encoder-side frame cache is different than the first portion of the encoder-side frame cache,

a receiver configured to receive a loss detection signal received from a decoder using a network feedback loop; and

wherein the encoder is configured to, responsive to receiving the loss detection signal at the receiver, encode a new frame using the first reference frame stored in the first portion of the encoder-side frame cache, the new frame including different data than the encoded data for which loss is detected.

19. The computer system of claim 18, wherein the second reference frame stored in the second portion of the encoder-side frame cache is updated to store a current frame of the plural frames after the decoding, for use in decoding a next frame of the plural frames.

20. The computer system of claim 18, wherein the video encoder includes:

a third portion of a encoder-side frame cache for storing a third interframe, wherein the third portion of the encoder-side frame cache corresponds to a third portion of a decoder-side frame cache at a video decoder, and wherein the third reference frame stored in the third portion of the encoder-side frame cache is the latest cached frame.