LONG TERM REFERENCE FRAME MANAGEMENT WITH ERROR FEEDBACK FOR COMPRESSED VIDEO COMMUNICATION

• US 20080247463A1
• Filed: 04/18/2007
• Published: 10/09/2008
• Est. Priority Date: 04/09/2007
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Friel, Joseph T., Mauchly, J. William, Buttimer, Maurice J.
• Granted Patent: US 8,494,049 B2
• Time in Patent Office: Days
• US Class Current: 375/240.12
• CPC Class Codes: H04N 19/105 Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/58 Motion compensation with long-term prediction, i.e. the reference frame for a current frame not being the temporally closest one H04N19/23 takes precedence
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/895 in combination with error concealment

Abstract

An apparatus, software encoded in tangible media, and a method at an encoder. The method includes sending compressed video data including a reference frame message to create a long term reference frame to a plurality of decoders at one or more destination points, receiving feedback from the decoders indicative of whether or not the decoders successfully received the reference frame message, and in the case that the received feedback is such that at least one of the decoders did not successfully receive the reference frame message or does not have the indicated recent frame, repeating sending a reference frame message to create the long term reference frame. Using the method can replaces I-frame error recovery with long term reference frames, even in the case where the reference frame management messages are lost to at least one decoder.

Claims

1. A method comprising:

an encoder at a source sending compressed video data including a reference frame message to create a long term reference frame from a recent frame indicated in the reference frame message, the sending being to a plurality of decoders at one or more destination points; and

receiving feedback from the plurality of decoders indicative of whether or not the decoders successfully received the reference frame message and have access to the indicated recent frame,

in the case that the received feedback is such that at least one of the decoders did not successfully receive the reference frame message or does not have the indicated recent frame, repeating sending a reference frame message to create the long term reference frame from an indicated recent frame to the plurality of decoders, and receiving feedback until the encoder has received feedback from each of the plurality of decoders indicating successful receipt of the reference frame message and indicating having access to the recent frame indicated in the reference frame message,

such that once the encoder has an indication of successful receipt and access at all decoders, the encoder, upon receiving feedback of an error from at least one of the decoders, is operable to use the long term reference frame as a reference frame for temporal predictive encoding of video, thereby enabling the encoder to avoid using an independently coded frame upon receiving feedback of an error.

2. A method as recited in claim 1, further comprising,

the encoder from time to time sending an additional reference frame message to create a different long term reference frame from a different recent frame indicated in the reference frame message, the sending being to the plurality of decoders at the one or more destination points, such the long term reference frame is updated from time to time with an updated frame, any updated frame not becoming valid for use as a reference frame until the encoder has received feedback from all decoders that each has successfully received the reference frame message and has the updated frame.

3. A method as recited in claim 2, wherein the sending from time to time is periodic.

4. A method as recited in claim 2, wherein the reference frame store at each decoder is configured to store more than one frame marked for use as a long term reference frame.

5. A method as recited in claim 4, further comprising designating at the encoder one of the valid long term reference frames to be a preferred long term reference frame to use as a reference frame for predictive coding when the encoder ascertains that there has been an error in receiving in at least one of the decoders.

6. A method as recited in claim 4, further comprising:

designating at the encoder one of the valid long term reference frames to be a preferred long term reference frame to use as a reference frame for predictive coding when the encoder ascertains that there has been an error in receiving in at least one of the decoders;

from time to time changing which valid long term reference frame is the preferred long term reference frame.

7. A method as recited in claim 2, wherein the sending of the reference frame message is controlled by a state machine.

8. A method as recited in claim 2, wherein commands are included in the reference frame message are such that the encoder can send the same reference frame message to all decoders independent of whether or not any destination decoder has previously sent an indication of an error, and independent of the state of the reference frame store in the destination decoders, including the case of there being disparity in the contents of the reference frames stores of at least two decoders, and there being no disparity.

9. A method as recited in claim 8, wherein commands are included in the reference frame message are such that if at least part of a frame sent by the encoder for use as the long term reference frame is not correctly received by any decoder, and other decoders correctly receive the frame sent by the encoder for use as the long term reference frame, the next time a reference frame message is sent and received correctly by all decoders, all decoders will have the same reference frame store state.

10. A method as recited in claim 8, wherein the reference frame message includes a command to unmark one frame previously marked in a destination decoder'"'"'s reference frame store for use as a short term reference frame and a command to unmark one frame previously marked in a destination decoder'"'"'s reference frame store for use as a long term reference frame, wherein at any particular destination decoder a command to unmark is ignored in the case the frame commanded to be unmarked is not present in the particular decoder'"'"'s reference frame store.

11. A method as recited in claim 1, wherein the indicated frame in the reference frame message is only valid for use as a long term reference frame as a result of the encoder'"'"'s receiving feedback from all decoders that each has successfully received the reference frame message and the indicated frame, and wherein each of the plurality of decoders includes a reference frame store operable to store a set of reference frames including one or more frames marked for use as possible long term reference frames, such that a valid long term reference frame is one that exists and is known to the encoder to exist in the reference frame store of each decoder and marked for use as a long term reference frame.

12. A method as recited in claim 1, wherein the plurality of decoders include more than one decoder of a decoding system at a single destination, the more than one decoder of the decoding system operable in parallel to each decode a different part of a frame.

13. A method as recited in claim 2, wherein the plurality of decoders include decoders at different destinations.

14. An apparatus comprising:

a network interface operable to be coupled to a network;

an encoder coupled to the network interface and operable to generate compressed video data and to embed therein a reference frame message to command any receiving decoder to create a long term reference frame from a recent frame indicated in the reference frame message,

a controller coupled to the network interface and the encoder and operable to cause the encoder to send the compressed video data with the reference frame message embedded therein, the sending being via the network to a plurality of decoders at one or more destination points, the controller further operable to accept feedback received from the plurality of decoders indicative of whether or not the decoders successfully received the reference frame message and have access to the indicated recent frame,

the controller further operable in the case that the received feedback is such that at least one of the decoders did not successfully receive the reference frame message or does not have the indicated recent frame, to cause the encoder to repeating sending a reference frame message to create the long term reference frame from an indicated recent frame to the plurality of decoders, the controller further operable to receiving feedback until feedback is received from each of the plurality of decoders indicating successful receipt of the reference frame message and indicating having access to the recent frame indicated in the reference frame message,

such that once the controller has an indication of successful receipt and access at all decoders, the controller can, upon indication that feedback of an error was received from at least one of the decoders, the controller is operable to cause the encoder use the long term reference frame as a reference frame for temporal predictive encoding of video, thereby enabling the encoder to avoid using an independently coded frame as a result of the apparatus receiving feedback of an error.

15. An apparatus as recited in claim 14, wherein the controller is operable to cause the encoder from time to time to sending an additional reference frame message to create a different long term reference frame from a different recent frame indicated in the reference frame message, the sending being to the plurality of decoders at the one or more destination points, such the long term reference frame is updated from time to time with an updated frame, any updated frame not becoming valid for use as a reference frame until the encoder has received feedback from all decoders that each has successfully received the reference frame message and has the updated frame.

16. An apparatus as recited in claim 15, wherein the reference frame store at each decoder is configured to store more than one frame marked for use as a long term reference frame.

17. An apparatus as recited in claim 16, wherein the controller further is operable to designate at the encoder one of the valid long term reference frames to be a preferred long term reference frame to use as a reference frame for predictive coding when the controller ascertains that there has been an error in receiving in at least one of the decoders.

18. An apparatus as recited in claim 16, wherein the controller further is operable to designate at the encoder one of the valid long term reference frames to be a preferred long term reference frame to use as a reference frame for predictive coding when the controller ascertains that there has been an error in receiving in at least one of the decoders, and wherein the controller is operable to from time to time change which valid long term reference frame is the preferred long term reference frame.

19. An apparatus as recited in claim 15, wherein the controller includes a state machine to control the sending of the reference frame message.

20. An apparatus as recited in claim 14, wherein the indicated frame in the reference frame message is only valid for use as a long term reference frame as a result of the encoder'"'"'s receiving feedback from all decoders that each has successfully received the reference frame message and the indicated frame, and wherein each of the plurality of decoders includes a reference frame store operable to store a set of reference frames including one or more frames marked for use as possible long term reference frames, such that a valid long term reference frame is one that exists and is known to the encoder to exist in the reference frame store of each decoder and marked for use as a long term reference frame.

21. An apparatus as recited in claim 14, wherein the plurality of decoders include more than one decoder of a decoding system at a single destination, the more than one decoder of the decoding system operable in parallel to each decode a different part of a frame.

22. An apparatus as recited in claim 14, wherein the plurality of decoders include decoders at different destinations.

23. An apparatus comprising:

means for encoding data to form compressed video data;

means for embedding a reference frame message with the compressed video data, the reference frame message to command a decoder to create a long term reference frame from a recent frame indicated in the reference frame message;

means for sending the compressed video data including the reference frame message to a plurality of decoders at one or more destination points; and

means for receiving feedback from the plurality of decoders, the feedback indicative of whether or not the decoders successfully received the reference frame message and have access to the indicated recent frame,

control means operable, in the case that the received feedback is such that at least one of the decoders did not successfully receive the reference frame message or does not have the indicated recent frame, to cause the means for encoding, means to embedding, and means for sending to repeat sending a reference frame message to create the long term reference frame from an indicated recent frame to the plurality of decoders, and operable to cause the means for receiving feedback until feedback has been received from each of the plurality of decoders indicating successful receipt of the reference frame message and indicating having access to the recent frame indicated in the reference frame message,

such that once the means for receiving has an indication of successful receipt and access at all decoders, the control means, upon the means for receiving feedback of an error from at least one of the decoders, is operable to cause the means for encoding to use the long term reference frame as a reference frame for temporal predictive encoding of video, thereby enabling the means for encoding to avoid using an independently coded frame upon the apparatus receiving feedback of an error.

24. An apparatus as recited in claim 23, wherein the control means is operable, from time to time to cause sending an additional reference frame message to create a different long term reference frame from a different recent frame indicated in the reference frame message, the sending being to the plurality of decoders at the one or more destination points, such the long term reference frame is updated from time to time with an updated frame, any updated frame not becoming valid for use as a reference frame until the apparatus has received feedback from all decoders that each has successfully received the reference frame message and has the updated frame.

25. Logic encoded in one or more tangible media at an encoder for execution and when executed operable to implement a method comprising:

an encoder at a source sending compressed video data including a reference frame message to create a long term reference frame from a recent frame indicated in the reference frame message, the sending being to a plurality of decoders at one or more destination points; and

receiving feedback from the plurality of decoders indicative of whether or not the decoders successfully received the reference frame message and have access to the indicated recent frame,

in the case that the received feedback is such that at least one of the decoders did not successfully receive the reference frame message or does not have the indicated recent frame, repeating sending a reference frame message to create the long term reference frame from an indicated recent frame to the plurality of decoders, and receiving feedback until the encoder has received feedback from each of the plurality of decoders indicating successful receipt of the reference frame message and indicating having access to the recent frame indicated in the reference frame message,

such that once the encoder has an indication of successful receipt and access at all decoders, the encoder, upon receiving feedback of an error from at least one of the decoders, is operable to use the long term reference frame as a reference frame for temporal predictive encoding of video, thereby enabling the encoder to avoid using an independently coded frame upon receiving feedback of an error.

26. Logic as recited in claim 25, further comprising,

the encoder from time to time sending an additional reference frame message to create a different long term reference frame from a different recent frame indicated in the reference frame message, the sending being to the plurality of decoders at the one or more destination points, such the long term reference frame is updated from time to time with an updated frame, any updated frame not becoming valid for use as a reference frame until the encoder has received feedback from all decoders that each has successfully received the reference frame message and has the updated frame.