METHOD AND APPARATUS FOR ERROR RESILIENT LONG TERM REFERENCING BLOCK REFRESH

• US 20120106632A1
• Filed: 10/28/2010
• Published: 05/03/2012
• Est. Priority Date: 10/28/2010
• Status: Abandoned Application

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Zhang, Dazhong, Zhou, Xiaosong, Wu, Hsi-Jung
• Application Number:
  US12/914,650
• Publication Number:
  US 20120106632A1
• Time in Patent Office: Days
• US Class Current: 375/240.12
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/89 involving methods or arrangements for detection of transmission errors at the decoder

Abstract

A system and method for coding video data wherein a pixel block may be coded for refresh with reference to an LTR frame that was successfully transmitted, or has a high probability of having been successfully transmitted from the encoder to the decoder. Not all pixel blocks in the frame may be refreshed at the same rate. Pixel blocks containing edge details, containing a significant object, or containing foreground image data may be refreshed more often than pixel blocks containing smooth, background, or relatively less significant image data.

Claims

1. A video coding method, comprising:

determining, with reference to an error resiliency policy, whether a pixel block in a frame is to be coded as a refresh pixel block;

if the pixel block is to be coded as a refresh pixel block, coding the pixel block according to predictive techniques with reference to a stored long term reference (LTR) frame; and

if the pixel block is not to be coded as a refresh pixel block, coding the pixel block according to predictive coding techniques with reference to a reference frame.

2. The method of claim 1, wherein the error resiliency mandates that each pixel block location in a frame area is to be coded as a refresh pixel block at a predetermined refresh rate.

3. The method of claim 2, further comprising increasing the refresh rate of a given pixel block location based on image content of the given pixel block.

4. The method of claim 2, further comprising increasing the refresh rate of a given pixel block location when the given pixel block contains an edge.

5. The method of claim 2, further comprising increasing the refresh rate of a given pixel block location when the given pixel block contains an object.

6. The method of claim 2, further comprising increasing the refresh rate of a given pixel block location when the given pixel block contains image content classified as foreground content.

7. The method of claim 1, further comprising:

storing according to the error resiliency policy, a refresh counter for each pixel block location in a frame area, wherein the determining includes evaluating a refresh counter for the pixel block;

after the frame is coded, identifying the pixel block(s) that have been coded predictively with reference to an LTR frame; and

resetting the refresh counters of the identified pixel block(s).

8. A video decoding method, comprising:

upon reception of coded video data representing a long term reference (LTR) frame,

decoding the coded LTR frame data,

storing the decoded LTR frame data, and

transmitting an acknowledgement of the coded LTR frame data to an encoder;

upon reception of coded video data representing a frame containing coded refresh pixel blocks, the refresh pixel blocks selected according to an error resiliency policy, decoding the coded refresh pixel blocks according to predictive decoding techniques, using the stored LTR frame data as a source of prediction.

9. The method of claim 8, wherein if a refresh pixel block is in an edge-free area of the frame, refreshing a neighboring pixel block in the edge-free area by interpolating the neighboring pixel block from the refresh pixel block.

10. A coded video signal, generated according to a process, comprising:

for each pixel block in a frame, determining, with reference to an error resiliency policy, whether the respective pixel block is to be coded as a refresh pixel block;

if the pixel block is to be coded as a refresh pixel block, coding the pixel block according to predictive techniques with reference to a stored long term reference (LTR) frame;

if the pixel block is not to be coded as a refresh pixel block, coding the pixel block according to predictive coding techniques with reference to a reference frame; and

transmitting the coded frame data from an encoder on a physical data path.

11. A video coding method, comprising:

for each pixel block of a frame, determining, with reference to an error resiliency policy, a refresh count of the respective pixel block;

if the refresh count value is close to a maximum refresh value of the error resiliency policy, searching among locally stored LTR frames for a stored pixel block to be used for predictive coding of the respective pixel block;

if the stored pixel block adequately matches the respective pixel block, coding the respective pixel block using the stored pixel block as a prediction reference; and

if the stored pixel block does not adequately match the respective pixel block, coding the respective pixel block using a stored pixel block of another reference frame as a prediction reference.

12. The coding method of claim 11, further comprising, if the refresh count value is not close to a maximum refresh value, searching among all locally-stored reference frames for a pixel block that matches the respective pixel block and coding the respective pixel block with reference to the stored pixel block identified therefrom.

13. The coding method of claim 11, wherein the refresh count value is determined to be close to the maximum refresh value if it is within a predetermined number of the maximum refresh value.

14. The coding method of claim 11, wherein the stored pixel block is determined to adequately match the respective pixel block in response to an estimate of prediction errors obtained from the stored pixel block and the respective pixel block.

15. The coding method of claim 14, further comprising comparing the error estimate is to an error threshold.

16. The coding method of claim 15, wherein the error estimate varies based on a difference between the refresh count value of the respective pixel block and the maximum refresh value.

17. The coding method of claim 11, further comprising, following coding of the frame, resetting refresh count values of all pixel blocks that have been coded with reference to an LTR frame.

18. A video coding method, comprising:

selecting a pixel block in a frame to be coded as a refresh pixel block;

determining if an LTR frame is available for use in coding the pixel block; and

coding the remaining frame according to predictive coding techniques;

wherein if an LTR frame is available, using the LTR frame as a reference for coding the pixel block according to predictive coding techniques.

19. The method of claim 18 wherein if an LTR frame is not available, coding the pixel block as an I-block.

20. The method of claim 18 wherein the selected pixel block is refreshed more often than a second pixel block in the frame.

21. The method of claim 18 wherein multiple LTR frames are selected for coding the pixel block.

22. The method of claim 18, wherein the first pixel block is selected as a refresh pixel block in part because the first pixel block'"'"'s image content contains edges.

23. The method of claim 18, wherein the first pixel block is selected as a refresh pixel block in part because the first pixel block'"'"'s image content contains an object.

24. The method of claim 18, wherein the first pixel block is selected as a refresh pixel block in part because the first pixel block'"'"'s image content is classified as foreground content.

25. The method of claim 18, wherein the LTR frame is a frame that has been acknowledged as successfully received by a decoder.

26. The method of claim 18, wherein the LTR frame is a frame that has a probability of having been successfully received by a decoder above a predetermined threshold.

27. The method of claim 26, wherein the frame has an increased probability of having been successfully received by the decoder if the frame is small.

28. The method of claim 26, wherein the frame has an increased probability of having been successfully received by the decoder if forward error correction is implemented at the decoder.

29. The method of claim 26, wherein the frame has an increased probability of having been successfully received by the decoder if network conditions are adequate for a successful transmission.

30. A method of decoding video data, comprising:

decoding frames of a received video data; and

identifying a pixel block of a received frame as a refresh pixel block

wherein if the identified pixel block is in an edge-free area of the frame, a neighboring pixel block in the edge-free area of the frame is refreshed by interpolating the neighboring pixel block from the identified pixel block.

31. A video coder, comprising:

a coding engine to code input video data according to predictive coding techniques;

a reference picture cache to store decoded video data of coded reference frames, the reference picture cache storing data of long term reference (LTR) frames which have been acknowledged by a decoder and non-LTR frames;

a controller, to control operation of the coding engine and, responsive to an error resiliency policy, determine whether a pixel block in a frame is to be coded as a refresh pixel block; and

if the pixel block is to be coded as a refresh pixel block, cause the coding engine to code the pixel block according to predictive techniques with reference to a stored long term reference (LTR) frame; and

if the pixel block to not to be coded as a refresh pixel block, cause the coding engine to code the pixel block according to predictive coding techniques with reference to a reference frame.

32. The video coder of claim 31, wherein the error resiliency mandates that the controller code each pixel block location in a frame area as a refresh pixel block at a predetermined rate.

33. The video coder of claim 32, wherein the refresh rate of a given pixel block location is increased based on image content of the given pixel block.

34. The video coder of claim 32, wherein the refresh rate of a given pixel block location is increased when the given pixel block contains an edge.

35. The video coder of claim 32, wherein the refresh rate of a given pixel block location is increased when the given pixel block contains an object.

36. The video coder of claim 32, wherein the refresh rate of a given pixel block location is increased when the given pixel block contains image content classified as foreground content.

37. The video coder of claim 31, wherein the controller stores a refresh counter for each pixel block location in a frame area, and the controller evaluates the refresh counter for a pixel block in a frame to determine whether the pixel block is to be coded as a refresh pixel block and after the frame is coded, the controller identifies the pixel block(s) that have been coded predictively with reference to an LTR frame and resets the refresh counters of the identified pixel block(s).

38. A video decoder, comprising:

a decoding engine to decode input coded video data representing a long term reference (LTR) frame data;

a reference picture cache to store the decoded LTR frame data; and

a controller, to control operation of the decoding engine and to transmit an acknowledgement of the coded LTR frame data to an encoder;

wherein upon reception of coded video data representing a frame containing coded refresh pixel blocks, the refresh pixel blocks selected according to an error resiliency policy, decoding the coded refresh pixel blocks according to predictive decoding techniques, using the stored LTR frame data as a source of prediction.

39. The decoder of claim 38, wherein if the identified pixel block is in an edge-free zone of the frame, a neighboring pixel block in the edge-free zone of the frame is refreshed by interpolating the neighboring pixel block from the identified pixel block.