Fragmented reference in temporal compression for video coding

• US 8,908,763 B2
• Filed: 06/25/2008
• Issued: 12/09/2014
• Est. Priority Date: 06/25/2008
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Raveendran, Vijayalakshmi R.
• Primary Examiner(s): Lee, Andrew
• Application Number:
  US12/145,900
• Publication Number:
  US 20090323809A1
• Time in Patent Office: 2,358 Days
• Field of Search: 375/240.12, 375/240.16, 375/240.22
• US Class Current: 375/240.12
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/426 using memory downsizing methods
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/513 Processing of motion vectors

Abstract

In general, this disclosure describes techniques for encoding and decoding sequences of video frames using fragmentary reference pictures. The disclosure presents video encoding and decoding techniques for modified temporal compression based on fragmented references rather than complete reference pictures. In a typical sequence of video frames, only a portion (i.e., a tile) of each frame includes moving objects. Moreover, in each frame, the moving objects tend to be confined to specific areas that are common among each frame in the sequence of video frames. As described herein, such common areas of motion are identified. Pictures are then extracted from the identified areas of the video frames. Because these pictures may represent only portions of the frames, this disclosure refers to these pictures as “fragments.” It is then these fragments that are used as reference pictures for generating predicted frames during a motion compensation process, rather than the entire frame.

Claims

1. A method comprising:

storing reference fragments comprising video data associated with a set of tiles of video frames in a sequence of video frames, wherein each tile in the set of tiles represents a subset of partitions for the respective video frame; and

performing motion compensation for a current tile of a current video frame based on one or more of the reference fragments;

wherein the current tile is co-located with the tiles in the set of tiles;

wherein the method further comprises automatically identifying the set of tiles; and

wherein automatically identifying the set of tiles comprises;

identifying a set of active partitions in each of the video frames, wherein each of the active partitions comprises at least one of a motion vector indicating at least some displacement relative to a partition of a video frame of the reference fragments, and a non-zero residual value vis-à

-vis a co-located partition of a video frame of the reference fragments; and

for each given one of the video frames, identifying a tile of the given one of the video frames such that the tile includes the identified set of active partitions of the given one of the video frames and partitions of the given one of the video frames that are co-located with active partitions in the identified sets of active partitions of other ones of the video frames.

2. The method of claim 1:

wherein performing motion compensation comprises generating a predictive fragment associated with the current tile of the current video frame; and

wherein the method further comprises;

extracting a fragment associated with the current tile from the current video frame;

using the extracted fragment and the predictive fragment to generate a residual fragment associated with the current tile of the current video frame;

performing a transform on the residual fragment in order to generate a set of coefficients;

quantizing the set of coefficients, thereby generating a set of quantized coefficients;

entropy encoding the set of quantized coefficients; and

outputting the entropy encoded set of quantized coefficients as part of a media object;

inverse quantizing the set of quantized coefficients, thereby generating an inverse quantized set of coefficients;

performing an inverse of the transform on the inverse quantized set of coefficients in order to generate a decoded residual fragment;

generating a reconstructed fragment using the predictive fragment and the decoded residual fragment; and

storing the reconstructed fragment for use as a reference fragment.

3. The method of claim 1,

wherein performing motion compensation comprises generating a predictive fragment associated with the current tile of the current video frame;

and

wherein the method further comprises;

receiving a media object that includes a set of data associated with the current tile of the current video frame;

generating a residual fragment based on the set of data;

using the predictive fragment and the residual fragment to generate a reconstructed fragment associated with the current tile of the current video frame; and

using the reconstructed fragment to reconstruct the current video frame.

4. The method of claim 3, wherein using the reconstructed fragment to reconstruct the current video frame comprises modifying a previously decoded complete video frame such that the reconstructed fragment is specified in a tile of the previously decoded complete video frame that is co-located with the current tile of the current video frame.

5. The method of claim 1,

wherein the reference fragments are interlaced;

and

wherein performing motion compensation for the current tile comprises performing de-interlacing motion compensation for the current tile based on one or more of the reference fragments.

6. The method of claim 1, wherein the method further comprises adding the current video frame between two video frames in the sequence of video frames as part of an operation to increase the frame rate of the sequence of video frames.

7. A device comprising:

a reference buffer that stores reference fragments comprising video data associated with a set of tiles of video frames in a sequence of video frames, wherein each tile in the set of tiles represents a subset of partitions for the respective video frame; and

a motion compensation unit that performs motion compensation for a current tile of a current video frame based on one or more of the reference fragments;

wherein the current tile is co-located with the tiles in the set of tiles; and

wherein the device further comprises a tile identification module that automatically identifies the set of tiles; and

wherein the tile identification module comprises;

a partition activity module that identifies a set of active partitions in each of the video frames, wherein each of the active partitions comprises a motion vector indicating at least some displacement relative to a partition of a video frame of the reference fragments, and wherein each of the active partitions comprises a non-zero residual value vis-à

-vis a co-located partition of a video frame of the reference fragments; and

a tile construction module that, for each given one of the video frames, identifies a tile of the given one of the video frames such that the tile includes the identified set of active partitions of the given one of the video frames and partitions of the given one of the video frames that are co-located with active partitions in the identified sets of active partitions of other ones of the video frames.

8. The device of claim 7,

wherein, when the motion compensation unit performs motion compensation for the current tile, the motion compensation unit generates a predictive fragment associated with the current tile of the current video frame;

and

wherein the device further comprises;

a fragment extraction module that extracts a fragment associated with the current tile from the current video frame,

a module that uses the extracted fragment and the predictive fragment to generate a residual fragment associated with the current tile of the current video frame;

a block transform unit that performs a transform on the residual fragment in order to generate a set of coefficients;

a quantization module that quantizes the set of coefficients, thereby generating a set of quantized coefficients;

an entropy encoding unit that entropy encodes the set of quantized coefficients;

an output module that outputs the entropy encoded set of quantized coefficients as part of a media object;

an inverse quantization module that inverse quantizes the set of quantized coefficients, thereby generating an inverse quantized set of coefficients;

an inverse transform unit that performs an inverse of the transform on the inverse quantized set of coefficients in order to generate a decoded residual fragment; and

a module that generates a reconstructed fragment using the predictive fragment and the decoded residual fragment,

wherein the reference buffer stores the reconstructed fragment for use as a reference fragment.

9. The device of claim 7,

wherein, when the motion compensation unit performs motion compensation for the current tile, the motion compensation unit generates a predictive fragment associated with the current tile of the current video frame;

and

wherein the device receives a media object that includes a set of data associated with the current tile of the current video frame; and

wherein the device further comprises;

a decoder that generates a residual fragment based on the set of data;

a module that uses the predictive fragment and the residual fragment to generate a reconstructed fragment associated with the current tile of the current video frame; and

a frame reassembly module that uses the reconstructed fragment to reconstruct the current video frame.

10. The device of claim 9, wherein the frame reassembly module modifies a previously decoded complete video frame such that the reconstructed fragment is specified in a tile of the previously decoded complete video frame that is co-located with the current tile of the current video frame.

11. The device of claim 7,

wherein the reference fragments are interlaced;

and

wherein the motion compensation unit performs the motion compensation for the current tile by performing de-interlacing motion compensation for the current tile based on one or more of the reference fragments.

12. The device of claim 7, wherein the device further comprises a post-processor that adds the current video frame between two video frames in the sequence of video frames as part of an operation to increase the frame rate of the sequence of video frames.

13. A device comprising:

means for storing reference fragments comprising video data associated with a set of tiles of video frames in a sequence of video frames, wherein each tile in the set of tiles represents a subset of partitions for the respective video frame; and

means for performing motion compensation for a current tile of a current video frame based on one or more of the reference fragments;

wherein the device further comprises means for automatically identifying the set of tiles;

wherein the means for automatically identifying the set of tiles comprises;

means for identifying a set of active partitions in each of the video frames wherein each of the active partitions comprises at least one of a motion vector indicating at least some displacement relative to a partition of a video frame of the reference fragments, and a non-zero residual value vis-à

-vis a co-located partition of a video frame of the reference fragments; and

means for identifying, for each given one of the video frames, a tile of the given one of the video frames such that the tile includes the identified set of active partitions of the given one of the video frames and partitions of the given one of the video frames that are co-located with active partitions in the identified sets of active partitions of other ones of the video frames; and

wherein the current tile is co-located with the tiles in the set of tiles.

14. The device of claim 13,

wherein, when the means for performing motion compensation performs motion compensation for the current tile, the means for performing motion compensation generates a predictive fragment associated with the current tile of the current video frame;

and

wherein the device further comprises;

means for extracting a fragment associated with the current tile from the current video frame,

means for using the extracted fragment and the predictive fragment to generate a residual fragment associated with the current tile of the current video frame;

means for performing a transform on the residual fragment in order to generate a set of coefficients;

means for quantizing the set of coefficients, thereby generating a set of quantized coefficients;

means for entropy encoding the set of quantized coefficients;

means for outputting the entropy encoded set of quantized coefficients as part of a media object;

means for inverse quantizing the set of quantized coefficients, thereby generating an inverse quantized set of coefficients;

means for performing an inverse of the transform on the inverse quantized set of coefficients in order to generate a decoded residual fragment; and

means for generating a reconstructed fragment using the predictive fragment and the decoded residual fragment,

wherein the means for storing the set of fragments stores the reconstructed fragment for use as a reference fragment.

15. The device of claim 13, wherein, when the means for performing motion compensation performs motion compensation for the current tile, the means for performing motion compensation generates a predictive fragment associated with the current tile of the current video frame;

wherein the device receives a media object that includes a set of data associated with the current tile of the current video frame; and

wherein the device further comprises;

means for generating a residual fragment based on the set of data;

means for using the predictive fragment associated with the current tile of the current video frame and the residual fragment to generate a reconstructed fragment associated with the current tile of the current video frame; and

means for using the reconstructed fragment to reconstruct the current video frame.

16. The device of claim 15, wherein the means for using the reconstructed fragment to reconstruct the current video frame modifies a previously decoded complete video frame such that the reconstructed fragment is specified in a tile of the previously decoded complete video frame that is co-located with the current tile of the current video frame.

17. The device of claim 13,

wherein the reference fragments are interlaced;

and

wherein the means for performing motion compensation performs the motion compensation for the current tile by performing de-interlacing motion compensation for the current tile based on one or more of the reference fragments.

18. The device of claim 13, wherein the device further comprises means for adding the current video frame between two video frames in the sequence of video frames as part of an operation to increase the frame rate of the sequence of video frames.

19. A computer-readable medium comprising executable instructions that, when executed by one or more processors, cause one or more processors to:

store reference fragments comprising video data associated with a set of tiles of video frames in a sequence of video frames, wherein each tile in the set of tiles represents a subset of partitions for the respective video frame; and

perform motion compensation for a current tile of a current video frame based on one or more of the reference fragments;

wherein the instructions further cause the one or more processors to automatically identify the set of tiles;

wherein the instructions cause the one or more processors to automatically identify the set of tiles at least in part by causing the one or more processors to;

identify a set of active partitions in each of the video frames, wherein each of the active partitions comprises at least one of a motion vector indicating at least some displacement relative to a partition of a video frame of the reference fragments, and a non-zero residual value vis-à

-vis a co-located partition of a video frame of the reference fragments; and

for each given one of the video frames, identify a tile of the given one of the video frames such that the tile includes the identified set of active partitions of the given one of the video frames and partitions of the given one of the video frames that are co-located with active partitions in the identified sets of active partitions of other ones of the video frames; and

wherein the current tile is co-located with the tiles in the set of tiles.

20. The computer-readable medium of claim 19,

wherein the instructions cause the one or more processors to perform motion compensation at least in part by causing the one or more processors to generate a predictive fragment associated with the current tile of the current video frame;

and

wherein the instructions further cause the one or more processors to;

extract a fragment associated with the current tile from the current video frame;

use the extracted fragment and the predictive fragment to generate a residual fragment;

perform a transform on the residual fragment in order to generate a set of coefficients;

quantize the set of coefficients, thereby generating a set of quantized coefficients;

entropy encode the set of quantized coefficients;

output the entropy encoded set of quantized coefficients as part of a media object;

inverse quantize the set of quantized coefficients, thereby generating an inverse quantized set of coefficients;

perform an inverse of the transform on the inverse quantized set of coefficients in order to generate a decoded residual fragment;

generate a reconstructed fragment using the predictive fragment and the decoded residual fragment; and

store the reconstructed fragment for use as a reference fragment.

21. The computer-readable medium of claim 19,

wherein the instructions cause the one or more processors to perform motion compensation at least in part by causing the one or more processors to generate a predictive fragment associated with the current tile of the current video frame;

and

wherein the instructions further cause the one or more processors to;

receive a media object that includes a set of data associated with the current tile of the current video frame;

generate a residual fragment based on the set of data;

use the predictive fragment associated with the tile of the current video frame and the residual fragment to generate a reconstructed fragment associated with the current frame; and

use the reconstructed fragment to reconstruct the current video frame.

22. The computer-readable medium of claim 21, wherein the instructions cause the one or more processors to use the reconstructed fragment to reconstruct the current video frames at least in part by causing the one or more processors to modify a previously decoded complete video frame such that the reconstructed fragment is specified in a tile of the previously decoded complete video frame that is co-located with the current tile of the current video frame.

23. The computer-readable medium of claim 19,

wherein the reference fragments are interlaced;

and

wherein performing motion compensation for the current tile comprises performing de-interlacing motion compensation for the current tile based on one or more of the reference fragments.

24. The computer-readable medium of claim 19, wherein the instructions further cause the one or more processors to adding the current video frame between two video frames in the sequence of video frames as part of an operation to increase the frame rate of the sequence of video frames.

25. An integrated circuit comprising:

circuitry that stores reference fragments comprising video data associated with a set of tiles of video frames in a sequence of video frames, wherein each tile in the set of tiles represents a subset of partitions for the respective video frame; and

circuitry that performs motion compensation for a current tile of a current video frame based on one or more of the reference fragments;

wherein the integrated circuit further comprises circuitry that automatically identifies the set of tiles;

wherein the circuitry that automatically identifies the set of tiles comprises;

circuitry that identifies a set of active partitions in each of the video frames, wherein each of the active partitions comprises at least one of a motion vector indicating at least some displacement relative to a partition of a video frame of the reference fragments, and a non-zero residual value vis-à

-vis a co-located partition of a video frame of the reference fragments; and

circuitry that identifies, for each given one of the video frames, a tile of the given one of the video frames such that the tile includes the identified set of active partitions of the given one of the video frames and partitions of the given one of the video frames that are co-located with active partitions in the identified sets of active partitions of other ones of the video frames; and

wherein the current tile is co-located with the tiles in the set of tiles.

26. The integrated circuit of claim 25,

wherein, when the circuitry that performs motion compensation performs motion compensation for the current tile, the circuitry that performs motion compensation generates a predictive fragment associated with the current tile of the current video frame;

and

wherein the integrated circuit further comprises;

circuitry that extracts a fragment associated with the current tile from the current video frame,

circuitry that uses the extracted fragment and the predictive fragment to generate a residual fragment associated with the current tile of the current video frame;

circuitry that performs a transform on the residual fragment in order to generate a set of coefficients;

circuitry that quantizes the set of coefficients, thereby generating a set of quantized coefficients;

circuitry that entropy encodes the set of quantized coefficients; and

circuitry that outputs the entropy encoded set of quantized coefficients as part of a media object;

circuitry that inverse quantizes the set of quantized coefficients, thereby generating an inverse quantized set of coefficients;

circuitry that performs an inverse of the transform on the inverse quantized set of coefficients in order to generate a decoded residual fragment; and

circuitry that generates a reconstructed fragment using the predictive fragment and the decoded residual fragment,

wherein the circuitry that stores the set of fragments stores the reconstructed fragment for use as a reference fragment.

27. The integrated circuit of claim 25,

wherein, when the circuitry that performs motion compensation performs motion compensation for the current tile, the circuitry that performs motion compensation generates a predictive fragment associated with the current tile of the current video frame;

wherein the integrated circuit receives a media object that includes a set of data associated with the current tile of the current video frame; and

wherein the integrated circuit further comprises;

circuitry that generates a residual fragment based on the set of data;

circuitry that uses the predictive fragment associated with the current tile of the current video frame and the residual fragment to generate a reconstructed fragment associated with the current tile of the current video frame; and

circuitry that uses the reconstructed fragment to reconstruct the current video frame.

28. The integrated circuit of claim 27, wherein the circuitry for using the reconstructed fragment to reconstruct the current video frame modifies a previously decoded complete video frame such that the reconstructed fragment is specified in a tile of the previously decoded complete video frame that is co-located with the current tile of the current video frame.

29. The integrated circuit of claim 25,

wherein the reference fragments are interlaced;

and

wherein the circuitry that performs motion compensation performs the motion compensation for the current tile by performing de-interlacing motion compensation for the current tile based on one or more of the reference fragments.

30. The integrated circuit of claim 25, wherein the integrated circuit further comprises circuitry that adds the current video frame between two video frames in the sequence of video frames as part of an operation to increase the frame rate of the sequence of video frames.

31. A method comprising:

storing reference fragments comprising video data associated with a set of tiles of video frames in a sequence of video frames, wherein each tile in the set of tiles represents a subset of partitions for the respective video frame; and

performing a video processing technique for a current tile of a current video frame based on one or more of the reference fragments;

wherein the current tile is co-located with the tiles in the set of tiles;

wherein the method further comprises automatically identifying the set of tiles; and

wherein automatically identifying the set of tiles comprises;

identifying a set of active partitions in each of the video frames, wherein each of the active partitions comprises at least one of a motion vector indicating at least some displacement relative to a partition of a video frame of the reference fragments, and a non-zero residual value vis-à

-vis a co-located partition of a video frame of the reference fragments;

for each given one of the video frames, identifying a tile of the given one of the video frames such that the tile includes the identified set of active partitions of the given one of the video frames and partitions of the given one of the video frames that are co-located with active partitions in the identified sets of active partitions of other ones of the video frames.

32. The method of claim 31, wherein performing the video processing technique comprises one of:

performing motion compensation for a current tile of a current video frame based on one or more of the reference fragments;

performing scaling for a current tile of a current video frame based on one or more of the reference fragments; and

performing edge enhancement for a current tile of a current video frame based on one or more of the reference fragments.

33. A device comprising:

a reference buffer that stores reference fragments comprising video data associated with a set of tiles of video frames in a sequence of video frames, wherein each tile in the set of tiles represents a subset of partitions for the respective video frame; and

a unit that performs a video processing technique for a current tile of a current video frame based on one or more of the reference fragments;

wherein the current tile is co-located with the tiles in the set of tiles; and

wherein the device further comprises a tile identification module that automatically identifies the set of tiles; and

wherein the tile identification module comprises;

a partition activity module that identifies a set of active partitions in each of the video frames, wherein each of the active partitions comprises a motion vector indicating at least some displacement relative to a partition of a video frame of the reference fragments, and wherein each of the active partitions comprises a non-zero residual value vis-à

-vis a co-located partition of a video frame of the reference fragments; and

a tile construction module that, for each given one of the video frames, identifies a tile of the given one of the video frames such that the tile includes the identified set of active partitions of the given one of the video frames and partitions of the given one of the video frames that are co-located with active partitions in the identified sets of active partitions of other ones of the video frames.

34. The device of claim 33, wherein the unit that performs the video processing technique comprises one of:

a motion compensation unit that perform motion compensation for a current tile of a current video frame based on one or more of the reference fragments;

a scaling unit that performs scaling for a current tile of a current video frame based on one or more of the reference fragments; and

an edge enhancement unit that performs edge enhancement for a current tile of a current video frame based on one or more of the reference fragments.

35. A device comprising:

means for storing reference fragments comprising video data associated with a set of tiles of video frames in a sequence of video frames, wherein each tile in the set of tiles represents a subset of partitions for the respective video frame; and

means for performing a video processing technique for a current tile of a current video frame based on one or more of the reference fragments;

wherein the device further comprises means for automatically identifying the set of tiles;

wherein the means for automatically identifying the set of tiles comprises;

means for identifying a set of active partitions in each of the video frames wherein each of the active partitions comprises at least one of a motion vector indicating at least some displacement relative to a partition of a video frame of the reference fragments, and a non-zero residual value vis-à

-vis a co-located partition of a video frame of the reference fragments; and

means for identifying, for each given one of the video frames, a tile of the given one of the video frames such that the tile includes the identified set of active partitions of the given one of the video frames and partitions of the given one of the video frames that are co-located with active partitions in the identified sets of active partitions of other ones of the video frames; and

wherein the current tile is co-located with the tiles in the set of tiles.

36. The device of claim 35, wherein means for performing the video processing technique comprises one of:

means for performing motion compensation for a current tile of a current video frame based on one or more of the reference fragments;

means for performing scaling for a current tile of a current video frame based on one or more of the reference fragments; and

means for performing edge enhancement for a current tile of a current video frame based on one or more of the reference fragments.

37. A computer-readable medium comprising executable instructions that, when executed by one or more processors, cause one or more processors to:

store reference fragments comprising video data associated with a set of tiles of video frames in a sequence of video frames, wherein each tile in the set of tiles represents a subset of partitions for the respective video frame; and

perform a video processing technique for a current tile of a current video frame based on one or more of the reference fragments;

wherein the instructions further cause the one or more processors to automatically identify the set of tiles;

wherein the instructions cause the one or more processors to automatically identify the set of tiles at least in part by causing the one or more processors to;

identify a set of active partitions in each of the video frames, wherein each of the active partitions comprises at least one of a motion vector indicating at least some displacement relative to a partition of a video frame of the reference fragments, and a non-zero residual value vis-à

-vis a co-located partition of a video frame of the reference fragments; and

for each given one of the video frames, identify a tile of the given one of the video frames such that the tile includes the identified set of active partitions of the given one of the video frames and partitions of the given one of the video frames that are co-located with active partitions in the identified sets of active partitions of other ones of the video frames; and

wherein the current tile is co-located with the tiles in the set of tiles.

38. The computer readable medium of claim 37, wherein the instructions cause the one or more processors to perform the video processing technique by performing one of:

motion compensation for a current tile of a current video frame based on one or more of the reference fragments;

scaling for a current tile of a current video frame based on one or more of the reference fragments; and

edge enhancement for a current tile of a current video frame based on one or more of the reference fragments.

39. An integrated circuit comprising:

circuitry that stores reference fragments comprising video data associated with a set of tiles of video frames in a sequence of video frames, wherein each tile in the set of tiles represents a subset of partitions for the respective video frame; and

circuitry that performs a video processing technique for a current tile of a current video frame based on one or more of the reference fragments;

wherein the integrated circuit further comprises circuitry that automatically identifies the set of tiles;

wherein the circuitry that automatically identifies the set of tiles comprises;

circuitry that identifies a set of active partitions in each of the video frames, wherein each of the active partitions comprises at least one of a motion vector indicating at least some displacement relative to a partition of a video frame of the reference fragments, and a non-zero residual value vis-à

-vis a co-located partition of a video frame of the reference fragments; and

circuitry that identifies, for each given one of the video frames, a tile of the given one of the video frames such that the tile includes the identified set of active partitions of the given one of the video frames and partitions of the given one of the video frames that are co-located with active partitions in the identified sets of active partitions of other ones of the video frames; and

wherein the current tile is co-located with the tiles in the set of tiles.

40. The integrated circuit of claim 39, wherein the circuitry that performs a video processing technique comprises one of:

circuitry that performs motion compensation for a current tile of a current video frame based on one or more of the reference fragments;

circuitry that performs scaling for a current tile of a current video frame based on one or more of the reference fragments; and

circuitry that performs edge enhancement for a current tile of a current video frame based on one or more of the reference fragments.