Method and apparatus for encoder assisted-frame rate up conversion (EA-FRUC) for video compression

• US 20060165176A1
• Filed: 07/20/2005
• Published: 07/27/2006
• Est. Priority Date: 07/20/2004
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Walker, Gordon Kent, Shi, Fang, Raveendran, Vijayalakshmi R.
• Granted Patent: US 8,374,246 B2
• Time in Patent Office: Days
• US Class Current: 375/240.160
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/142 Detection of scene cut or scene change
• CPC Class Codes: H04N 19/15 by monitoring actual compressed data size at the memory before deciding storage at the transmission buffer
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/194 involving only two passes
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/583 Motion compensation with overlapping blocks
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression

Abstract

An Encoder Assisted Frame Rate Up Conversion (EA-FRUC) system that utilizes video coding and pre-processing operations at the video encoder to exploit the FRUC processing that will occur in the decoder in order to improve compression efficiency and reconstructed video quality is disclosed. One operation of the EA-FRUC system involves determining whether to encode a frame in a sequence of frames of a video content by determining a spatial activity in a frame of the sequence of frames; determining a temporal activity in the frame; determining a spatio-temporal activity in the frame based on the determined spatial activity and the determined temporal activity; determining a level of a redundancy in the source frame based on at least one of the determined spatial activity, the determined temporal activity, and the determined spatio-temporal activity; and, encoding the non-redundant information in the frame if the determined redundancy is within predetermined thresholds.

Claims

1. A method for construction of a video sequence including a sequence of frames comprising:

determining an amount of a type of activity in the sequence of frames, the type of activity selected from a group consisting of spatial activity, temporal and spatial-temporal activity;

determining a redundancy in the activity; and

,

encoding the frame if the determined redundancy is below a predetermined threshold.

2. The method of claim 1, wherein determining the spatial activity in the sequence of frames comprises determining an amount of texture information in at least one frame of the sequence.

3. The method of claim 1, wherein determining the temporal activity in the sequence of frames comprises determining an amount of temporal correlation and redundancy between at least two frames in the sequence of frames.

4. The method of claim 1, wherein determining the spatial-temporal activity in the frame of the sequence of frames comprises determining an amount of temporal correlation and redundancy of an amount of texture information between at least two frames in the sequence of frames.

5. The method of claim 1, wherein determining the redundancy in the activity comprises determining redundancy using at least one spatial activity measure selected from the group consisting of a contrast ratio measure, a spatial complexity measure and a variance measure.

6. The method of claim 1, wherein determining the redundancy in the activity comprises determining redundancy using at least one temporal activity measure selected from the group consisting of a motion field intensity measure, a temporal complexity measure, a sum of absolute differences measure.

7. The method of claim 1, wherein determining the redundancy in the activity comprises determining redundancy by comparing at least two activity measures selected from the group consisting of a correlation of spatial activity measures between a plurality of neighboring frames, a directivity measure, a joint behaviors between regions with varying spatial activity measures, a motion field intensity measure, a temporal complexity measure, and a sum of absolute differences measure.

8. A method for determining differential information between two frames comprising determining a difference measure selected from a group consisting of a pixel differences measure, a motion information differences measure, a mode decision threshold measure and an interpolated frame refinement measure, wherein the determination of the difference measure is performed using a frame rate up conversion process.

9. A method for coding differential information comprising using at least one technique selected from a group consisting of a motion compensation process, a motion vector transform process, a motion vector quantization process and entropy coding process, the at least one technique being specified in a video coding standard, wherein a standards-compliant processor is able to process the differential information in conjunction with a frame rate up conversion process to generate a video frame.

10. A method for processing a video bitstream, the video bitstream having differential information contained therein, the method comprising:

using an entropy coding technique to code differential information in the video bitstream selected from a group consisting of a variable length coding technique, a Huffman coding technique, and an arithmetic coding technique; and

,

conveying the coded information in a user data syntax specified in a video coding standard.

11. The method of claim 10, further comprising generating a standards compliant video bitstream.

12. A method for processing a video bitstream with differential information encoded therein, the encoded differential information being stored in a user data syntax, the method comprising:

extracting the encoded differential information from the user data syntax;

decoding the differential information; and

generating a video frame using the decoded differential information in a frame rate up conversion process.

13. A computer readable medium having instructions stored thereon for causing a computer to execute a method for construction of a video sequence including a sequence of frames, the method comprising:

determining an amount of a type of activity in the sequence of frames, the type of activity selected from a group consisting of spatial activity, temporal and spatial-temporal activity;

determining a redundancy in the activity; and

encoding the frame if the determined redundancy is below a predetermined threshold.

14. The computer readable medium of claim 13, wherein determining the spatial activity in the sequence of frames comprises determining an amount of texture information in at least one frame of the sequence.

15. The computer readable medium of claim 13, wherein determining the temporal activity in the sequence of frames comprises determining an amount of temporal correlation and redundancy between at least two frames in the sequence of frames.

16. The computer readable medium of claim 13, wherein determining the spatial-temporal activity in the frame of the sequence of frames comprises determining an amount of temporal correlation and redundancy of an amount of texture information between at least two frames in the sequence of frames.

17. The computer readable medium of claim 13, wherein determining the redundancy in the activity comprises determining redundancy using at least one spatial activity measure selected from the group consisting of a contrast ratio measure, a spatial complexity measure and a variance measure.

18. The computer readable medium of claim 13, wherein determining the redundancy in the activity comprises determining redundancy using at least one temporal activity measure selected from the group consisting of a motion field intensity measure, a temporal complexity measure, a sum of absolute differences measure.

19. The computer readable medium of claim 13, wherein determining the redundancy in the activity comprises determining redundancy by comparing at least two activity measures selected from the group consisting of a correlation of spatial activity measures between a plurality of neighboring frames, a directivity measure, a joint behaviors between regions with varying spatial activity measures, a motion field intensity measure, a temporal complexity measure, and a sum of absolute differences measure.

20. A computer readable medium having instructions stored thereon for causing a computer to execute a method for determining differential information between two frames comprising determining a difference measure selected from a group consisting of a pixel differences measure, a motion information differences measure, a mode decision threshold measure and an interpolated frame refinement measure, wherein the determination of the difference measure is performed using a frame rate up conversion process.

21. A computer readable medium having instructions stored thereon for causing a computer to execute a method for coding differential information comprising using at least one technique selected from a group consisting of a motion compensation process, a motion vector transform process, a motion vector quantization process and entropy coding process, the at least one technique being specified in a video coding standard, wherein a standards-compliant processor is able to process the differential information in conjunction with a frame rate up conversion process to generate a video frame.

22. A computer readable medium having instructions stored thereon for causing a computer to execute a method for processing a video bitstream, the video bitstream having differential information contained therein, the method comprising:

using an entropy coding technique to code differential information in the video bitstream selected from a group consisting of a variable length coding technique, a Huffman coding technique, and an arithmetic coding technique; and

conveying the coded information in a user data syntax specified in a video coding standard.

23. The computer readable medium of claim 22, further comprising generating a standards compliant video bitstream.

24. A computer readable medium having instructions stored thereon for causing a computer to execute a method for processing a video bitstream with differential information encoded therein, the encoded differential information being stored in a user data syntax, the method comprising:

extracting the encoded differential information from the user data syntax; and

decoding the differential information; and

generating a video frame using a frame rate up conversion process.

25. An apparatus for construction of a video sequence including a sequence of frames comprising:

means for determining an amount of a type of activity in the sequence of frames, the type of activity selected from a group consisting of spatial activity, temporal and spatial-temporal activity;

means for determining a redundancy in the activity; and

means for encoding the frame if the determined redundancy is below a predetermined threshold.

26. The apparatus of claim 25, wherein the means for determining the spatial activity in the sequence of frames comprises means for determining an amount of texture information in at least one frame of the sequence.

27. The apparatus of claim 25, wherein the means for determining the temporal activity in the sequence of frames comprises means for determining an amount of temporal correlation and redundancy between at least two frames in the sequence of frames.

28. The apparatus of claim 25, wherein the means for determining the spatial-temporal activity in the frame of the sequence of frames comprises means for determining an amount of temporal correlation and redundancy of an amount of texture information between at least two frames in the sequence of frames.

29. The apparatus of claim 25, wherein the means for determining the redundancy in the activity comprises means for determining redundancy using at least one spatial activity measure selected from the group consisting of a contrast ratio measure, a spatial complexity measure and a variance measure.

30. The apparatus of claim 25, wherein the means for determining the redundancy in the activity comprises means for determining redundancy using at least one temporal activity measure selected from the group consisting of a motion field intensity measure, a temporal complexity measure, a sum of absolute differences measure.

31. The apparatus of claim 25, wherein the means for determining the redundancy in the activity comprises means for determining redundancy by comparing at least two activity measures selected from the group consisting of a correlation of spatial activity measures between a plurality of neighboring frames, a directivity measure, a joint behaviors between regions with varying spatial activity measures, a motion field intensity measure, a temporal complexity measure, and a sum of absolute differences measure.

32. An apparatus for determining differential information between two frames comprising means for determining a difference measure selected from a group consisting of a pixel differences measure, a motion information differences measure, a mode decision threshold measure and an interpolated frame refinement measure, wherein the determination of the difference measure is performed using a frame rate up conversion process.

33. An apparatus for coding differential information comprising means for using at least one technique selected from a group consisting of a motion compensation process, a motion vector transform process, a motion vector quantization process and entropy coding process, the at least one technique being specified in a video coding standard, wherein a standards-compliant processor is able to process the differential information in conjunction with a frame rate up conversion process to generate a video frame.

34. An apparatus for processing a video bitstream, the video bitstream having differential information contained therein, comprising:

means for using an entropy coding technique to code differential information in the video bitstream selected from a group consisting of a variable length coding technique, a Huffman coding technique, and an arithmetic coding technique; and

means for conveying the coded information in a user data syntax specified in a video coding standard.

35. The apparatus of claim 34, further comprising means for generating a standards compliant video bitstream.

36. An apparatus for processing a video bitstream with differential information encoded therein, the encoded differential information being stored in a user data syntax, comprising:

means for extracting the encoded differential information from the user data syntax;

means for decoding the differential information; and

means for generating a video frame using the decoded differential information in a frame rate up conversion process.