Method and apparatus for encoder assisted-frame rate up conversion (EA-FRUC) for video compression

• US 9,521,411 B2
• Filed: 02/06/2013
• Issued: 12/13/2016
• Est. Priority Date: 07/20/2004
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Raveendran, Vijayalakshmi R., Shi, Fang, Walker, Gordon Kent
• Primary Examiner(s): PHILIPPE, GIMS S
• Application Number:
  US13/760,991
• Publication Number:
  US 20130188742A1
• Time in Patent Office: 1,406 Days
• Field of Search: 375/E7.19, 375/E7.254, 375/240.16, 375/240.23, 375/240.15, 382/236, 382/239, 382/300, 382/233, 382/232
• US Class Current: 1/1
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/142 Detection of scene cut or scene change
• CPC Class Codes: H04N 19/15 by monitoring actual compressed data size at the memory before deciding storage at the transmission buffer
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/194 involving only two passes
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/583 Motion compensation with overlapping blocks
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression

Abstract

An Encoder Assisted Frame Rate Up Conversion (EA-FRUC) system that utilizes video coding and pre-processing operations at the video encoder to exploit the FRUC processing that will occur in the decoder in order to improve compression efficiency and reconstructed video quality is disclosed. One operation of the EA-FRUC system involves determining whether to encode a frame in a sequence of frames of a video content by determining a spatial activity in a frame of the sequence of frames; determining a temporal activity in the frame; determining a spatio-temporal activity in the frame based on the determined spatial activity and the determined temporal activity; determining a level of a redundancy in the source frame based on at least one of the determined spatial activity, the determined temporal activity, and the determined spatio-temporal activity; and, encoding the non-redundant information in the frame if the determined redundancy is within predetermined thresholds.

Claims

1. A method for processing a video bitstream, the method comprising:

using an entropy coding technique to code differential information in the video bitstream selected from a group consisting of a variable length coding technique, a Huffman coding technique, and an arithmetic coding technique, wherein the differential information comprises differential information with respect to a predicted frame coding and an interpolated frame generated by a frame rate up conversion process; and

conveying the coded information in a user data syntax specified in a video coding standard.

2. The method of claim 1, further comprising:

generating a standards compliant video bitstream which includes the coded information.

3. The method of claim 1, further comprising:

using the differential information of the coded information to assist a decoder with frame rate up conversion process.

4. The method of claim 3, wherein the decoder assistance provided by the differential information of the coded information is to enhance a quality of reconstructed video.

5. The method of claim 3, wherein the decoder assistance provided by the differential information of the coded information is to reduce a computational load on the decoder.

6. The method of claim 5, wherein the computational load on the decoder is reduced by the coded information aiding a frame rate up conversion engine of the decoder in making decisions during a mode decision process.

7. The method of claim 1, further comprising:

determining the differential information between two frames by determining a difference measure selected from a group consisting of a pixel differences measure, a motion information differences measure, a mode decision threshold measure and an interpolated frame refinement measure, wherein the determination of the difference measure is performed using a frame rate up conversion process.

8. The method of claim 1, wherein the entropy coding process is specified in a video coding standard, and wherein a standards-compliant processor is able to process the differential information in conjunction with a frame rate up conversion process to generate a video frame.

9. The method of claim 1, further comprising:

processing the video bitstream with differential information encoded therein by extracting the coded differential information from the user data syntax, decoding the differential information, and generating a video frame using the decoded differential information in a frame rate up conversion process.

10. A non-transitory tangible computer readable medium having instructions stored thereon for causing a computer to execute a method for processing a video bitstream, the video bitstream having differential information contained therein, the method comprising:

using an entropy coding technique to code differential information in the video bitstream selected from a group consisting of a variable length coding technique, a Huffman coding technique, and an arithmetic coding technique, wherein the differential information comprises differential information with respect to a predicted frame coding and an interpolated frame generated by a frame rate up conversion process; and

conveying the coded information in a user data syntax specified in a video coding standard.

11. The computer readable medium of claim 10, further comprising:

generating a standards compliant video bitstream.

12. The computer readable medium of claim 10, further comprising:

using the differential information of the coded information to assist a decoder with the frame rate up conversion process.

13. The computer readable medium of claim 12, wherein the decoder assistance provided by the differential information of the coded information is to enhance a quality of reconstructed video.

14. The computer readable medium of claim 12, wherein the decoder assistance provided by the differential information of the coded information is to reduce a computational load on the decoder.

15. The computer readable medium of claim 14, wherein the computational load on the decoder is reduced by the coded information aiding a frame rate up conversion engine of the decoder in making decisions during a mode decision process.

16. The computer readable medium of claim 10, further comprising:

determining the differential information between two frames by determining a difference measure selected from a group consisting of a pixel differences measure, a motion information differences measure, a mode decision threshold measure and an interpolated frame refinement measure, wherein the determination of the difference measure is performed using a frame rate up conversion process.

17. The computer readable medium of claim 10, wherein the entropy coding process is specified in a video coding standard, and wherein a standards-compliant processor is able to process the differential information in conjunction with a frame rate up conversion process to generate a video frame.

18. The computer readable medium of claim 10, further comprising:

processing the video bitstream with differential information encoded therein by extracting the coded differential information from the user data syntax, decoding the differential information, and generating a video frame using the decoded differential information in a frame rate up conversion process.

19. An apparatus for processing a video bitstream, the video bitstream having differential information contained therein, the apparatus comprising:

means for entropy coding differential information in the video bitstream with a coding technique selected from a group consisting of a variable length coding technique, a Huffman coding technique, and an arithmetic coding technique, wherein the differential information comprises differential information with respect to a predicted frame coding and an interpolated frame generated by a frame rate up conversion process; and

means for conveying the coded information in a user data syntax specified in a video coding standard.

20. The apparatus of claim 19, further comprising:

means for generating a standards compliant video bitstream.

21. The apparatus of claim 19, further comprising:

means for using the differential information of the coded information to assist a decoder with frame rate up conversion process.

22. The apparatus of claim 21, wherein the decoder assistance provided by the differential information of the coded information is to enhance a quality of reconstructed video.

23. The apparatus of claim 21, wherein the decoder assistance provided by the differential information of the coded information is to reduce a computational load on the decoder.

24. The apparatus of claim 23, wherein the computational load on the decoder is reduced by the coded information aiding a frame rate up conversion engine of the decoder in making decisions during a mode decision process.

25. The apparatus of claim 19, further comprising:

means for determining the differential information between two frames by determining a difference measure selected from a group consisting of a pixel differences measure, a motion information differences measure, a mode decision threshold measure and an interpolated frame refinement measure, wherein the determination of the difference measure is performed using a frame rate up conversion process.

26. The apparatus of claim 19, wherein the entropy coding process is specified in a video coding standard, and wherein a standards-compliant processor is able to process the differential information in conjunction with a frame rate up conversion process to generate a video frame.

27. The apparatus of claim 19, further comprising:

means for processing the video bitstream with differential information encoded therein by extracting the coded differential information from the user data syntax, decoding the differential information, and generating a video frame using the decoded differential information in a frame rate up conversion process.