EFFICIENT SPATIO-TEMPORAL VIDEO UP-SCALING

• US 20100135395A1
• Filed: 12/03/2008
• Published: 06/03/2010
• Est. Priority Date: 12/03/2008
• Status: Abandoned Application

• Current Assignee: Sharp Kabushiki Kaisha (Hon Hai Precision Industry Co., Ltd.)
• Original Assignee: Sharp Kabushiki Kaisha (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Servais, Marc Paul, Kay, Andrew
• Application Number:
  US12/327,011
• Publication Number:
  US 20100135395A1
• Time in Patent Office: Days
• US Class Current: 375/240.160
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/156 Availability of hardware or computational resources, e.g. encoding based on power-saving criteria
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/61 in combination with predictive coding

Abstract

A method of performing spatio-temporal up-scaling includes receiving an input video having a sequence of input frames, analyzing the input video to estimate motion vectors associated with the sequence of input frames, and determining corresponding motion compensation errors associated with the motion vectors. The method further includes determining an extent to which computational resources are to be respectively allocated to spatially up-scaling the sequence of input frames and temporally up-scaling the sequence of input frames, based on the estimated motion vectors and corresponding motion compensation errors. In addition, the method includes spatio-temporally up-scaling the sequence of input frames based on the determined extent.

Claims

1. A method of performing spatio-temporal up-scaling, comprising:

receiving an input video comprising a sequence of input frames,

analyzing the input video to estimate motion vectors associated with the sequence of input frames, and determining corresponding motion compensation errors associated with the motion vectors;

determining an extent to which computational resources are to be respectively allocated to spatially up-scaling the sequence of input frames and temporally up-scaling the sequence of input frames, based on the estimated motion vectors and corresponding motion compensation errors; and

spatio-temporally up-scaling the sequence of input frames based on the determined extent.

2. The method of claim 1, wherein when the motion vectors and corresponding motion compensation errors are indicative of relatively fast motion and small motion compensation error, the computational resources are biased towards temporal up-scaling.

3. The method of claim 1, wherein when the motion vectors and corresponding motion compensation errors are indicative of relatively slow motion and small motion compensation error, the computational resources are biased towards spatial up-scaling.

4. The method of claim 1, wherein when the corresponding motion compensation errors are indicative of relatively large motion compensation error, the computational resources are biased towards spatial up-scaling.

5. The method of claim 1, wherein the extent is determined frame-by-frame for the sequence of input frames.

6. The method of claim 1, wherein the extent is determined region-by-region within a given frame in the sequence of input frames.

7. The method of claim 1, wherein the motion vectors are estimated using at least one of block matching, phase correlation and gradient-based techniques.

8. The method of claim 2, wherein the computational resources are biased towards temporal up-scaling by applying motion-compensated frame interpolation for temporal up-scaling, and at least one of bicubic or bilinear scaling for spatial up-scaling, in the spatio-temporal up-scaling step.

9. The method of claim 3, wherein the computational resources are biased towards spatial up-scaling by applying multi-frame super resolution for spatial up-scaling, and frame repetition for temporal up-scaling, in the spatio-temporal up-scaling step.

10. The method of claim 4, wherein the computational resources are biased towards spatial up-scaling by applying single-frame spatial up-scaling for spatial up-scaling, and frame repetition for temporal scaling, in the spatio-temporal up-scaling step.

11. The method of claim 1, wherein the determining step comprises selecting a spatio-temporal up-scaling mode for the spatio-temporal up-scaling step from among a plurality of different predefined spatio-temporal up-scaling modes, based on the estimated motion vectors and corresponding motion compensation errors.

12. The method of claim 11, wherein each of the plurality of different predefined spatio-temporal up-scaling modes comprises a different combination of spatial up-scaling and temporal up-scaling methods.

13. The method of claim 11, wherein at least one of the plurality of different predefined spatio-temporal up-scaling modes utilizes motion estimation for at least one of spatial up-scaling and temporal up-scaling, and another of the plurality of different predefined spatio-temporal up-scaling modes does not utilize motion estimation.

14. The method of claim 13, wherein selection between the at least one predefined spatio-temporal up-scaling mode and the another predefined spatio-temporal up-scaling mode is based on whether the corresponding motion compensation error exceeds a predefined error threshold value.

15. The method of claim 1, wherein the corresponding motion compensation errors associated with the motion vectors are calculated using at least one of mean square error and mean absolute difference techniques.

16. The method of claim 11, wherein the analyzing step comprises calculating a motion speed metric using the motion vectors, and selection between the different predefined spatio-temporal up-scaling modes is based on whether the corresponding motion speed metric exceeds a predefined speed threshold value.

17. The method of claim 16, wherein the motion speed metric is based on at least one of average motion, maximum motion and maximum motion vector gradient of motion vectors in a given frame or region within the sequence of input frames.

18. A computer program comprising code stored on a computer-readable medium which, when executed by a computer, causes the computer to carry out the steps of:

receiving an input video comprising a sequence of input frames,

analyzing the input video to estimate motion vectors associated with the sequence of input frames, and determining corresponding motion compensation errors associated with the motion vectors;

determining an extent to which computational resources are to be respectively allocated to spatially up-scaling the sequence of input frames and temporally up-scaling the sequence of input frames, based on the estimated motion vectors and corresponding motion compensation errors; and

spatio-temporally up-scaling the sequence of input frames based on the determined extent.

19. A method of performing spatio-temporal up-scaling, comprising:

receiving an input video data comprising a sequence of input frames;

performing a combination of spatial up-scaling and temporal up-scaling to the sequence of input frames; and

modifying the combination dynamically as a function of at least one property of the sequence of input frames.

20. The method of claim 19, wherein the at least one property comprises motion exhibited in the sequence of input frames.