Using low-resolution frames to increase frame rate of high-resolution frames

• US 10,284,810 B1
• Filed: 11/08/2017
• Issued: 05/07/2019
• Est. Priority Date: 11/08/2017
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Shoa Hassani Lashdan, Alireza, Wong, Justin, Liu, Xiaojian, Toy, Louie Lee
• Primary Examiner(s): Yenke, Brian P
• Assistant Examiner(s): Haiem, Sean N.
• Application Number:
  US15/807,078
• Publication Number:
  US 20190141287A1
• Time in Patent Office: 545 Days
• Field of Search: None
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: H04N 7/0127 by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51

Abstract

A device obtains a plurality of original high-resolution frames and a low-resolution frame. The device generates, based on the plurality of original high-resolution frames, a first additional high-resolution frame. The first additional high-resolution frame and the low-resolution frame correspond to a same output time. The device generates a down-sampled frame by down-sampling the first additional higher-resolution frame. Additionally, the device determines, based on comparisons of blocks of the low-resolution frame and blocks of the down-sampled frame, a plurality of weights. The device generates an up-sampled frame by up-sampling the low-resolution frame. The device generates a second additional high-resolution frame based on a weighted average of the up-sampled frame and the first additional high-resolution frame. The weighted average is based on the plurality of weights. The second additional high-resolution frame corresponds to the same output time as the first additional high-resolution frame and the low-resolution frame.

Claims

1. A method of increasing a frame rate of video data performed by one or more devices, the method comprising:

obtaining a plurality of original higher-resolution frames and a lower-resolution frame, each of the plurality of original higher-resolution frames having a first resolution, the lower-resolution frame having a second resolution lower than the first resolution;

generating, based on the plurality of original higher-resolution frames, a first additional higher-resolution frame, the first additional higher-resolution frame having the first resolution, the first additional higher-resolution frame and the lower-resolution frame corresponding to a same particular output time;

generating a down-sampled frame by down-sampling the first additional higher-resolution frame, the down-sampled frame having the second resolution, wherein;

a set of corresponding blocks includes a first block, a second block, a third block, and a fourth block, the first block being in the down-sampled frame, the second block being in the lower-resolution frame, the third block being in an up-sampled frame, and the fourth block being in the first additional higher-resolution frame, and

the first block, the second block, the third block, and the fourth block correspond to a same region of an image represented in the down-sampled frame, the lower-resolution frame, the up-sampled frame, and the first additional higher-resolution frame;

determining, based on comparisons of blocks of the lower-resolution frame and blocks of the down-sampled frame, a plurality of weights, wherein determining the plurality of weights comprises;

determining a difference metric that indicates a degree of difference between the first block and the second block; and

determining, based on the difference metric, a weight for the set of corresponding blocks, the weight for the set of corresponding blocks being one of the weights in the plurality of weights;

generating the up-sampled frame by up-sampling the lower-resolution frame, the up-sampled frame having the first resolution; and

generating a second additional higher-resolution frame based on a weighted average of the up-sampled frame and the first additional higher-resolution frame, wherein;

the weighted average is based on the plurality of weights, and wherein the second additional higher-resolution frame corresponds to the particular output time, and

generating the second additional higher-resolution frame comprises, for each sample in the third block, determining, based on the weight for the set of corresponding blocks, a sample in the second additional higher-resolution frame as a weighted average of the sample in the third block and a corresponding sample in the fourth block.

2. The method of claim 1, further comprising:

outputting the plurality of original higher-resolution frames and the second additional higher-resolution frame for display.

3. The method of claim 1, wherein determining the sample in the second additional higher-resolution frame comprises determining the sample in the second additional higher-resolution frame such that the sample in the second additional higher-resolution frame is equal to (w·

s₁+(1−

w)·

s₂), where w is the weight, s₁ is the sample in the third block, and s₂ is the sample in the fourth block.

4. The method of claim 1, wherein generating the first additional higher-resolution frame comprises:

extrapolating the first additional higher-resolution frame from the plurality of original higher-resolution frames.

5. The method of claim 4, wherein the particular output time is an output time of the first additional higher-resolution frame, and extrapolating the first additional higher-resolution frame comprises:

partitioning a first original higher-resolution frame of the plurality of original higher-resolution frames into a plurality of blocks; and

for each respective block in the plurality of blocks of the first original higher-resolution frame;

searching for a corresponding block in a second original higher-resolution frame of the plurality of original higher-resolution frames;

determining a motion vector indicating a spatial difference from the corresponding block in the second original higher-resolution frame to the respective block of the first original higher-resolution frame;

scaling the motion vector based on a temporal difference between an output time of the first original higher-resolution frame and the output time of the first additional higher-resolution frame; and

including the corresponding block in the first additional higher-resolution frame at a location indicated by the scaled motion vector.

6. The method of claim 1, wherein generating the first additional higher-resolution frame comprises:

interpolating the first additional higher-resolution frame from the plurality of original higher-resolution frames.

7. The method of claim 1, wherein interpolating the first additional higher-resolution frame comprises:

partitioning a first original higher-resolution frame of the plurality of original higher-resolution frames into a plurality of blocks; and

for each respective block in the plurality of blocks of the first original higher-resolution frame;

searching for a corresponding block in a second original higher-resolution frame of the plurality of original higher-resolution frames;

determining a motion vector indicating a spatial difference from the corresponding block in the second original higher-resolution frame to the respective block of the first original higher-resolution frame;

scaling the motion vector based on a temporal difference between an output time of the first original higher-resolution frame and the output time of the first additional higher-resolution frame; and

generating a prediction block based on a weighted average of the respective block in the first original higher-resolution frame and the corresponding block in the second original higher-resolution frame; and

including the prediction block in the first additional higher-resolution frame at a location indicated by the scaled motion vector.

8. The method of claim 1, wherein the blocks of the lower-resolution frame and the blocks of the down-sampled frame are each 2×

2 or larger.

9. A device comprising:

a memory configured to store a plurality of original higher-resolution frames and a lower-resolution frame, each of the plurality of original higher-resolution frames having a first resolution, the lower-resolution frame having a second resolution lower than the first resolution; and

one or more processing circuits configured to;

generate, based on the plurality of original higher-resolution frames, a first additional higher-resolution frame, the first additional higher-resolution frame having the first resolution, the first additional higher-resolution frame and the lower-resolution frame corresponding to a same particular output time;

generate a down-sampled frame by down-sampling the first additional higher-resolution frame, the down-sampled frame having the second resolution, wherein;

a set of corresponding blocks includes a first block, a second block, a third block, and a fourth block, the first block being in the down-sampled frame, the second block being in the lower-resolution frame, the third block being in an up-sampled frame, and the fourth block being in the first additional higher-resolution frame, and

the first block, the second block, the third block, and the fourth block correspond to a same region of an image represented in the down-sampled frame, the lower-resolution frame, the up-sampled frame, and the first additional higher-resolution frame;

determine, based on comparisons of blocks of the lower-resolution frame and blocks of the down-sampled frame, a plurality of weights, wherein the one or more processing circuits are configured such that, as part of determining the plurality of weights, the one or more processing circuits;

determine a difference metric that indicates a degree of difference between the first block and the second block; and

determine, based on the difference metric, a weight for the set of corresponding blocks, the weight for the set of corresponding blocks being one of the weights in the plurality of weights;

generate the up-sampled frame by up-sampling the lower-resolution frame, the up-sampled frame having the first resolution; and

generate a second additional higher-resolution frame based on a weighted average of the up-sampled frame and the first additional higher-resolution frame, wherein the weighted average is based on the plurality of weights, wherein;

the second additional higher-resolution frame corresponds to the particular output time, and

the one or more processing circuits are configured such that, as part of generating the second additional higher-resolution frame, the one or more processing circuits determine, for each sample in the third block, based on the weight for the set of corresponding blocks, a sample in the second additional higher-resolution frame as a weighted average of the sample in the third block and a corresponding sample in the fourth block.

10. The device of claim 9, further comprising a display device configured to display the plurality of original higher-resolution frames and the second additional higher-resolution frame.

11. The device of claim 9, wherein the one or more processing circuits are configured such that, as part of determining the sample in the second additional higher-resolution frame, the one or more processing circuits determine the sample in the second additional higher-resolution frame such that the sample in the second additional higher-resolution frame is equal to (w·

s₁+(1−

w)·

s₂), where w is the weight, s₁ is the sample in the third block, and s₂ is the sample in the fourth block.

12. The device of claim 9, wherein the one or more processing circuits are configured such that, as part of generating the first additional higher-resolution frame, the one or more processing circuits:

extrapolate the first additional higher-resolution frame from the plurality of original higher-resolution frames.

13. The device of claim 12, wherein the particular output time is an output time of the first additional higher-resolution frame, and the one or more processing circuits are configured such that, as part of extrapolating the first additional higher-resolution frame, the one or more processing circuits:

partition a first original higher-resolution frame of the plurality of original higher-resolution frames into a plurality of blocks; and

for each respective block in the plurality of blocks of the first original higher-resolution frame;

search for a corresponding block in a second original higher-resolution frame of the plurality of original higher-resolution frames;

determine a motion vector indicating a spatial difference from the corresponding block in the second original higher-resolution frame to the respective block of the first original higher-resolution frame;

scale the motion vector based on a temporal difference between an output time of the first original higher-resolution frame and the output time of the first additional higher-resolution frame; and

include the corresponding block in the first additional higher-resolution frame at a location indicated by the scaled motion vector.

14. The device of claim 9, wherein the one or more processing circuits are configured such that, as part of generating the first additional higher-resolution frame, the one or more processing circuits:

interpolate the first additional higher-resolution frame from the plurality of original higher-resolution frames.

15. The device of claim 9, wherein the one or more processing circuits are configured such that, as part of interpolating the first additional higher-resolution frame, the one or more processing circuits:

partition a first original higher-resolution frame of the plurality of original higher-resolution frames into a plurality of blocks;

for each respective block in the plurality of blocks of the first original higher-resolution frame;

search for a corresponding block in a second original higher-resolution frame of the plurality of original higher-resolution frames;

determine a motion vector indicating a spatial difference from the corresponding block in the second original higher-resolution frame to the respective block of the first original higher-resolution frame;

scale the motion vector based on a temporal difference between an output time of the first original higher-resolution frame and the output time of the first additional higher-resolution frame; and

generate a prediction block based on a weighted average of the respective block in the first original higher-resolution frame and the corresponding block in the second original higher-resolution frame; and

include the prediction block in the first additional higher-resolution frame at a location indicated by the scaled motion vector.

16. The device of claim 9, wherein the blocks of the lower-resolution frame and the blocks of the down-sampled frame are each 2×

2 or larger.

17. A device comprising:

means for storing a plurality of original higher-resolution frames and a lower-resolution frame, each of the plurality of original higher-resolution frames having a first resolution, the lower-resolution frame having a second resolution lower than the first resolution;

means for generating, based on the plurality of original higher-resolution frames, a first additional higher-resolution frame, the first additional higher-resolution frame having the first resolution, the first additional higher-resolution frame and the lower-resolution frame corresponding to a same particular output time;

means for generating a down-sampled frame by down-sampling the first additional higher-resolution frame, the down-sampled frame having the second resolution, wherein;

a set of corresponding blocks includes a first block, a second block, a third block, and a fourth block, the first block being in the down-sampled frame, the second block being in the lower-resolution frame, the third block being in an up-sampled frame, and the fourth block being in the first additional higher-resolution frame, and

the first block, the second block, the third block, and the fourth block correspond to a same region of an image represented in the down-sampled frame, the lower-resolution frame, the up-sampled frame, and the first additional higher-resolution frame;

means for determining, based on comparisons of blocks of the lower-resolution frame and blocks of the down-sampled frame, a plurality of weights, wherein the means for determining the plurality of weights comprises;

means for determining a difference metric that indicates a degree of difference between the first block and the second block; and

means for determining, based on the difference metric, a weight for the set of corresponding blocks, the weight for the set of corresponding blocks being one of the weights in the plurality of weights;

means for generating the up-sampled frame by up-sampling the lower-resolution frame, the up-sampled frame having the first resolution; and

means for generating a second additional higher-resolution frame based on a weighted average of the up-sampled frame and the first additional higher-resolution frame, wherein;

the weighted average is based on the plurality of weights, wherein the second additional higher-resolution frame corresponds to the particular output time, and

the means for generating the second additional higher-resolution frame comprises means for determining, for each sample in the third block, based on the weight for the set of corresponding blocks, a sample in the second additional higher-resolution frame as a weighted average of the sample in the third block and a corresponding sample in the fourth block.

18. A non-transitory computer-readable data storage medium having instructions stored thereon that, when executed, cause one or more processing circuits to:

obtain a plurality of original higher-resolution frames and a lower-resolution frame, each of the plurality of original higher-resolution frames having a first resolution, the lower-resolution frame having a second resolution lower than the first resolution;

generate, based on the plurality of original higher-resolution frames, a first additional higher-resolution frame, the first additional higher-resolution frame having the first resolution, the first additional higher-resolution frame and the lower-resolution frame corresponding to a same particular output time;

generate a down-sampled frame by down-sampling the first additional higher-resolution frame, the down-sampled frame having the second resolution, wherein;

a set of corresponding blocks includes a first block, a second block, a third block, and a fourth block, the first block being in the down-sampled frame, the second block being in the lower-resolution frame, the third block being in an up-sampled frame, and the fourth block being in the first additional higher-resolution frame, and

the first block, the second block, the third block, and the fourth block correspond to a same region of an image represented in the down-sampled frame, the lower-resolution frame, the up-sampled frame, and the first additional higher-resolution frame;

determine, based on comparisons of blocks of the lower-resolution frame and blocks of the down-sampled frame, a plurality of weights, wherein as part of causing the one or more processing circuits to determine the plurality of weights, execution of the instructions causes the one or more processing circuits to;

determine a difference metric that indicates a degree of difference between the first block and the second block; and

determine, based on the difference metric, a weight for the set of corresponding blocks, the weight for the set of corresponding blocks being one of the weights in the plurality of weights;

generate the up-sampled frame by up-sampling the lower-resolution frame, the up-sampled frame having the first resolution; and

generate a second additional higher-resolution frame based on a weighted average of the up-sampled frame and the first additional higher-resolution frame, wherein;

the weighted average is based on the plurality of weights, wherein the second additional higher-resolution frame corresponds to the particular output time, and

as part of causing the one or more processing circuits to generate the second additional higher-resolution frame, execution of the instructions causes the one or more processing circuits to determine, for each sample in the third block, based on the weight for the set of corresponding blocks, a sample in the second additional higher-resolution frame as a weighted average of the sample in the third block and a corresponding sample in the fourth block.