FRAME EXTRAPOLATION VIA MOTION VECTORS

• US 20200137409A1
• Filed: 10/26/2018
• Published: 04/30/2020
• Est. Priority Date: 10/26/2018
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: MICHAIL, Ashraf Ayman, BOULTON, Michael George
• Application Number:
  US16/171,969
• Publication Number:
  US 20200137409A1
• Time in Patent Office: Days
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/521 for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution

Abstract

Examples are disclosed that relate to producing an extrapolated frame based on motion vectors. One example provides a computing device comprising a logic machine and a storage machine comprising instructions executable by the logic machine to, for each block of one or more blocks of pixels in rendered image data, generate a motion vector indicating motion between a current frame and a prior frame, and for each block of the one or blocks, extrapolate a predicted block of pixels from the current frame based on the motion vector and one or more prior motion vectors for the block, the one or more prior motion vectors determined via one or more corresponding frames preceding the prior frame. The instructions are further executable to produce an extrapolated frame comprising the predicted block of pixels for each block of the one or more blocks, and display the extrapolated frame.

Claims

1. A computing device, comprising:

a logic machine; and

a storage machine comprising instructions executable by the logic machine to

for each block of one or more blocks of pixels in rendered image data, generate a motion vector indicating motion between a current frame and a prior frame;

for each block of the one or blocks, extrapolate a predicted block of pixels from the current frame based on the motion vector and one or more prior motion vectors for the block, the one or more prior motion vectors determined via one or more corresponding frames preceding the prior frame;

produce an extrapolated frame comprising the predicted block of pixels for each block of the one or more blocks; and

display the extrapolated frame.

2. The computing device of claim 1, wherein the instructions are executed in response to detecting that the current frame does not meet a framerate condition.

3. The computing device of claim 1, wherein the predicted block of pixels is extrapolated based upon a magnitude and a direction of the motion vector and respective magnitudes and respective directions of the one or more prior motion vectors.

4. The computing device of claim 1, wherein a respective contribution of each of the one or more prior motion vectors to the predicted block of pixels decays as a number of frames separating the current frame and the corresponding frame increases.

5. The computing device of claim 1, wherein the motion vector is weighted with a weight, the weight determined based upon a spatial correspondence between the motion vector for the block and one or more current motion vectors in spatially proximate blocks.

6. The computing device of claim 5, wherein the spatial correspondence is determined via a kernel, and wherein the kernel is configured to output a respective weight for each of the one or more current motion vectors.

7. The computing device of claim 1, wherein the instructions executable to generate the motion vector for each block of the one or more blocks are executed on a video encoder.

8. The computing device of claim 1, wherein the motion vector is generated via an application that generates the rendered image data.

9. The computing device of claim 1, further comprising instructions executable to determine whether the motion vector for the block meets a discard condition, and not use the motion vector for generating the predicted block of pixels.

10. At a computing device, a method, comprising:

for each block of one or more blocks of pixels in rendered image data, generating a motion vector indicating motion between a current frame and a prior frame;

for each block of the one or blocks, extrapolating a predicted block of pixels from the current frame based on the motion vector and one or more prior motion vectors for the block, the one or more prior motion vectors determined via one or more corresponding frames preceding the prior frame;

producing an extrapolated frame comprising the predicted block of pixels for each block of the one or more blocks; and

displaying the extrapolated frame.

11. The method of claim 10, wherein the method is executed in response to detecting that the current frame does not meet a framerate condition.

12. The method of claim 10, wherein the predicted block of pixels is extrapolated based upon a magnitude and a direction of the motion vector and respective magnitudes and respective directions of the one or more prior motion vectors.

13. The method of claim 10, wherein a respective contribution of each of the one or more prior motion vectors to the predicted block of pixels decays as a number of frames separating the current frame and the corresponding frame increases.

14. The method of claim 10, wherein the motion vector is weighted with a weight, the weight determined based upon a spatial correspondence between the motion vector for the block and one or more current motion vectors in spatially proximate blocks.

15. The method of claim 14, wherein the spatial correspondence is determined via a kernel, and wherein the kernel is configured to output a respective weight for each of the one or more current motion vectors.

16. The method of claim 10, wherein the motion vector generated for each block of the one or more blocks is generated via a video encoder.

17. The method of claim 10, wherein the motion vector is generated via an application that generates the rendered image data.

18. The method of claim 10, further comprising determining whether the motion vector for the block meets a discard condition, and not using the motion vector for extrapolating the predicted block of pixels for the block if the discard condition is met.

19. A computing device, comprising:

a logic machine; and

a storage machine comprising instructions executable by the logic machine to

for each block of one or more blocks of pixels in rendered image data, generate a motion vector indicating motion between a current frame and a prior frame;

for each block of the one or blocks, extrapolate a predicted block of pixels from the current frame based upon a spatial correspondence between the motion vector and one or more current motion vectors in spatially proximate blocks, and also based upon a temporal correspondence between the motion vector and one or more prior motion vectors for the block, the one or more prior motion vectors determined via one or more corresponding frames preceding the prior frame;

produce an extrapolated frame comprising the predicted block of pixels for each block of the one or more blocks; and

display the extrapolated frame.

20. The computing device of claim 19, wherein a respective contribution of each of the one or more prior motion vectors to the predicted block of pixels decays as a number of frames separating the current frame and the corresponding frame increases.