LOW-COMPLEXITY METHOD FOR GENERATING SYNTHETIC REFERENCE FRAMES IN VIDEO CODING

• US 20170111652A1
• Filed: 03/16/2016
• Published: 04/20/2017
• Est. Priority Date: 10/15/2015
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Davies, Thomas
• Granted Patent: US 10,805,627 B2
• Time in Patent Office: Days
• CPC Class Codes: H04N 19/189 characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures

Abstract

Presented herein are techniques for a low-complexity process of generating an artificial frame that can be used for prediction. At least a first reference frame and a second reference frame of a video signal are obtained. A synthetic reference frame is generated from the first reference frame and the second reference frame. Reference blocks from each of the first reference frame and the second reference frame are combined to derive an interpolated block of the synthetic reference frame.

Claims

1. A method comprising:

obtaining at least a first reference frame and a second reference frame of a video signal; and

generating a synthetic reference frame from the first reference frame and the second reference frame by;

dividing the synthetic reference frame into a plurality of blocks;

searching for motion vectors in the first reference frame and the second reference frame for each of the plurality of blocks in the synthetic reference frame;

deriving motion vector information for each of the plurality of blocks in the synthetic reference frame from motion vectors identified in each of the first reference frame and the second reference frame;

identifying reference blocks in each of the first reference frame and the second reference frame using the motion vector information for each of the plurality of blocks in the synthetic reference frame; and

combining reference blocks from each of the first reference frame and the second reference frame to derive an interpolated block of the synthetic reference frame.

2. The method of claim 1, wherein a temporal position of the synthetic reference frame with respect to the first reference frame and the second reference frames is determined by integer weights that are proportional to a distance between the synthetic reference frame and the first and second reference frames, respectively, such that a temporally nearer reference frame has a larger weight.

3. The method of claim 2, wherein dividing comprises dividing the plurality of blocks into a plurality of sub-blocks, searching comprises identifying, for each sub-block, a primary motion vector in a farther one of the first and second reference frames, and deriving comprises deriving a secondary motion vector in a nearer one of the first and second reference frames from the primary motion vector by scaling the primary motion vector to an appropriate scale.

4. The method of claim 3, wherein the primary motion vector is determined to pixel accuracy and the secondary motion vector is rounded to achieve a same level of accuracy as the primary motion vector.

5. The method of claim 3, wherein searching comprises matching a block or sub-block in the farther one of the first and second reference frames obtained by displacing a corresponding block from the synthetic reference frame by the primary motion vector, with a block or sub-block in the nearer one of the first and second reference frames obtained by displacing the same block in the synthetic reference frame by the secondary motion vector.

6. The method of claim 5, wherein searching comprises performing hierarchical motion estimation at multiple scales starting with smaller blocks using motion vector information at resolution layer n for motion vector information derivation at next larger resolution layer n+1.

7. The method of claim 6, further comprising scaling down the first and second reference frames R0 and R1, respectively, by a factor of ½

vertically and horizontally to form reference frames R0(n), R1(n) of size width/2′

×

height/2′

, wherein motion vectors from a lower layer when scaled by a factor 2 are available as motion vector candidates for a next higher layer, as well as motion vectors from spatial neighbors at the same layer, and wherein block sizes are the same at each layer, so that each block at layer n+1 corresponds to 4 blocks in layer n.

8. The method of claim 7, wherein searching comprises, for each of the plurality of blocks in raster order:

determining if motion estimation can be bypassed;

if motion estimation cannot be bypassed, determining candidate motion vectors from lower layer blocks and from neighbor blocks in raster order; and

performing adaptive cross searching around each candidate motion vector to determine a best motion vector among all those searched for each block.

9. The method of claim 8, wherein searching further comprises, for each of the sub-blocks in raster order, determining a best merge candidate motion vector from either a motion vector of an original block of which a given sub-block is a part or one of the motion vectors of one of neighboring sub-blocks to the given sub-block.

10. The method of claim 7, wherein determining if motion estimation can be bypassed comprises:

for a given block, computing a skip vector from neighboring blocks or a default value; and

determining whether the skip vector is valid based on a value computed for each sub-block of the given block; and

if the skip vector is determined to be valid, bypassing motion estimation.

11. The method of claim 6, wherein for a given block, candidates for motion vector search include spatial neighbors above and to the left of the given block and specific sets of guide motion vector arrays at co-located positions, below the given block and to the right of the given block, wherein a main guide motion vector array is an array of motion data formed from performing motion estimation at a next smaller size in the hierarchical motion estimation.

12. The method of claim 3, wherein scaling comprises using first and second weights in computing the secondary motion vector from the primary motion vector, and wherein the weights are equal or unequal.

13. The method of claim 1, wherein the obtaining and the generating are separately performed at each of an encoder and a decoder.

14. The method of claim 13, further comprising:

transmitting from the encoder to the decoder one or more parameters for the decoder to use in performing the generating of the synthetic reference frame.

15. The method of claim 14, wherein the one or more parameters include:

a global motion vector, motion vector search ranges and number of searches, a skip threshold to determine when to bypass motion estimation, and size of the blocks.

16. The method of claim 1, wherein combining comprises averaging the reference blocks from each of the first reference frame and the second reference frame.

17. The method of claim 16, wherein the averaging comprising an unweighted averaging or a weighted averaging.

18. The method of claim 17, wherein the weighted averaging uses integer weights proportional to a distance between the synthetic reference frame and the first reference frame, and a distance between the synthetic reference frame and the second reference frame.

19. An apparatus comprising:

a communication interface unit configured to enable communications over a communication channel;

a processor configured to;

obtain at least first and second reference frames of a video signal;

generate a synthetic reference frame from the first reference frame and the second reference frame by;

dividing the synthetic reference frame into a plurality of blocks;

searching for motion vectors in the first reference frame and the second reference frame for each of the plurality of blocks in the synthetic reference frame;

deriving motion vector information for each of the plurality of blocks in the synthetic reference frame from motion vectors identified in each of the first reference frame and the second reference frame;

identifying reference blocks in each of the first reference frame and the second reference frame using the motion vector information for each of the plurality of blocks in the synthetic reference frame; and

combining reference blocks from each of the first reference frame and the second reference frame to derive an interpolated block of the synthetic reference frame.

20. The apparatus of claim 19, wherein the processor is configured to divide the plurality of blocks into a plurality of sub-blocks, identify, for each sub-block, a primary motion vector in a farther one of the first and second reference frames, and derive a secondary motion vector in a nearer one of the first and second reference frames from the primary motion vector by scaling the primary motion vector to an appropriate scale.

21. The apparatus of claim 20, wherein the processor is configured to match a block or sub-block in the farther one of the first and second reference frames obtained by displacing a corresponding block from the synthetic reference frame by the primary motion vector, with a block or sub-block in the nearer one of the first and second reference frames obtained by displacing the same block in the synthetic reference frame by the secondary motion vector.

22. One or more non-transitory computer readable storage media encoded with software comprising computer executable instructions and when the software is executed operable to perform operations comprising:

obtaining at least first and second reference frames of a video signal; and

generating a synthetic reference frame from the first reference frame and the second reference frame by;

dividing the synthetic reference frame into a plurality of blocks;

searching for motion vectors in the first reference frame and the second reference frame for each of the plurality of blocks in the synthetic reference frame;

deriving motion vector information for each of the plurality of blocks in the synthetic reference frame from motion vectors identified in each of the first reference frame and the second reference frame;

identifying reference blocks in each of the first reference frame and the second reference frame using the motion vector information for each of the plurality of blocks in the synthetic reference frame; and

combining reference blocks from each of the first reference frame and the second reference frame to derive an interpolated block of the synthetic reference frame.

23. The non-transitory computer readable storage media of claim 22, wherein the instructions operable for dividing comprise instructions operable for dividing the plurality of blocks into a plurality of sub-blocks, the instructions operable for searching comprise instructions operable for identifying, for each sub-block, a primary motion vector in a farther one of the first and second reference frames, and the instructions operable for deriving comprise instructions operable for deriving a secondary motion vector in a nearer one of the first and second reference frames from the primary motion vector by scaling the primary motion vector to an appropriate scale.

24. The non-transitory computer readable storage media of claim 23, wherein the instructions operable for searching comprise instructions operable for matching a block or sub-block in a farther one of the first and second reference frames obtained by displacing a corresponding block from the synthetic reference frame by the primary motion vector, with a block or sub-lock in the nearer one of the first and second reference frames obtained by displacing the same block in the synthetic reference frame by the secondary motion vector.