Methods and systems to estimate motion based on reconstructed reference frames at a video decoder

• US 9,538,197 B2
• Filed: 12/22/2014
• Issued: 01/03/2017
• Est. Priority Date: 07/03/2009
• Status: Active Grant

• Current Assignee: Tahoe Research Limited (f/k/a Learndale Limited) (Vector Capital Corporation)
• Original Assignee: Intel Corporation
• Inventors: Xu, Lidong, Chiu, Yi-Jen, Zhang, Wenhao
• Primary Examiner(s): Moore, Jr., Michael J
• Application Number:
  US14/578,802
• Publication Number:
  US 20150304682A1
• Time in Patent Office: 743 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: H04N 19/103 Selection of coding mode or of prediction mode
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/52 by predictive encoding
• CPC Class Codes: H04N 19/57 Motion estimation characterised by a search window with variable size or shape
• CPC Class Codes: H04N 19/573 Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/61 in combination with predictive coding

Abstract

Methods and systems to apply motion estimation (ME) based on reconstructed reference pictures in a B frame or in a P frame at a video decoder. For a P frame, projective ME may be performed to obtain a motion vector (MV) for a current input block. In a B frame, both projective ME and mirror ME may be performed to obtain an MV for the current input block. The ME process can be performed on sub-partitions of the input block, which may reduce the prediction error without increasing the amount of MV information in the bitstream. Decoder-side ME can be applied for the prediction of existing inter frame coding modes, and traditional ME or the decoder-side ME can be adaptively selected to predict a coding mode based on a rate distribution optimization (RDO) criterion.

Claims

1. A machine-implemented method to estimate motion at a video decoder, comprising:

defining, at the decoder, a motion estimation search window for a first coding block of a first frame of a video;

selecting, at the decoder, a motion vector to a block of a second frame of the video and a motion vector to a block of a third frame of the video based on the search window, distances between the first frame and each of the second and third frames, and a measure of visual similarity between the respective blocks of the second and third frames;

deriving, at the decoder, a motion vector for the first block based on the selected motion vector to a block of the second frame and the selected motion vector to a block of the third frame;

storing the motion vector for the first block to a memory coupled with, or included in, the video decoder; and

predicting the first block based on the stored motion vector and the block of the second or third frame.

2. The machine-implemented method of claim 1, wherein the selecting includes:

identifying motion vectors to blocks of the second frame based on the search window;

computing motion vectors to blocks of the third frame based on respective ones of the motion vectors to blocks of the second frame and the distances between the first frame and each of the second and third frames; and

selecting one of the motion vectors to blocks of the second frame and a corresponding one of the motion vectors to blocks of third frame based on a measure of visual similarity between the respective blocks of the second and third frames.

3. The machine-implemented method of claim 1, wherein the deriving a motion vector for the first block includes:

deriving the motion vector for the first block based on a combination of the selected motion vector to a block of the second frame, the selected motion vector to a block of the third frame, and the distances between the first frame and each of the second and third frames.

4. The machine-implemented method of claim 1, wherein the second frame is a forward reference frame relative to the first frame and the third frame is a backward frame relative to the first frame, and wherein the selecting includes:

identifying motion vectors to blocks of the forward reference frame based on the search window;

computing mirror motion vectors to blocks of the backward reference frame based on respective ones of the motion vectors to blocks of the forward reference frame and the distances between the first frame and each of the forward and backward reference frames; and

selecting one of the motion vectors to blocks of the forward reference frame and a corresponding one of the mirror motion vectors to blocks of the backward reference frame based on a measure of visual similarity between the respective blocks of the forward and backward reference frames.

5. The machine-implemented method of claim 3, wherein the deriving a motion vector for the first block includes:

predicting a bi-directional motion vector for the first block based on a combination of the selected motion vector to a block of the forward reference frame, the selected mirror motion vector to a block of the backward reference frame, and the distances between the first frame and each of the forward and backward reference frames.

6. The machine-implemented method of claim 1, wherein the second frame is a first forward reference frame relative to the first frame and the third frame is a second forward frame relative to the first frame, and wherein the selecting includes:

identifying motion vectors to blocks of the first forward reference frame based on the search window;

computing projective motion vectors to blocks of the second forward reference frame based on respective ones of the motion vectors to blocks of the first forward reference frame and the distances between the first frame and each of the first and second forward reference frames; and

selecting one of the motion vectors to blocks of the first forward reference frame and a corresponding one of the projective motion vectors to blocks of the second forward reference frame based on a measure of visual similarity between the respective blocks of the first and second forward reference frames.

7. The machine-implemented method of claim 6, wherein the deriving a motion vector for the first block includes:

predicting a forward motion vector for the first block based on a combination of the selected motion vector to a block of the first forward reference frame, the selected projective motion vector to a block of the second forward reference frame, and the distances between the first frame and each of the first and second forward reference frames.

8. A video decoding apparatus, comprising:

a memory; and

a processor coupled to the memory, the processor to;

define a motion estimation search window for a first coding block of a first frame of a video;

select a motion vector to a block of a second frame of the video and a motion vector to a block of a third frame of the video based on the search window, distances between the first frame and each of the second and third frames, and a measure of visual similarity between the respective blocks of the second and third frames;

derive a motion vector for the first block based on the selected motion vector to a block of the second frame and the selected motion vector to a block of the third frame; and

predict the first block based on the stored motion vector and the block of the second or third frame.

9. The apparatus of claim 8, wherein the processor is further to:

identify motion vectors to blocks of the second frame based on the search window;

compute motion vectors to blocks of the third frame based on respective ones of the motion vectors to blocks of the second frame and the distances between the first frame and each of the second and third frames; and

select one of the motion vectors to blocks of the second frame and a corresponding one of the motion vectors to blocks of third frame based on a measure of visual similarity between the respective blocks of the second and third frames.

10. The apparatus of claim 8, wherein the processor is further to:

derive the vector for the first block based on a combination of the selected motion vector to a block of the second frame, the selected motion vector to a block of the third frame, and the distances between the first frame and each of the second and third frames.

11. The apparatus of claim 8, wherein the second frame is a forward reference frame relative to the first frame and the third frame is a backward frame relative to the first frame, and wherein the processor is further to:

identify motion vectors to blocks of the forward reference frame based on the search window;

compute mirror motion vectors to blocks of the backward reference frame based on respective ones of the motion vectors to blocks of the forward reference frame and the distances between the first frame and each of the forward and backward reference frames; and

select one of the motion vectors to blocks of the forward reference frame and a corresponding one of the mirror motion vectors to blocks of the backward reference frame based on a measure of visual similarity between the respective blocks of the forward and backward reference frames.

12. The apparatus of claim 11, processor is further to:

predict a bi-directional motion vector for the first block based on a combination of the selected motion vector to a block of the forward reference frame, the selected mirror motion vector to a block of the backward reference frame, and the distances between the first frame and each of the forward and backward reference frames.

13. The apparatus of claim 8, wherein the second frame is a first forward reference frame relative to the first frame and the third frame is a second forward frame relative to the first frame, and wherein the processor is further to:

identify motion vectors to blocks of the first forward reference frame based on the search window;

compute projective motion vectors to blocks of the second forward reference frame based on respective ones of the motion vectors to blocks of the first forward reference frame and the distances between the first frame and each of the first and second forward reference frames; and

select one of the motion vectors to blocks of the first forward reference frame and a corresponding one of the projective motion vectors to blocks of the second forward reference frame based on a measure of visual similarity between the respective blocks of the first and second forward reference frames.

14. The apparatus of claim 13, wherein the processor is further to:

predict a forward motion vector for the first block based on a combination of the selected motion vector to a block of the first forward reference frame, the selected projective motion vector to a block of the second forward reference frame, and the distances between the first frame and each of the first and second forward reference frames.

15. A non-transitory computer readable medium encoded with a computer program that includes instructions to cause a video decoding processor to:

define a motion estimation search window for a first coding block of a first frame of a video;

select a motion vector to a block of a second frame of the video and a motion vector to a block of a third frame of the video based on the search window, distances between the first frame and each of the second and third frames, and a measure of visual similarity between the respective blocks of the second and third frames;

derive a motion vector for the first block based on the selected motion vector to a block of the second frame and the selected motion vector to a block of the third frame

store the motion vector for the first block to a memory coupled to the processor; and

predict the first block based on the stored motion vector and the block of the second or third frame.

16. The non-transitory computer readable medium of claim 15, further including instructions to cause the processor to:

identify motion vectors to blocks of the second frame based on the search window;

compute motion vectors to blocks of the third frame based on respective ones of the motion vectors to blocks of the second frame and the distances between the first frame and each of the second and third frames; and

select one of the motion vectors to blocks of the second frame and a corresponding one of the motion vectors to blocks of third frame based on a measure of visual similarity between the respective blocks of the second and third frames.

17. The non-transitory computer readable medium of claim 15, further including instructions to cause the processor to:

derive the vector for the first block based on a combination of the selected motion vector to a block of the second frame, the selected motion vector to a block of the third frame, and the distances between the first frame and each of the second and third frames.

18. The non-transitory computer readable medium of claim 15, wherein the second frame is a forward reference frame relative to the first frame and the third frame is a backward frame relative to the first frame, further including instructions to cause the processor to:

identify motion vectors to blocks of the forward reference frame based on the search window;

compute mirror motion vectors to blocks of the backward reference frame based on respective ones of the motion vectors to blocks of the forward reference frame and the distances between the first frame and each of the forward and backward reference frames; and

select one of the motion vectors to blocks of the forward reference frame and a corresponding one of the mirror motion vectors to blocks of the backward reference frame based on a measure of visual similarity between the respective blocks of the forward and backward reference frames.

19. The non-transitory computer readable medium of claim 18, further including instructions to cause the processor to:

predict a bi-directional motion vector for the first block based on a combination of the selected motion vector to a block of the forward reference frame, the selected mirror motion vector to a block of the backward reference frame, and the distances between the first frame and each of the forward and backward reference frames.

20. The non-transitory computer readable medium of claim 15, wherein the second frame is a first forward reference frame relative to the first frame and the third frame is a second forward frame relative to the first frame, further including instructions to cause the processor to:

identify motion vectors to blocks of the first forward reference frame based on the search window;

compute projective motion vectors to blocks of the second forward reference frame based on respective ones of the motion vectors to blocks of the first forward reference frame and the distances between the first frame and each of the first and second forward reference frames; and

select one of the motion vectors to blocks of the first forward reference frame and a corresponding one of the projective motion vectors to blocks of the second forward reference frame based on a measure of visual similarity between the respective blocks of the first and second forward reference frames.