Segmentation-based parameterized motion models

• US 10,506,249 B2
• Filed: 12/12/2017
• Issued: 12/10/2019
• Est. Priority Date: 03/15/2017
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Mukherjee, Debargha, Liu, Yuxin, Parker, Sarah
• Primary Examiner(s): Suh, Joseph
• Application Number:
  US15/838,748
• Publication Number:
  US 20180270497A1
• Time in Patent Office: 728 Days
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 19/517 by encoding
• CPC Class Codes: H04N 19/521 for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors
• CPC Class Codes: H04N 19/54 using feature points or meshes
• CPC Class Codes: H04N 19/543 using regions
• CPC Class Codes: H04N 19/547 Motion estimation performed in a transform domain
• CPC Class Codes: H04N 19/557 Motion estimation characterised by stopping computation or iteration based on certain criteria, e.g. error magnitude being too large or early exit
• CPC Class Codes: H04N 19/80 Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence

Abstract

A method includes segmenting the video frame with respect to a reference frame resulting in a segmentation, determining a first motion vector for the current block based on the segmentation, determining a second motion vector for the current block using translational motion compensation, and encoding, for the current block, the one of the first motion vector and the second motion vector corresponding to a smaller error. The segmentation includes a segment containing the current block and a parameterized motion model for the segment. Another method includes identifying a parameterized motion model corresponding to a motion model type, associating the parameterized motion model with a segment of a reference frame, and, in response to determining that the current block is encoded using the parameterized motion model, decoding the current block using the parameterized motion model.

Claims

1. A method of encoding a current block of a video frame, comprising:

partitioning the video frame into blocks, wherein the blocks comprise the current block;

for each of a plurality of reference frames, performing steps comprising;

segmenting, based on motion model types each having a respective complexity, the video frame with respect to the each reference frame to obtain a reference-frame-based segmentation,

wherein the reference-frame-based segmentation comprises segments,

wherein each segment of the segments comprises a respective subset of the blocks of the video frame, and

wherein a segment of the segments comprises the current block and a respective parameterized motion model for the segment;

selecting, for the segment comprising the current block, a least complex motion model type that has a value of an error metric below a threshold, wherein the error metric for a parameterized motion model of the respective parameterized motion models is;

E=Σ

α

|c_xy−

w_xy|^0.6

where α

is a weight value, c_xy is a first pixel at (x, y) in the video frame, and w_xy is a second pixel at (x, y) in a warped frame, the warped frame corresponding to the video frame warped according to the parameterized motion model; and

selecting, for the segment, a selected parameterized motion model corresponding to the least complex motion model type;

determining a first motion vector for the current block based on the selected parameterized motion model;

determining a second motion vector for the current block using translational motion compensation; and

encoding, for the current block, the one of the first motion vector and the second motion vector corresponding to a smaller error.

2. The method of claim 1, wherein the segmenting the video frame with respect to the each reference frame to obtain the reference-frame-based segmentation comprises:

determining, using a motion model type, parameters of the respective parameterized motion model.

3. The method of claim 2, wherein the motion model type is selected from a set comprising a homographic motion model type, an affine motion model type, a similarity motion model type, and translational motion model type.

4. The method of claim 2, wherein the encoding the one of the first motion vector and the second motion vector comprises:

encoding the parameters corresponding to the least complex motion model type.

5. The method of claim 1, wherein the encoding the one of the first motion vector and the second motion vector comprises:

encoding a motion model type corresponding to the parameterized motion model.

6. An apparatus configured to encode a current block of a video frame, the apparatus comprising:

a processor configured to execute instructions stored in a non-transitory storage medium to;

segment the video frame with respect to a reference frame resulting in a segmentation, the segmentation comprising a segment containing the current block and a parameterized motion model for the segment, wherein to segment the video frame with respect to the reference frame comprises to;

segment the video frame with respect to the reference frame based on one or more of a plurality of motion model types to provide respective segmentations, each of the plurality of motion model types having a respective complexity;

calculate a global motion error for each segmentation; and

select the segmentation corresponding to a least complex motion model type corresponding to an error metric being below a threshold, wherein the error metric is;

E=Σ

α

|c_xy−

w_xy|^β

where α

is a weight value, β

is a value that is less than 1, c_xy is a first pixel at (x, y) in the video frame, and w_xy is a second pixel at (x, y) in a warped frame, the warped frame corresponding to the video frame warped according to the parameterized motion model;

determine a first motion vector for the current block based on the segmentation;

determine a second motion vector for the current block using translational motion compensation; and

encode, for the current block, the one of the first motion vector and the second motion vector corresponding to a smaller error.

7. The apparatus of claim 6, wherein the instructions to segment the video frame with respect to a reference frame resulting in a segmentation include instructions executable by the processor to:

determine, using a motion model type, parameters of the parameterized motion model.

8. The apparatus of claim 7, wherein the motion model type is selected from a set comprising a homographic motion model type, an affine motion model type, a similarity motion model type, and translational motion model type.

9. The apparatus of claim 7, wherein the instructions to encode the one of the first motion vector and the second motion vector include instructions executable by the processor to:

encode the parameters of the parameterized motion model.

10. A method of encoding a current block of a video frame, comprising:

segmenting the video frame with respect to a reference frame resulting in a segmentation, the segmentation comprising a segment containing the current block;

for each of a plurality of motion model types, determining a parameterized motion model for the segment, each of the plurality of the motion model types having a respective complexity;

for each of the determined parameterized motion models, determining a value of an error metric;

selecting the determined parameterized motion model corresponding to a least complex motion model type that has a respective value of the error metric below a threshold;

determining a first motion vector for the current block based on the selected parameterized motion model;

determining a second motion vector for the current block using translational motion compensation; and

encoding, for the current block, the one of the first motion vector and the second motion vector corresponding to a smaller error.

11. The method of claim 10, wherein the error metric for a parameterized motion model is:

E=Σ

α

|c_xy−

w_xy|^0.6

where α

is a weight value, c_xy is a first pixel at (x, y) in the video frame, and w_xy is a second pixel at (x, y) in a warped frame, the warped frame corresponding to the video frame warped according to that parameterized motion model.

12. The method of claim 11, wherein the smaller error is the error corresponding to a best rate-distortion value.