Encoding and decoding motion via image segmentation

• US 9,154,799 B2
• Filed: 04/07/2011
• Issued: 10/06/2015
• Est. Priority Date: 04/07/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Bankoski, James, Wilkins, Paul, Xu, Yaowu
• Primary Examiner(s): DIEP, NHON THANH
• Application Number:
  US13/081,662
• Publication Number:
  US 20120257677A1
• Time in Patent Office: 1,643 Days
• US Class Current: 1/1
• CPC Class Codes: H04N 19/105 Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation

Abstract

A system and method for encoding and decoding a video image having a plurality of frames is disclosed. The method includes a previously reconstructed frame that is segmented to create an alternate reference frame used to encode a current frame. The segments of the previously reconstructed frame are assigned motion parameters, which are used to compile the alternate reference frame. The current frame is encoded and the segmentation parameters and motion parameters are transmitted in the bitstream. The segmentation parameters and motion parameters can be used by a decoder to recreate the alternate reference frame in order to decode the encoded current frame.

Claims

1. A method for encoding a current frame of a video signal having a plurality of frames, each frame having a plurality of pixels, the method comprising:

segmenting a previously reconstructed frame into a plurality of image segments using a segmentation parameter, at least one image segment from the plurality of image segments having a segment identifier;

determining, using a processor, a motion parameter for each respective image segment from the plurality of image segments;

creating an alternate reference frame from the previously reconstructed frame using the motion parameter for each respective image segment from the plurality of image segments; and

encoding the current frame using the alternate reference frame for motion prediction.

2. The method of claim 1 wherein the previously reconstructed frame is one of a golden frame, an original alternate reference frame, or a last frame to be encoded.

3. The method of claim 1, further comprising:

identifying an association between the current frame and the previously reconstructed frame;

identifying an association between the current frame and the motion parameter for each respective image segment from the plurality of image segments; and

identifying an association between the current frame and the segmentation parameter.

4. The method of claim 3, further comprising:

including at least one of the identified associations in the encoded output.

5. The method of claim 1 wherein segmenting the previously reconstructed frame comprises using at least one of color segmentation or motion segmentation, and the segmentation parameter comprises at least one of a color or a motion vector for each pixel of the previously reconstructed frame.

6. The method of claim 1 wherein segmenting the previously reconstructed frame comprises performing at least one of k-means clustering, binary tree deconstruction, quad-tree deconstruction, or oct-tree deconstruction.

7. The method of claim 1 wherein determining the motion parameter for each respective image segment from the plurality of image segments comprises:

performing a motion vector search on pixels making up each respective image segment from the plurality of image segments; and

selecting a motion vector based on the motion vector search for each respective image segment from the plurality of image segments as the motion parameter.

8. The method of claim 1 wherein creating the alternate reference frame comprises:

calculating a new position for each pixel of each respective image segment from the plurality of image segments using the motion parameter of each respective image segment from the plurality of image segments;

comparing the new position to a range of positions of the alternate reference frame; and

on a condition that the new position is within the range of positions of the alternate reference frame, copying a value associated with the pixel in the new position in the alternate reference frame.

9. The method of claim 1 wherein creating the alternate reference frame comprises:

for each image segment in the plurality of image segments;

calculating a new position for each pixel of the image segment using the motion parameter of the image segment;

comparing the new position to a range of positions of the alternate reference frame; and

on a condition that the new position is within the range of positions of the alternate reference frame, copying a value associated with the pixel in the new position in the alternate reference frame.

10. The method of claim 9 wherein creating the alternate reference frame comprises:

selecting an unfilled pixel of the alternate reference frame that has no value copied to it after values for all pixels of the previously reconstructed frame having the new position within the range of positions of the alternate reference frame are copied into the new position in the alternate reference frame;

determining a desired value for the unfilled pixel; and

copying the desired value to the unfilled pixel.

11. The method of claim 10 wherein determining the desired value comprises selecting one of a value of a nearby filled pixel, an average of more than one nearby filled pixel, or a value of the pixel at the same location in the current frame to be encoded.

12. The method according to claim 9, further comprising:

initializing values for each pixel in the alternate reference frame to zero and labeling each pixel in the new frame as unfilled before comparing the new position to the range of positions of the alternate reference frame; and

on a condition that the new position is within the range of positions of the alternate reference frame, changing a label for the pixel to filled.

13. The method of claim 1 wherein the motion parameter for each respective image segment from the plurality of image segments is at least one of a motion vector or a parameter reflecting at least one of a zoom, a rotation, or a shear of the image segment.

14. The method of claim 1 wherein creating the alternate reference frame from the previously reconstructed frame includes omitting an image segment and a respective motion parameter, wherein the image segment includes fewer than a predetermined number of pixels.

15. The method of claim 1, wherein encoding the current frame includes using the alternate reference frame for motion prediction in response to determining whether to use the alternate reference frame or the previously reconstructed frame for motion prediction.

16. The method of claim 1, wherein the alternate reference frame is not a reconstructed frame.