Continuous block tracking for temporal prediction in video encoding

• US 9,621,917 B2
• Filed: 11/04/2014
• Issued: 04/11/2017
• Est. Priority Date: 03/10/2014
• Status: Active Grant

• Current Assignee: Euclid Discoveries LLC
• Original Assignee: Euclid Discoveries LLC
• Inventors: Kottke, Dane P., Guo, John J., Lee, Jeyun, Park, Sangseok, Weed, Christopher, Kwan, Justin, Lee, Nigel
• Primary Examiner(s): Ustaris, Joseph
• Assistant Examiner(s): VOLENTINE, REBECCA A
• Application Number:
  US14/532,947
• Publication Number:
  US 20150256850A1
• Time in Patent Office: 889 Days
• CPC Class Codes: H04N 19/105 Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/517 by encoding
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/53 Multi-resolution motion estimation; Hierarchical motion estimation
• CPC Class Codes: H04N 19/56 Motion estimation with initialisation of the vector search, e.g. estimating a good candidate to initiate a search
• CPC Class Codes: H04N 19/58 Motion compensation with long-term prediction, i.e. the reference frame for a current frame not being the temporally closest one H04N19/23 takes precedence

Abstract

Continuous block tracking (CBT) tracks macroblock locations over reference frames to produce better inter-predictions than conventional block-based motion estimation/compression. CBT includes frame-to-frame tracking, estimating motion from a frame to a previous frame, and continuous tracking, related frame-to-frame motion vectors to block tracks. Frame-to-frame tracking may include block based or hierarchical motion estimations. CBT combined with enhanced predictive zonal search may create unified motion estimation. Accumulated CBT results may form trajectories for trajectory-based CBT predictions. Metrics measuring continuous track and motion vector quality can assess relative priority of CBT prediction against non-tracker-based predictions and to modify encoding choices. Continuous tracks can be analyzed for goodness-of-fit to translational motion models, with outliers removed from encoding. Translational motion models can be extended to entire frames in adaptive picture type selection. Outputs from CBT used in look-ahead processing, via look-ahead tracking, may provide rate control and scene change detection for the current frame being encoded.

Claims

1. A method of encoding video data, the method comprising:

processing a plurality of source video frames having non-overlapping input blocks to be encoded; and

for each input block to be encoded, applying continuous block tracking (CBT) for initial motion estimation within a model-based inter-prediction process to produce CBT motion vector candidates by;

providing frame-to-frame tracking of each input block in a current frame referenced to a source video frame resulting in a set of frame-to-frame motion vectors configured to specify, for each input block, a location of a matching region in a temporally-adjacent source video frame; and

providing continuous tracking over multiple reference frames by relating frame-to-frame motion vectors over the multiple reference frames, the continuous tracking resulting in a set of continuous tracking motion vectors configured to specify, for each input block, a location of a matching region in a temporally non-adjacent source video frame; and

combining the CBT motion vector candidates with enhanced predictive zonal search (EPZS) motion vector candidates to form an aggregate set of initial CBT/EPZS motion vector candidates in a unified motion estimation framework.

2. The method as in claim 1 wherein the CBT motion vector candidates are determined by:

filtering the initial set of CBT/EPZS motion vector candidates separately by an approximate rate-distortion criterion, resulting in a “

best”

CBT candidate and “

best”

EPZS candidate;

performing fine motion estimation on the best CBT and best EPZS candidates; and

selecting the best initial inter-prediction motion vector candidates between the best CBT and the best EPZS motion vector candidates by means of rate-distortion optimization.

3. The method as in claim 1 wherein the combining the CBT motion vector candidates with enhanced predictive zonal search (EPZS) motion vector candidates is done at an earlier stage via approximate rate-distortion optimization, causing the CBT motion vector candidates and EPZS motion vector candidates to be unified, resulting in a single “

best”

CBT/EPZS candidate, and

wherein fine motion estimation is performed on the single best CBT/EPZS candidate.

4. The method as in claim 2 wherein selecting the best inter-prediction motion vector candidates further includes using encoding mode generation and final rate-distortion analysis to determine the best inter-prediction motion vector candidate.

5. The method as in claim 2 wherein filtering the initial set of CBT/EPZS motion vector candidates further includes refining the combined CBT/EPZS motion vector candidates using feedback from rate-distortion scores of neighboring blocks, the feedback determining the number of candidates to pass forward for final rate-distortion optimization.

6. The method as in claim 1 wherein combining the CBT motion vector candidates with the EPZS motion vector candidates further includes supplementing the combined CBT and EPZS motion vector candidates with one or more additional candidates including at least one of:

a random motion vector candidate, a median predictor, and a (0,

     0) motion vector candidate.

7. The method as in claim 6 wherein the random motion vector candidate is configured with fine motion estimation to find the best candidate in its local neighborhood.

8. The method as in claim 6 wherein the (0, 0) motion vector does not undergo fine motion estimation and is explicitly included as a candidate in final rate-distortion optimization.

9. The method as in claim 2 wherein filtering the initial set of CBT/EPZS motion vector candidates further includes removing duplicate motion vector estimates using a proximity criterion.

10. The method as in claim 1 wherein the predictive zonal search (EPZS) motion vector candidates are constructed by:

generating a set of initial motion candidates for the input block, based on (i) motion vectors of neighboring blocks that have already been encoded, and (ii) motion vectors of a co-located block and neighboring blocks in a previous reference frame;

using the EPZS, filtering the initial set of motion candidates to narrow the set; and

performing fine motion estimation on the narrowed set to select the EPZS motion vector candidates.

11. A video compression system for encoding video data comprising:

a video encoder configured to compress video data by;

processing a plurality of source video frames having non-overlapping input blocks to be encoded; and

for each input block to be encoded, applying continuous block tracking (CBT) for initial motion estimation within a model-based inter-prediction process to produce CBT motion vector candidates by;

providing frame-to-frame tracking of each input block in a current frame referenced to a source video frame resulting in a set of frame-to-frame motion vectors configured to specify, for each input block, a location of a matching region in a temporally-adjacent source video frame; and

providing continuous tracking over multiple reference frames by relating frame-to-frame motion vectors over the multiple reference frames, the continuous tracking resulting in a set of continuous tracking motion vectors configured to specify, for each input block, a location of a matching region in a temporally non-adjacent source video frame; and

combining the CBT motion vector candidates with enhanced predictive zonal search (EPZS) motion vector candidates to form an aggregate set of initial CBT/EPZS motion vector candidates in a unified motion estimation framework.

12. The video compression system as in claim 11 wherein the CBT motion vector candidates are determined by:

filtering the initial set of CBT/EPZS motion vector candidates separately by an approximate rate-distortion criterion, resulting in a “

best”

CBT candidate and “

best”

EPZS candidate;

performing fine motion estimation on the best CBT and best EPZS candidates; and

selecting the best initial inter-prediction motion vector candidates between the best CBT and the best EPZS motion vector candidates by means of rate-distortion optimization.

13. The video compression system as in claim 11 wherein the combining the CBT motion vector candidates with enhanced predictive zonal search (EPZS) motion vector candidates is done at an earlier stage via approximate rate-distortion optimization, causing the CBT motion vector candidates and EPZS motion vector candidates to be unified, resulting in a single “

best”

CBT/EPZS candidate, and

wherein fine motion estimation is performed on the single best CBT/EPZS candidate.

14. The video compression system as in claim 12 wherein selecting the best inter-prediction motion vector candidates further includes using encoding mode generation and final rate-distortion analysis to determine the best inter-prediction motion vector candidate.

15. The video compression system as in claim 12 wherein filtering the initial set of CBT/EPZS motion vector candidates further includes refining the combined CBT/EPZS motion vector candidates using feedback from rate-distortion scores of neighboring blocks, the feedback determining the number of candidates to pass forward for final rate-distortion optimization.

16. The video compression system as in claim 11 wherein combining the CBT motion vector candidates with the EPZS motion vector candidates further includes supplementing the combined CBT and EPZS motion vector candidates with one or more additional candidates including at least one of:

a random motion vector candidate, a median predictor, and a (0,

     0) motion vector candidate.

17. The video compression system as in claim 16 wherein the random motion vector candidate is configured with fine motion estimation to find the best candidate in its local neighborhood.

18. The video compression system as in claim 16 wherein the (0, 0) motion vector does not undergo fine motion estimation and is explicitly included as a candidate in final rate-distortion optimization.

19. The video compression system as in claim 12 wherein filtering the initial set of CBT/EPZS motion vector candidates further includes removing duplicate motion vector estimates using a proximity criterion.

20. The video compression system as in claim 11 wherein the predictive zonal search (EPZS) motion vector candidates are constructed by:

generating a set of initial motion candidates for the input block, based on (i) motion vectors of neighboring blocks that have already been encoded, and (ii) motion vectors of a co-located block and neighboring blocks in a previous reference frame;

using the EPZS, filtering the initial set of motion candidates to narrow the set; and

performing fine motion estimation on the narrowed set to select the EPZS motion vector candidates.