System and method for video encoding using constructed reference frame

• US 8,385,404 B2
• Filed: 12/05/2008
• Issued: 02/26/2013
• Est. Priority Date: 09/11/2008
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Bankoski, James, Xu, Yaowu, Wilkins, Paul
• Primary Examiner(s): Orgad, Edan
• Assistant Examiner(s): Baum, Ronald
• Application Number:
  US12/329,041
• Publication Number:
  US 20100061461A1
• Time in Patent Office: 1,544 Days
• Field of Search: 375/240
• US Class Current: 375/240
• CPC Class Codes: H04N 19/105 Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/127 Prioritisation of hardware or computational resources
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/179 the unit being a scene or a shot
• CPC Class Codes: H04N 19/23 with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/80 Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence

Abstract

Disclosed herein is a method for digital video encoding prediction comprising creating a constructed reference frame using an encoder and compressing a series of source video frames using the constructed reference frame to obtain a bitstream including a compressed digital video signal for a subsequent decoding process. The constructed reference frame is omitted from the series of digital video frames during the subsequent viewing process.

Claims

1. A method of encoding a series of source video frames to be transmitted to a decoder to produce a series of decoded video frames for viewing, the method comprising:

creating a constructed reference frame using an encoder,

wherein the constructed reference frame is a frame in addition to the series of source video frames;

encoding the constructed reference frame

to obtain an encoded constructed reference frame;

compressing the series of source video frames

to obtain a compressed digital video signal,

at least two of the series of source video frames compressed using the constructed reference frame; and

generating a bitstream

including

the encoded constructed reference frame and

the compressed digital video signal

for transmission to the decoder

for a subsequent decoding process

wherein the constructed reference frame is omitted

from the series of decoded video frames

during a subsequent viewing process,

wherein the series of decoded video frames was obtained from the compressed digital video signal.

2. The method according to claim 1 wherein

creating a constructed reference frame using the encoder comprises encoding a copy of an existing source frame into the bitstream before an appearance of the existing source frame in its order in the series;

and wherein

compressing the series of source video frames includes encoding the existing source frame in its order in the series using the copy of the existing source frame.

3. The method according to claim 1 wherein

encoding the copy of the existing source frame comprises encoding the copy at a first quality level;

and wherein

encoding the existing source frame in its order in the series includes encoding the existing frame at a second quality level different from the first quality level.

4. The method according to claim 1 wherein creating a constructed reference frame using the encoder comprises:

using temporal filtering to remove video noise from several source frames;

calculating an error score for each macroblock of the several source frames; and

copying a macroblock of each source frame having the lowest error score into the constructed reference frame.

5. The method according to claim 1 wherein creating a constructed reference frame using the encoder comprises:

creating the constructed reference using a background of at least one source frame of the series.

6. The method according to claim 5 wherein creating the constructed reference frame using a background comprises:

calculating a motion vector of each of a plurality of macroblocks over several source frames of the series, wherein each of the plurality of macroblocks includes a plurality of pixels;

copying a macroblock having a motion vector below a predetermined limit into the constructed reference frame.

7. The method according to claim 1 wherein the series of source video frames has a predetermined size;

and wherein the constructed reference frame has a size larger than the predetermined size developed using image super resolution.

8. The method according to claim 7 wherein compressing the series of source video frames comprises providing high quality border regions to the at least two of the series of source video frames using the constructed reference frame.

9. The method according to claim 7 wherein creating the constructed reference frame using the encoder comprises:

obtaining a starting source frame N;

finding a global motion vector that matches each source frame from the starting source frame N to a source frame N+X wherein X is a positive integer;

creating a bounding region that covers a set of frames from the starting source frame N to the source frame N+X when aligned on top of each other by use of the global motion vector;

creating a new image that is large enough to cover the entire bounding region as it is moved about;

selecting a plurality of candidate pixels from a sequence of frames for each of the plurality of pixels in the new image, each of the plurality of candidate pixels associated with a respective candidate set;

calculating an average intensity value for each respective candidate set; and

storing the average intensity value with the associated pixel of the plurality of pixels in the new image as a pixel in the constructed reference frame.

10. The method according to claim 9 wherein finding the global motion vector comprises using at least one of a full search global motion vector search method and a diamond global motion vector search method.

11. The method according to claim 1, further comprising:

scoring each macroblock within several source frames of the series based on a scoring criterion, wherein each macroblock includes a plurality of pixels;

computing an overall score for each of the several source frames; and

selecting one of the source frames for creating the constructed reference frame based on the overall scores.

12. The method according to claim 11 wherein the scoring criterion comprises a ratio comparing intra-prediction to inter-prediction.

13. The method according to claim 11 wherein the scoring criterion comprises a motion vector.

14. The method according to claim 11 wherein the scoring criterion comprises a decay in a prediction quality.

15. The method according to claim 11 wherein the scoring criterion comprises a conclusion of a zoom in or a zoom out.

16. A method of encoding a series of source video frames to be transmitted to a decoder to produce a series of decoded video frames for viewing, the method comprising:

A) encoding each block of a frame of the series of source video frames using an intra mode prediction;

B) encoding each block of the frame of the series of source video frames using an inter mode prediction using a motion vector and a reconstruction buffer of a previous frame in the series of source video frames;

C) calculating a first reconstructed error score of each block of the frame encoded using the intra mode prediction;

D) calculating a second reconstructed error score of each block of the frame encoded using the inter mode prediction;

E) calculating a cumulative best score of the frame by summing a lower of either the first reconstructed error score or the second reconstructed error score of each block of the frame;

F) summing the first reconstructed error score of each block of the frame to create a cumulative intra error score;

G) calculating a percentage of blocks of the frame where the second reconstructed error score is less than the first reconstructed error score;

H) calculating a motion vector for each block of the frame;

I) calculating an average motion vector of blocks of the frame using the calculated motion vectors;

J) performing steps A) through I) for each frame of the series of source video frames;

K) creating a first constructed reference frame using an encoder wherein the constructed reference frame is a frame in addition to the series of source video frames;

L) defining a minimum number and a maximum number of frames in the series of source video frames that can be encoded using a single constructed reference frame;

M) selecting a frame N of the series of source video frames;

N) comparing the percentage of blocks of a next frame N+1 subsequent to the frame N where the second reconstructed error score is less than the first reconstructed error score to a value based on the average motion vector of blocks of the frame N;

O) incrementing a boost score for the first constructed reference frame using the cumulative intra error score, and a cumulative inter error score and a lesser of the percentage of blocks of the next frame N+1 where the second reconstructed error score is less than the first reconstructed error score and the value based on the average motion vector of blocks of the frame N;

P) performing steps N) and O) for at least one additional frame in the series of source video frames that is subsequent to the frame N in place of the frame N;

Q) calculating a bit allocation for the first constructed reference frame using the boost score;

R) encoding the first constructed reference frame into a bitstream for transmission to the decoder using the bit allocation for the first constructed reference frame;

S) compressing at least the minimum number of frames and up to the maximum number of frames in the series of source video frames starting at the frame N using the first constructed reference frame to obtain a compressed digital video signal for inclusion in the bitstream for a subsequent decoding process wherein the first constructed reference frame is omitted from the series of decoded video frames during a subsequent viewing process, the series of decoded video frames obtained from the compressed digital video signal.

17. The method according to claim 16, further comprising:

T) creating a second constructed reference frame using the encoder wherein the second constructed reference frame is a frame in addition to the series of source video frames;

U) performing steps M) through Q) with the second constructed reference frame instead of the first constructed reference frame and starting with a first frame after a last frame in the series of source video frames compressed using the first constructed reference frame as the frame N;

V) encoding the second constructed reference frame into the bitstream for transmission to the decoder using the bit allocation for the second constructed reference frame, the encoded second constructed reference frame located in the bitstream after the last frame in the series of source video frames compressed using the first constructed reference frame;

W) using the second constructed reference frame to compress up to the maximum number of frames in the series of source video frames starting with the first frame after the last frame in the series of source video frames compressed using the first constructed reference frame; and

X) including those of the series of source video frames compressed using the second constructed reference in the bitstream after the second constructed reference frame as part of the compressed digital video signal for the subsequent decoding process wherein the second constructed reference frame is also omitted from the series of decoded video frames during the subsequent viewing process.

18. The method according to claim 16 wherein step P) comprises performing steps N) and O) for each frame subsequent to the frame N in the series of source video frames up to a final frame no higher than the maximum number of frames and wherein step S) comprises compressing up through the final frame and starting at the frame N using the first constructed reference frame unless at least one of the following conditions is met:

a first accumulated value indicating randomness of movement in each frame up to the final frame exceeds a first predetermined value;

a second accumulated value indicating a balance of motion vectors pointing away from a center of each frame compared to those pointing towards the center of each frame up to the final frame exceeds a second predetermined value; and

the boost score obtained as a result of performing step O) using the final frame as the frame N is less than a third predetermined value plus the boost score obtained as a result of performing step O) using a frame immediately preceding the final frame as the frame N.

19. The method according to claim 18, further comprising:

dividing an average absolute motion vector by the average motion vector of blocks of each frame up to the final frame calculated in step I); and

calculating the first accumulated value by summing results of the dividing step.

20. A method of viewing a series of decoded video frames from a compressed digital video signal obtained by compressing a series of source video frames, the method comprising:

decoding a first constructed reference frame from a bitstream including the compressed digital video signal wherein the first constructed reference frame is a frame in addition to the series of source video frames;

using the first constructed reference frame to decode a first set of at least two sequential frames of the series of decoded video frames from a portion of the compressed digital video signal in the bitstream;

decoding a second constructed reference frame present in the bitstream after the portion of the compressed digital video signal wherein the second constructed reference frame is a frame in addition to the series of source video frames;

using the second constructed reference frame to decode a second set of at least two sequential frames of the series of decoded video frames from another portion of the compressed digital video signal in the bitstream; and

arranging the first set and the second set in sequential order for viewing, wherein the first constructed reference frame and the second constructed reference frame are omitted from the sequential order.