REAL-TIME VIDEO CODING/DECODING

• US 20090067504A1
• Filed: 09/05/2008
• Published: 03/12/2009
• Est. Priority Date: 09/07/2007
• Status: Active Grant

• Current Assignee: Beamr Imaging Ltd.
• Original Assignee: Vanguard Software Solutions, Inc. (Dolby Laboratories Incorporated)
• Inventors: MARTEMYANOV, ALEXEY, TERTEROV, NIKOLAY, ZHELUDKOV, ALEXANDER
• Granted Patent: US 8,023,562 B2
• Time in Patent Office: Days
• US Class Current: 375/240.160
• CPC Class Codes: H04N 19/42 characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation H04N19/635 takes precedence
• CPC Class Codes: H04N 19/523 with sub-pixel accuracy
• CPC Class Codes: H04N 19/56 Motion estimation with initialisation of the vector search, e.g. estimating a good candidate to initiate a search
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression

Abstract

A video codec having a modular structure for encoding/decoding a digitized sequence of video frames in a multi-core system is described. The video codec comprises a memory unit; a multithreading engine. and a plurality of control and task modules organized in a tree structure, each module corresponding to a coding operation. The modules communicate with each other by control messages and shared memory. The control modules control all coding logic and workflow, and lower level task modules perform tasks and provide calculations upon receiving messages from the control task modules. The multithreading engine maintains context of each task and assigns at least one core to each task for execution. The method of coding/decoding comprises denoising, core motion estimation, distributed motion estimation, weighted texture prediction and error resilient decoding.

Claims

1. A video codec having a modular structure for encoding/decoding a digitized sequence of video frames in a multi-core system, the video codec comprising:

a memory unit;

a multithreading engine;

a plurality of control modules to control all coding logic and workflow; and

a plurality of task modules to execute tasks assigned by the multithreading engine;

wherein the modules are organized in a tree structure, each module corresponding to a coding operation, the modules share the memory and communicate with each other by control messages, and the multithreading engine initializes tasks, maintains context of each task and assigns at least one of the plurality of task modules to each task for execution, sends and handles messages, provides synchronization and communication.

2. The video codec of claim 1, wherein the tree structure is flexible and changeable depending on a number of available modules and tasks.

3. The video codec of claim 1, wherein the control modules control all coding logic and workflow and performance of the task modules.

4. The video codec of claim 3, wherein the task modules include separate modules for long lasting calculations.

5. The video codec having a modular structure according to claim 1, the plurality of task module comprising:

at least one module to provide pre-processing temporal denoising;

at least one module to perform core motion estimation;

at least one module to perform distributed motion estimation;

at least one module to perform weighted prediction; and

at least one module to perform error resilience algorithm.

6. A method of encoding a digitized sequence of video frames in a multi-core system, the method comprising:

performing core motion estimation; and

determining weighted texture prediction use from a difference between normalized density functions for luminance histograms of an original frame and of a reference frame; and

calculating parameters for weighted texture prediction.

7. The method of encoding a digitized sequence of video frames according to claim 6, wherein the core motion estimation comprises:

dividing a current video frame and a reference video frame into blocks of pixels; and

performing core motion estimation for the current frame based on similarity between pixels of a block of the current frame and pixels of blocks of the reference frame by calculating a function of distortion Q(F,R, X, Y).

8. A method of encoding of claim 7, wherein the core motion estimation comprises:

defining in the current frame an initial block B(F, X_INIT, Y_INIT) with coordinates X_INIT, Y_INIT;

selecting in the reference frame a plurality of N candidate blocks with coordinates (X_j,Y_j), where j=0, 1. . . N, using a predetermined criterion;

calculating values of Q(F,R, X_j, Y_j) for all N candidate blocks; and

comparing minimum values of Q(F,R, X_j, Y_j) with predetermined thresholds to find coordinates for motion estimation X_j=X_MIN, Y_j=Y_MIN from the function of distortion having a lowest value;

Q_MIN=Q(F,R, X_MIN, Y_MIN).

9. The method of claim 8, wherein the motion estimation algorithm comprises:

calculating Q(F, R, X_MIN+j*Dx_MIN, Y_MIN+j*Dy_MIN) for increasing j until inequality (I) is true, or a boundary of the frame is reached;

Q(F,R, X_MIN+j*Dx_MIN,Y_MIN+j*Dy_MIN)<

=Q(F,R, X_MIN+(j+1)*Dx_MIN,Y_MIN+(j+1)*Dy_MIN)  

 

(I)

where j=0, 1, 2. . . J, and J—

is minimum nonnegative integer value for which the inequality (I) is true;

Dx and Dy are steps in horizontal and vertical direction;

defining parameters X_MIN=X_MIN+J*Dx_MIN, Y_MIN=Y_MIN+J*Dy_MIN;

calculating Q_MIN=Q(F,R, X_MIN, Y_MIN);

finding one resulting coordinate pair X_MIN, Y_MIN for Q_MIN=Q(F, R, X_MIN, Y_MIN) from the following coordinate pairs;

(X_MIN+Dx_MIN, Y_MIN+Dy_MIN),

(X_MIN−

Dx_MIN, Y_MIN+Dy_MIN),

(X_MIN+Dx_MIN, Y_MIN−

Dy_MIN), and

(X_MIN−

Dx_MIN, Y_MIN−

Dy_MIN).

10. A method of encoding a digitized sequence of video frames in a multi-core system, comprising distributed motion estimation for encoding successive frames using one reference frame and the core motion estimation algorithm of claim 9, wherein the distributed motion estimation comprises:

reducing frame size using reduction coefficient in each spatial dimension;

calculating and storing a plurality of spatially reduced frames;

selecting reduced block size in reduced frames as 8×

8;

calculating and storing motion vector sets for reduced frame pairs using the core motion estimation algorithm;

up-scaling the motion vectors for each block according to the reduction coefficients in each dimension and defining a rough full-pel motion vector for each block of an up-scaled frame;

calculating a full-pel motion vector by applying the core motion estimation algorithm to the rough full-pel motion vector as an additional candidate, and reducing search range according to the reduction coefficients; and

calculating a quarter-pel motion vector for each block by applying the core motion estimation algorithm to the full-pel motion vector as a starting point and a search range of 2.

11. The method of claim 6, wherein the weighted texture prediction comprises:

performing estimate for normalized density functions for histogram H_o of the original frame and histogram H_r of the reference frame;

comparing the difference between the density function for H_o and the density function H_r with a predetermined threshold, and determining that the weighted prediction is to be applied if the difference is less than the threshold;

calculating weighted prediction parameters A and B for at least one block of pixels P(x, y);

A*L(x+mx, y+my)+B;

A=So/Sr; and

B=Mo−

A*Mr;

where L(x, y) is luminance of the reference frame;

(mx, my) is motion vector for the block P(x, y);

So is deviation for the original frame;

Sr is deviation for the reference frame;

Mo is mean value for the original frame; and

Mr is mean value for the reference frame.

12. The method according to claim 6, further comprising a pre-processing temporal denoising, including:

providing motion estimation by creating a texture prediction frame for a current frame from a previous frame and a corresponding motion vector;

performing deblocking of the prediction frame; and

modifying of the current frame based on at least one pixel of the current frame and at least one pixel of the prediction frame;

I=I+sign(I−

P)·

F(abs(I−

P)),

where;

I is the pixel of the current frame,

P is same position pixel of the prediction frame after deblocking,

abs is the absolute value function,

sign is a sign function (sign(x)=−

1, if x<

0, sign(x)=0, if x=0, and sign(x)=1, if x>

0), and

F is a discrete time function.

13. The method of claim 12, wherein the discrete time function has the following properties:

F(D) is sufficiently smooth inside interval [0;

T₂];

F(0)=0;

F(D)≧

D/2 and F(D)≦

D, if D∈

[0;

T₀];

F(D)=D/2, if D∈

[T₀;

T₁); and

F(D)≦

D/2 and F(D)≧

0, if D∈

[T₁;

T₂);

where T₀, T₁ and T₂ are predetermined parameters which control strength of noise suppression, 0≦

T₀<

T₁<

T₂.

14. The method according to claim 13, wherein the de-noising is not performed if a motion vector function mv_cost exceeds a pre-defined threshold.

15. A method of encoding/decoding a digitized sequence of video frames in a multi-core system, the method comprising:

encoding the sequence of video frames by performing core motion estimation and weighted texture prediction; and

decoding encoded sequence of video frames using high motion update and low motion update for error resilience,

wherein the error resilience comprises;

defining each macroblock as a high motion macroblock or a low motion macroblock,

defining for each macroblock of a current frame a set of macroblocks including the macroblock of the current frame and at least one macroblock located at the same position in previous frames; and

making a choice between INTRA and INTER coding mode for each macroblock of high motion update frame and each macroblock of low motion update frame.

16. The method of encoding/decoding of claim 15, wherein the choice between INTRA and INTER coding mode fro error resilience is based on the following:

the INTRA coding mode is used for high motion update frames if cost_intra*K_H<

cost_inter, and the set contains at least one high motion macroblock, and

the INTRA coding mode is used for low motion update frames if cost_intra*K_L<

cost_inter, and C(T_L) contains at least one high or low motion macroblock,

where

K_H—

INTRA cost scaling factor for high motion frames,

K_L—

INTRA cost scaling factor for low motion frames,

T_H—

high motion INTRA update period;

T_L—

low motion INTRA update period;

D—

high motion INTRA update depth.

17. The method of encoding/decoding of claim 16, wherein a macroblock is defined as a high motion macroblock, when

S>

=MVT_H, and

SD>

=MVTD_H; and

a macroblock is defined as a low motion macroblock, when

MVT_L<

=S<

MVT_H, and

MVTD_L<

=SD<

MVTD_H,

where S is a sum of absolute values of motion vector components of the macroblock,

SD is a sum of absolute values of motion vector prediction differences components of the macroblock,

MVT_H is a high motion threshold for the sum of absolute values of the motion vector components, and

MVTD_H is a high motion threshold for the sum of absolute values of the motion vector prediction differences components; and

MVT_L is a low motion threshold for the sum of absolute values of the motion vector components;

MVTD_L is a low motion threshold for the sum of absolute values of the motion vector prediction differences components.