METHOD OF ENCODING/DECODING OF MULTIVIEW VIDEO SEQUENCE BASED ON ADAPTIVE COMPENSATION OF LOCAL ILLUMINATION MISMATCHES AT INTERFRAME PREDICTION

• US 20140177712A1
• Filed: 12/24/2013
• Published: 06/26/2014
• Est. Priority Date: 12/26/2012
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: FARTUKOV, Alexey Mikhailovich, Oh, Byung Tae, Kovliga, Igor Mironovich, Mishourovsky, Michael Naumovich
• Granted Patent: US 9,615,089 B2
• Time in Patent Office: Days
• US Class Current: 375/240.12
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/196 being specially adapted for the computation of encoding parameters, e.g. by averaging previously computed encoding parameters processing of motion vectors H04N19/513
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/86 involving reduction of coding artifacts, e.g. of blockiness

Abstract

A technology for processing digital signals is provided, and more particularly, a method of encoding or decoding a multiview video sequence based on adaptive compensation of local illumination mismatches between frames of the multiview video sequence is provided. The method may be performed using a correction parameter used to correct a change in illumination. The correction parameter may be computed based on at least one of a first estimate estD_i,j, a second estimate estR_i,j, a value R_i,j of a pixel of a reference block, and values T^D_k and T^R_k of restored pixels.

Claims

1. A method of adaptively compensating for a local illumination mismatch in inter-frame prediction during encoding and decoding of a multiview video sequence, the method being performed by a computer system, and comprising:

obtaining a value of a pixel of a current encoded block included in an encoded frame, and a value of a pixel of a reference block included in a reference frame;

obtaining values of pixels restored by encoding and then decoding, from a vicinity of the current encoded block, and from a vicinity of the reference block;

excluding from consideration a pixel included in the vicinity of the current encoded block and a pixel included in the vicinity of the reference block, the pixels being estimated to be unreliable for computation of a correction parameter used to correct a change in illumination by at least one preset criterion;

determining a numerical ratio between pixels of the reference block, a numerical ratio between reliable pixels located in the vicinity of the current encoded block, and a numerical ratio between reliable pixels located in the vicinity of the reference block;

computing the correction parameter, based on the determined numerical ratios considering an initial value; and

estimating a change in values of pixels of the reference block,

wherein, when the change is estimated to be essential, the values of the pixels of the reference block are corrected using the computed correction parameter.

2. The method of claim 1, wherein a procedure comprising the determining of the numerical ratios, the computing of the correction parameter, and the estimating of the change comprises:

detecting an adjacent vicinity of the current encoded block, and an adjacent vicinity of the reference block so that pixels included in the vicinities are already restored;

computing at least one of a first estimate estD_i,j and a second estimate estR_i,j for each of positions (i, j) of pixels in the reference block, based on a value T^D_k of a restored pixel obtained from a specified adjacent vicinity of the current encoded frame, and a value T^R_k of a pixel obtained from a predefined adjacent vicinity of the reference block, k having a value of “

0”

to “

N−

1,”

N indicating a number of pixels obtained from the specified adjacent vicinity of the current encoded block, and the number of the pixels being equal to a number of pixels obtained from a specified adjacent vicinity of the reference block;

computing the correction parameter for each of the positions of the pixels in the reference block, based on at least one of the first estimate estD_i,j, the second estimate estR_i,j, a value R_i,j of a pixel of the reference block, the value T^D_k of the restored pixel, and the value T^R_k of the pixel; and

correcting the values of the pixels of the reference block, using the correction parameter.

3. The method of claim 2, wherein a procedure comprising the computing of the at least one of the first estimate and the second estimate, and the computing of the correction parameter comprises:

setting an initial value BaseD for the first estimate, and an initial value BaseR for the second estimate so that both the initial values BaseD and BaseR are nonnegative or nonpositive numbers;

computing a weight coefficient W_k(i,j) based on an intensity of a pixel of the reference block in a position (i,j), and an intensity of a pixel with an index k from a preset adjacent vicinity of the reference block;

computing the first estimate based on estD_i,j=BaseD+Σ

<sub>k=0 . . . N−

1</sub>W_k(i,j)×

T_k^D;

computing the second estimate based on estR_i,j=BaseR+Σ

<sub>k=0 . . . N−

1</sub>W_k(i,j)×

T_k^R;

computing the correction parameter to be

4. The method of claim 2, wherein a position of the adjacent vicinity of the current encoded block, and a position of the adjacent vicinity of the reference block are detected adaptively, instead of being initially set.

5. The method of claim 2, wherein the computing of at least one of the first estimate estD_i,j and the second estimate estR_i,j comprises performing preliminary resampling down of a part of pixels located in a preset adjacent vicinity of the current encoded block, and a part of pixels located in the preset adjacent vicinity of the reference block, wherein the part of the pixels located in the preset adjacent vicinity of the current encoded block corresponds to the part of the pixels located in the preset adjacent vicinity of the reference block.

6. The method of claim 2, further comprising:

estimating a possible change of each of the values of the pixels in the reference block, based on a preset numerical criterion, and the first estimate estD_i,j and the second estimate estR_i,j, the change being determined to be essential or unessential, depending on a computed value of numerical criterion.

7. A method of encoding a multiview video sequence based on adaptive compensation of a local illumination mismatch in inter-frame prediction, an encoded frame being represented as a set of non-overlapping areas of pixels used as units of coding, and the method being performed by a computer system and comprising:

selecting a current encoded block belonging to a current coding unit;

determining a reference block in a reference frame, the reference block being used for generation of block-prediction for the current encoded block, and pixels of the reference block being already completely encoded and decoded;

computing a correction parameter used to correct a change in illumination of the reference block;

correcting the illumination, by adjusting values of all pixels of the reference block, based on the determined correction parameter;

generating the block-prediction for the current encoded block, using the reference block with the corrected illumination; and

decoding the current encoded block, through the generated block-prediction,

wherein a procedure comprising the computing of the correction parameter and the correcting of the illumination comprises;

obtaining values of pixels restored by encoding and then decoding, from a vicinity of the current encoded block of the encoded frame, and from a vicinity of the reference block;

excluding from consideration a pixel included in the vicinity of the current encoded block and a pixel included in the vicinity of the reference block, the pixels being estimated to be unreliable for computation of the correction parameter used to correct the change in the illumination by at least one preset criterion;

computing the correction parameter, based on an initial value and using only a reliable pixel; and

estimating a change in values of pixels of the reference block,

wherein, when the change is estimated to be essential, the values of the pixels of the reference block are corrected using the computed correction parameter.

8. The method of claim 7, wherein the selecting of the current encoded block comprises determining a size of the current encoded block, a relative allocation of the vicinity of the current encoded block and the vicinity of the reference block, and a size of each of the vicinity of the current encoded block and the vicinity of the reference block, and

wherein pixels of the current encoded block are used to compute the correction parameter based on a decoding parameter of the current coding unit.

9. The method of claim 8, wherein the size of the current encoded block, the relative allocation of the vicinity of the current encoded block and the vicinity of the reference block, and the size of each of the vicinity of the current encoded block and the vicinity of the reference block are determined based on a type of applied decorrelated conversion, and

wherein the pixels of the reference block are used to compute the correction parameter based on the decoding parameter of the current coding unit.

10. The method of claim 7, wherein the determining of the reference block comprises selecting a single reference frame from among all reference frames, the selected reference frame being used to search for the reference block, by a predetermined rule, and information on the selected reference frame not being encoded.

11. The method of claim 7, wherein the determining of the reference block comprises detecting a motion vector based on a similarity function that additionally considers mismatches in average levels of illumination of the current encoded block and a candidate for the reference block.

12. The method of claim 7, wherein the encoding of the current encoded block comprises:

encoding an additional syntactic element for the current coding unit, to which the current encoded block belongs; and

determining, based on a value of the additional syntactic element, whether a change in a value of a pixel of a reference block of the current coding unit is essential or unessential.

13. The method of claim 7, wherein the encoding of the current encoded block comprises:

encoding a single additional syntactic element for each subblock of the current coding unit, the single additional syntactic element having a unique index of a reference frame in the current coding unit; and

determining, based on a value of the additional syntactic element, whether the change in the values of the pixels of the reference block is essential or unessential.

14. The method of claim 7, wherein the correction parameter is not encoded during the encoding of the current encoded block.

15. A method of decoding a multiview video sequence based on adaptive compensation of a local illumination mismatch in inter-frame prediction, a decoded frame being represented as a set of non-overlapping areas of pixels used as units of coding, and the method being performed by a computer system and comprising:

selecting a current decoded block belonging to a current coding unit;

determining a reference block for the current decoded block, the reference block being included in a reference frame;

computing a correction parameter used to correct a change in illumination of the determined reference block;

correcting the illumination, by adjusting values of all pixels of the reference block, based on the determined correction parameter;

generating a block-prediction for the current decoded block, using the reference block with the corrected illumination; and

decoding the current decoded block through the generated block-prediction,

wherein a procedure comprising the computing of the correction parameter and the correcting of the illumination comprises;

obtaining values of pixels restored by encoding and then decoding, from a vicinity of the current decoded block of the decoded frame, and from a vicinity of the reference block;

excluding from consideration a pixel included in the vicinity of the current decoded block and a pixel included in the vicinity of the reference block, the pixels being estimated to be unreliable for computation of the correction parameter used to correct the change in the illumination by at least one preset criterion;

computing the correction parameter, based on found numerical ratios considering an initial value; and

correcting values of pixels of the reference block, using the computed correction parameter, when a change in the values of the pixels of the reference block is estimated to be essential.

16. The method of claim 15, wherein the selecting of the current decoded block comprises determining a size of the current decoded block, a relative allocation of the vicinity of the current decoded block and the vicinity of the reference block, and a size of each of the vicinity of the current decoded block and the vicinity of the reference block, and

wherein pixels of the current decoded block are used to compute the correction parameter based on a decoding parameter of the current coding unit.

17. The method of claim 16, wherein the size of the current decoded block, the relative allocation of the vicinity of the current decoded block and the vicinity of the reference block, and the size of each of the vicinity of the current decoded block and the vicinity of the reference block are determined based on a type of applied decorrelated conversion, and

wherein the pixels of the reference block are used to compute the correction parameter.

18. The method of claim 15, wherein the selecting of the current decoded block comprises determining the size of the current decoded block based on decoded information in an input bitstream.

19. The method of claim 15, wherein the decoding of the current decoded block comprises:

decoding an additional syntactic element for the current coding unit, to which the current decoded block belongs; and

determining, based on a value of the additional syntactic element, whether a change in a value of a pixel of a reference block of the current coding unit is essential or unessential.

20. The method of claim 15, wherein the decoding of the current decoded block comprises decoding an additional correction parameter used to correct the change in the illumination, with usage of the correction parameter, the additional correction parameter being placed in an input bitstream, and being computed based on the obtained values of pixels.