GENERATIVE ADVERSARIAL NETWORK BASED INTRA PREDICTION FOR VIDEO CODING

• US 20200137384A1
• Filed: 10/24/2018
• Published: 04/30/2020
• Est. Priority Date: 10/24/2018
• Status: Active Grant

• Current Assignee: City University of Hong Kong
• Original Assignee: City University of Hong Kong
• Inventors: Kwong, Tak Wu Sam, Zhu, Linwel, Zhang, Yun
• Granted Patent: US 11,159,789 B2
• Time in Patent Office: Days
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: H04N 19/11 among a plurality of spatial predictive coding modes
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/593 involving spatial prediction techniques
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards

Abstract

Systems and methods which provide Generative Adversarial Network (GAN) based intra prediction for video coding are described. GAN based intra prediction as implemented according to embodiments may be modeled as an inpainting task. For example, intra prediction may be formulated as a learning based inpainting task, wherein a latent variable is designed to control different generation modes. GAN based intra prediction provided according to embodiments of the invention may be implemented alone or in combination with one or more other video compression technique, such as a direction intra prediction technique. The intra prediction module of such a HEVC encoder/decoder may be redesigned to also apply GAN based inpainting in intra prediction, wherein Rate-Distortion Optimization (RDO) may be performed to select the best intra prediction mode between the intra prediction approaches.

Claims

1. A method for video coding, the method comprising:

providing a Generative Adversarial Network (GAN) based intra prediction engine having a plurality of intra prediction modes configured for GAN based intra prediction using a latent variable; and

generating, by the GAN based intra prediction engine, a plurality of prediction blocks for an input image to be coded using the plurality of intra prediction modes configured for GAN based intra prediction.

2. The method of claim 1, wherein the GAN based intra prediction engine is configured to apply the latent variable to produce versions of intra prediction triodes in addition to and different from intra prediction modes produced by a direction based intra prediction engine implemented in parallel with the GAN based intra prediction engine.

3. The method of claim 1, further comprising:

training the GAN based intra prediction engine using the latent variable to control different generation modes of the plurality of intra prediction modes.

4. The method of claim 3, wherein different versions of the input image are generated for training the GAN based intra prediction engine using the latent variable, wherein differences of the different versions of the input image lie in initial pixel values of a missing part to be predicted.

5. The method of claim 1, wherein the GAN based intra prediction comprises inpainting intra prediction.

6. The method of claim 1, wherein the plurality of intra prediction modes configured for GAN based intra prediction comprise inpainting intra prediction modes configured using the latent variable.

7. The method of claim 6, wherein the GAN based intra prediction engine comprises generation network having a plurality of convolutional layers used for predicting parts of the input image.

8. The method of claim 7, wherein training the CAN based intra prediction engine further comprises:

using a discriminator network having a global discriminator and a local discriminator, wherein the global discriminator and the local discriminator each have a plurality of convolutional layers.

9. The method of claim 1, wherein generating the plurality of predicted blocks for the input image comprises:

partitioning the input image into multiple blocks; and

predicting blocks using GAN based intra prediction.

10. The method of claim 1, wherein the generating a plurality of prediction blocks comprises:

for a particular prediction block, treating three blocks adjacent to the particular prediction block in the input image as GAN based intra prediction inputs; and

inferring the particular prediction block from the three adjacent blocks.

11. The method of claim 10, wherein the three adjacent blocks are above-left, above, and left of the particular prediction block in the input image.

12. The method of claim 10, wherein the generating a plurality of prediction blocks further comprises:

implementing pixelwise multiplication of a first mask and the input image, wherein the pixelwise multiplication provides pixels of a portion of the input image having pixels of the particular prediction block omitted; and

implementing pixelwise multiplication of a second mask and the particular prediction block resulting from the inferring the particular prediction block from the three adjacent blocks, wherein the pixelwise multiplication provides isolated pixels of the particular prediction block.

13. The method of claim 12, wherein the generating a plurality of prediction blocks further comprises:

summing the pixels of the portion of the input image having pixels of the particular prediction block omitted and the isolated pixels of the particular prediction block omitted to provide an output image from the GAN based intra prediction engine.

14. The method of claim 1, further comprising:

providing a direction based intra prediction engine having a plurality of intra prediction modes configured for direction based intra prediction;

generating, by the direction based intra prediction engine, a plurality of prediction blocks for the input image to be coded using the plurality of intra prediction modes configured for direction based intra prediction; and

selecting an intra prediction mode for video coding with respect to the input image based on azo intra prediction mode of the GAN based intra prediction and the direction based intra prediction having a minimum Rate Distortion (RD) cost.

15. A method for video coding, the method comprising:

generating a plurality of prediction blocks for an input image to be coded by inpainting based intra prediction using a plurality of inpainting intra prediction modes;

generating a plurality of prediction blocks for the input image to be coded by direction based intra prediction using a plurality of direction intra prediction modes; and

selecting an intra prediction mode for video coding with respect to the input image based on the intra prediction mode of the inpainting based intra prediction and the direction based intra prediction having a minimum Rate Distortion (RD) cost.

16. The method of claim 15, wherein the plurality of inpainting intra prediction modes are configured for intra prediction using a latent variable.

17. The method of claim 16, wherein generating the plurality of prediction blocks for the input image using the plurality of inpainting intra prediction modes using the latent variable comprises:

initially assigning pixel values in each block to be predicted of the plurality of prediction blocks the same values; and

changing the initially assigned pixel values from 0 to 2^k−

1 for each inpainting intra prediction mode of the plurality of inpainting intra prediction modes, where k is bit depth.

18. The method of claim 15, wherein the inpainting based intra prediction is performed by a Generative Adversarial Network (GAN) based inpainting intra prediction engine.

19. The method of claim 18, further comprising:

training the GAN based inpainting intra prediction engine using a latent variable to control different generation modes of the plurality of inpainting intra prediction modes.

20. The method of claim 19, wherein different versions of the input image are generated for the training using the latent variable, wherein differences of the different versions of the input image lie in initial pixel values of a missing part to be predicted.

21. The method of claim 18, wherein the GAN based intra prediction engine comprises generation network having a plurality of convolutional layers used for predicting parts of the input image.

22. The method of claim 21, wherein training the GAN based intra prediction engine further comprises:

using a discriminator network having a global discriminator and a local discriminator, wherein the global discriminator and the local discriminator each have a plurality of convolutional layers.

23. A system for video coding, the system comprising:

a Generative Adversarial Network (GAN) based inpainting intra prediction engine configured to generate a plurality of prediction blocks for an input image to be coded by inpainting based intra prediction using a plurality of inpainting intra prediction modes;

a direction based intra prediction engine configured to generate a plurality of prediction blocks for the input image to be coded by direction based intra prediction using a plurality of direction intra prediction modes; and

prediction model selection logic configured to select an intra prediction mode for video coding with respect to the input image based on the intra prediction mode of the inpainting based intra prediction and the direction based intra prediction.

24. The system of claim 23, wherein the prediction model selection logic selects the intra prediction mode for video coding from the inpainting based intra prediction and the direction based intra prediction having a minimum Rate Distortion (RD) cost.

25. The system of claim 23, further comprising:

prediction model flag encoding logic configured to signal a selected prediction mode of the inpainting based intra prediction and the direction based intra prediction to a corresponding decoder.