Training end-to-end video processes

• US 10,681,361 B2
• Filed: 12/27/2017
• Issued: 06/09/2020
• Est. Priority Date: 02/23/2016
• Status: Active Grant

• Current Assignee: Magic Pony Technology Limited (X Holdings Corp.)
• Original Assignee: Magic Pony Technology Limited (X Holdings Corp.)
• Inventors: Wang, Zehan, Bishop, Robert David, Huszar, Ferenc, Theis, Lucas
• Primary Examiner(s): Duley, Janese
• Application Number:
  US15/855,518
• Publication Number:
  US 20180139458A1
• Time in Patent Office: 895 Days
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 19/86 involving reduction of coding artifacts, e.g. of blockiness

Abstract

Methods and systems for optimising the quality of visual data. Specifically, methods and systems for preserving visual information during compression and decompression. An example method for optimising visual data includes using a pre-processing hierarchical algorithm to optimise visual data prior to encoding the visual data in visual data processing; and using a post-processing hierarchical algorithm to enhance visual data following decoding visual data in visual data processing.

Claims

1. A method of encoding and decoding pixel-based data, the method comprising:

using a pre-processing neural network on input pixel-based visual data to produce optimized pixel-based visual data;

encoding the optimized pixel-based visual data using an encoder to produce encoded pixel-based visual data;

providing the encoded pixel-based visual data and a reference to a post-processing neural network to a decoder;

decoding the encoded pixel-based visual data using the decoder to produce decoded pixel-based visual data; and

using the post-processing neural network corresponding to the reference to enhance the decoded pixel-based visual data following decoding visual data in visual data processing,

wherein the reference identifies a post-processing neural network in a library of neural networks or identifies parameters for a post-processing neural network transmitted with the encoded pixel-based visual data.

2. The method of claim 1, wherein the pre-processing neural network and the post-processing neural network have been jointly trained to recreate pixel-based visual data being encoded and decoded.

3. The method of claim 1, wherein the pre-processing neural network and the post-processing neural network have been jointly trained using a differential approximation based on a codec used by the encoder and by the decoder.

4. The method of claim 1, wherein the pre-processing neural network and the post-processing neural network are jointly trained as a single neural network including a pre-encoder layer;

a codec layer; and

a post-decoder layer.

5. The method of claim 1, where the input pixel-based visual data comprises two or more frames of pixel-based visual data.

6. The method of claim 5, wherein the reference to the post-processing neural network differs at least for a first two or more frames of the input pixel-based visual data and a second two or more frames of the input pixel-based visual data.

7. The method of claim 1, further comprising adding the post-processing neural network to the library responsive to the post-processing neural network being transmitted with the encoded pixel-based visual data.

8. The method of claim 1, wherein standardised features of the input pixel-based visual data are extracted and used to select the reference to the post-processing neural network.

9. The method of claim 1, wherein the reference to the post-processing neural network is based on generating the highest quality version of the input pixel-based visual data, wherein quality can be defined by any of:

an error rate;

a bit error rate;

a peak signal-to-noise ratio;

or a structural similarity index.

10. The method of claim 1, wherein the input pixel-based visual data is a portion of each frame in a sequence of frames.

11. The method of claim 1, wherein the post-processing neural network uses a learned filter in enhancing the decoded pixel-based visual data.

12. The method of claim 1, wherein the optimized pixel-based visual data is a lower resolution than the input pixel-based visual data.

13. The method of claim 1, wherein the enhanced decoded pixel-based visual data contains a reduced number of artefacts than the decoded pixel-based visual data.

14. The method of claim 1, wherein the post-processing neural network uses super-resolution techniques to generate the enhanced decoded pixel-based visual data.

15. The method of claim 1, wherein the pre-processing neural network uses a spatio-temporal approach to generate the optimized pixel-based visual data.

16. The method of claim 1, wherein the enhanced decoded pixel-based visual data is a higher resolution than the decoded pixel-based visual data.

17. A computing device comprising:

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the computing device to perform operations including;

using a pre-processing neural network on input pixel-based visual data to produce optimized visual data,

encoding the optimized visual data using an encoder to produce encoded pixel-based visual data,

providing the encoded pixel-based visual data and a reference to a post-processing neural network to a decoder,

decoding the encoded pixel-based visual data using the decoder to produce decoded pixel-based visual data, and

using the post-processing neural network corresponding to the reference to enhance the decoded pixel-based visual data following decoding visual data in visual data processing,

wherein the reference identifies a post-processing neural network in a library of neural networks or identifies parameters for a post-processing neural network transmitted with the encoded pixel-based visual data.

19. The computing device of claim 17, wherein reference to the post-processing neural network differs at least for a first two or more frames of the input pixel-based visual data and a second two or more frames of the input pixel-based visual data.

20. The computing device of claim 17, wherein standardised features of the decoded pixel-based visual data are extracted and used to select the post-processing neural network from a library of networks.

18. A non-transitory computer program product comprising instructions that, when executed by a processor, cause a computing device to perform operations including:

using a pre-processing neural network on input pixel-based visual data to produce optimized pixel-based visual data;

encoding the optimized pixel-based visual data using an encoder to produce encoded pixel-based visual data;

providing the encoded pixel-based visual data and a reference to a post-processing neural network to a decoder;

decoding the encoded pixel-based visual data using the decoder to produce decoded pixel-based visual data; and

using the post-processing neural network corresponding to the reference to enhance the decoded pixel-based visual data following decoding visual data in visual data processing,

wherein the reference identifies a post-processing neural network in a library of neural networks or identifies parameters for a post-processing neural network transmitted with the encoded pixel-based visual data.