Training end-to-end video processes

• US 10,666,962 B2
• Filed: 09/18/2017
• Issued: 05/26/2020
• Est. Priority Date: 03/31/2015
• Status: Active Grant

• Current Assignee: Magic Pony Technology Limited (X Holdings Corp.)
• Original Assignee: Magic Pony Technology Limited (X Holdings Corp.)
• Inventors: Wang, Zehan, Bishop, Robert David, Huszar, Ferenc, Theis, Lucas
• Primary Examiner(s): Duley, Janese
• Application Number:
  US15/707,294
• Publication Number:
  US 20180131953A1
• Time in Patent Office: 981 Days
• CPC Class Codes: G06T 3/4046 using neural networks
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 9/002 using neural networks
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/147 according to rate distortion criteria rate-distortion as a criterion for motion estimation H04N19/567
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/177 the unit being a group of pictures [GOP]
• CPC Class Codes: H04N 19/19 using optimisation based on Lagrange multipliers
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/80 Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression

Abstract

Disclosed is method for training a plurality of visual processing algorithms for processing visual data. The method includes using a pre-processing hierarchical algorithm to process the visual data prior to encoding the visual data in visual data processing, and using a post-processing hierarchical algorithm to further process the visual data following decoding visual data in visual data processing. The encoding and decoding are performed with respect to a predetermined visual data codec and may be content specific.

Claims

1. A method for jointly training a pre-processing neural network and a post-processing neural network for a visual data encoding and decoding process, the method comprising:

determining a differential approximation of the encoding and decoding process;

receiving one or more sections of input visual data;

using a pre-processing neural network to process the input visual data and to output pre-processed visual data;

applying the encoding and decoding process to the pre-processed visual data to generate decoded visual data;

using a post-processing neural network to further process the decoded visual data and to output reconstructed visual data;

comparing the reconstructed visual data with the input visual data using a metric; and

updating parameters of the pre-processing neural network and the post-processing neural network using the differential approximation of the encoding and decoding process and based on the comparing of the reconstructed visual data with the input visual data.

2. The method according to claim 1, wherein updating the parameters of the pre-processing neural network and the post-processing neural network includes optimising the pre-processing neural network and the post-processing neural network with respect to the metric.

3. The method according to claim 1, wherein one or more parameters associated with the pre-processing neural network is stored in a library for re-use in encoding alternative visual data similar to the visual data used for training.

4. The method according to claim 1, further comprising:

transmitting the one or more updated parameters associated with the pre-processing neural network to a device configured to use the pre-processing neural network with the updated parameters in encoding visual data similar to the input visual data.

5. The method according to claim 1, further comprising:

transmitting the one or more updated parameters with the post-processing neural network with processed visual data to a remote device configured to use the post-processing neural network with the updated parameters, wherein the remote device has another pre-processing neural network associated with the post-processing neural network.

6. The method according to claim 1, wherein at least one of:

the pre-processing neural network includes a layer that generalises the visual data processing, and

the post-processing neural network includes a layer that generalises the visual data processing.

7. The method according to claim 1, wherein receiving a plurality of predetermined bit rate for use in training the pre-processing neural network or in training the post-processing neural network.

8. The method according to claim 1, wherein input visual data includes at least one of:

a single frame of visual data,

a sequence of frames of visual data, and

a region within a frame or sequence of frames of visual data.

9. The method according to claim 1, wherein the pre-processing neural network is different for each section of the visual data.

10. The method according to claim 1, wherein the post-processing neural network is developed using a machine learning approach.

11. The method according to claim 1, wherein the post-processing neural network is a non-linear neural network including at least one convolutional neural networks.

12. The method according to claim 1, wherein one of:

the pre-processing neural network is used as a first filter when encoding the visual data, and

the post-processing neural network is used as a second filter when decoding the visual data.

13. The method according to claim 1, wherein the pre-processing neural network uses a spatio-temporal approach.

14. The method according to claim 1, wherein using the pre-processing or post-processing neural network includes at least one of:

training the neural network,

generating the neural network, and

developing the neural network.

15. The method according to claim 2, wherein the pre-processing neural network and the post-processing neural network are optimized based on a tradeoff between compression and reconstruction error.

16. The method according to claim 1, wherein the differential approximation of the encoding and decoding process includes neural network layer.

17. The method according to claim 1, wherein the updating the parameters of the pre-processing neural network and the post-processing neural network include applying a backpropagation algorithm using gradients of the differential approximation to the encoding and decoding process.