Generative methods of super resolution

• US 10,692,185 B2
• Filed: 12/28/2017
• Issued: 06/23/2020
• Est. Priority Date: 03/18/2016
• Status: Active Grant

• Current Assignee: Magic Pony Technology Limited (X Holdings Corp.)
• Original Assignee: Magic Pony Technology Limited (X Holdings Corp.)
• Inventors: Wang, Zehan, Shi, Wenzhe, Huszar, Ferenc, Bishop, Robert David
• Primary Examiner(s): Allison, Andrae S
• Application Number:
  US15/856,923
• Publication Number:
  US 20180122048A1
• Time in Patent Office: 908 Days
• Field of Search: None
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 3/4046 using neural networks
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 3/4076 using the original low-resolution images to iteratively correct the high-resolution images
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution

Abstract

A method for training an algorithm to process at least a section of received visual data using a training dataset and reference dataset. The method comprises an iterative method with iterations comprising: generating a set of training data using the algorithm; comparing one or more characteristics of the training data to one or more characteristics of at least a section of the reference dataset; and modifying one or more parameters of the algorithm to optimise processed visual data based on the comparison between the characteristic of the training data and the characteristic of the reference dataset. The algorithm may output the processed visual data with the same content as the at least a section of received visual data. Some aspects and/or implementations provide for improved super-resolution of lower quality images to produce super-resolution images with improved characteristics (e.g. less blur, less undesired smoothing) compared to other super-resolution techniques.

Claims

1. A method for training an algorithm to process at least a section of received visual data using a training dataset and a reference dataset, the method being an iterative method with each iteration comprising:

generating a set of training data from the training dataset using the algorithm;

determining one or more characteristics of the training data, wherein the one or more characteristics include a statistical distribution of the training data;

determining one or more characteristics of the reference dataset, wherein the one or more characteristics include a statistical distribution of the reference dataset;

comparing the one or more characteristics of the training data to the one or more characteristics of the reference dataset; and

modifying one or more parameters of the algorithm to optimise processed visual data based on the comparison between the one or more characteristics of the training data and the one or more characteristics of the reference dataset,

wherein the algorithm outputs the processed visual data with the same content as the at least a section of received visual data.

2. The method of claim 1, wherein the training dataset comprises a plurality of visual data.

3. The method of claim 1, wherein the reference dataset comprises a plurality of visual data.

4. The method of claim 1, wherein generating a set of training data using the algorithm uses all the data from the training dataset.

5. The method of claim 1, wherein the received visual data is low-resolution visual data.

6. The method of claim 1, wherein the processed visual data has a higher-resolution than the received visual data.

7. The method of claim 1, wherein the processed visual data is produced by the algorithm being configured to be used for any of:

removing compression artefacts;

dynamic range enhancement;

image generation and synthesis;

image inpainting;

image de-mosaicing; and

denoising.

8. The method of claim 1, wherein comparing the one or more characteristics of the training data to the one or more characteristics of the reference dataset further comprises training a denoising auto-encoder algorithm via a least-squares procedure to capture the one or more characteristics.

9. The method of claim 1, wherein comparing the one or more characteristics of the training data to the one or more characteristics of the reference dataset further comprises an adversarial training procedure.

10. The method of claim 9, wherein the adversarial training procedure employs a binary classifier that discriminates between enhanced visual data and reference data.

11. The method of claim 1, wherein comparing the one or more characteristics of the training data to the one or more characteristics of the reference dataset further comprises using a regularised auto-encoder algorithm to minimise any differences between the one or more characteristics.

12. The method of claim 1, wherein the one or more characteristics of the training data and the one or more characteristics of the reference dataset are representative of the same characteristics.

13. The method of claim 1, wherein the algorithm preserves the content of the visual data.

14. The method of claim 1, wherein the algorithm preserves the content of the visual data by comparing one or more characteristics of an input of the algorithm and one or more characteristics of an output of the algorithm.

15. The method of claim 1, wherein comparing involves assessing the similarity between one or more characteristics of an input of the algorithm and one or more characteristics of an output of the algorithm.

16. The method of claim 15, wherein the one or more characteristics are calculated by a characteristic algorithm.

17. The method of claim 16, wherein the characteristic algorithm is developed using learning approaches, such as sufficient statistics based on activations in a convolutional neural network.

18. The method of claim 1, wherein comparing the one or more characteristics of the training data to the one or more characteristics of the reference dataset is performed without reference to corresponding input and output pairs.

19. An apparatus comprising:

at least one processor; and

at least one memory including computer program code which, when executed by the at least one processor, causes the apparatus to perform an iterative method, wherein to perform an iteration of the iterative method, the computer program code causes the apparatus to;

generate a set of training data using an algorithm for processing at least a section of received visual data using a training dataset and reference dataset;

determine one or more characteristics of the training data, wherein the one or more characteristics include a statistical distribution of the training data;

determine one or more characteristics of the reference dataset, wherein the one or more characteristics include a statistical distribution of the reference dataset;

compare the one or more characteristics of the training data to the one or more characteristics of the reference dataset; and

modify one or more parameters of the algorithm to optimise processed visual data based on the comparison between the one or more characteristics of the training data and the one or more characteristics of the reference dataset,

wherein the algorithm outputs the processed visual data with the same content as the at least a section of received visual data.

20. A non-transitory computer-readable medium having computer-readable code stored thereon, the computer-readable code, when executed by at least one processor, cause the processor to perform an iterative method, wherein to perform an iteration of the iterative method, the instructions cause the processor to:

generate a set of training data using an algorithm for processing at least a section of received visual data using a training dataset and reference dataset;

determine one or more characteristics of the training data, wherein the one or more characteristics include a statistical distribution of the training data;

determine one or more characteristics of the reference dataset, wherein the one or more characteristics include a statistical distribution of the reference dataset;

compare the one or more characteristics of the training data to the one or more characteristics of the reference dataset; and

modify one or more parameters of the algorithm to optimise processed visual data based on the comparison between the characteristic of the training data and the characteristic of the reference dataset,

wherein the algorithm outputs the processed visual data with the same content as the at least a section of received visual data.