Accelerating machine optimisation processes

• US 10,516,890 B2
• Filed: 08/18/2017
• Issued: 12/24/2019
• Est. Priority Date: 02/19/2015
• Status: Expired due to Fees

• Current Assignee: Magic Pony Technology Limited (X Holdings Corp.)
• Original Assignee: Magic Pony Technology Limited (X Holdings Corp.)
• Inventors: Wang, Zehan, Bishop, Robert David, Shi, Wenzhe, Caballero, Jose, Aitken, Andrew Peter, Totz, Johannes
• Primary Examiner(s): Hsieh, Ping Y
• Application Number:
  US15/680,708
• Publication Number:
  US 20180129918A1
• Time in Patent Office: 858 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 3/4046 using neural networks
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/60 using machine learning, e.g. neural networks
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/142 Detection of scene cut or scene change
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/172 : the region being a picture, frame or field
• CPC Class Codes: H04N 19/176 : the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/177 : the unit being a group of pictures [GOP]
• CPC Class Codes: H04N 19/31 : in the temporal domain
• CPC Class Codes: H04N 19/33 : in the spatial domain
• CPC Class Codes: H04N 19/36 : Scalability techniques involving formatting the layers as a function of picture distortion after decoding, e.g. signal-to-noise [SNR] scalability
• CPC Class Codes: H04N 19/46 : Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/463 : by compressing encoding parameters before transmission
• CPC Class Codes: H04N 19/59 : involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/63 : using sub-band based transform, e.g. wavelets
• CPC Class Codes: H04N 19/80 : Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 19/86 : involving reduction of coding artifacts, e.g. of blockiness
• CPC Class Codes: H04N 19/87 : involving scene cut or scene change detection in combination with video compression
• CPC Class Codes: H04N 7/0117 : involving conversion of the spatial resolution of the incoming video signal for graphics images G09G2340/0407

Abstract

A method for training learned hierarchical algorithms, the method comprising the steps of receiving input data and generating metrics from the input data. At least one hierarchical algorithm is then selected from a plurality of predetermined hierarchical algorithms based on comparing the generated metrics from the input data and like metrics for each of the plurality of predetermined hierarchical algorithms. The selected hierarchical algorithm is developed based on the input data and the developed hierarchical algorithm is outputted.

Claims

1. A method for training learned hierarchical algorithms, the method comprising the steps of:

receiving input data;

generating metric data from the input data, the metric data measuring quality of output data produced for the input data by a plurality of pre-trained hierarchical algorithms stored in a library, wherein each of the plurality of pre-trained hierarchical algorithms is associated with respective metric data and each is trained on different input data;

selecting at least one hierarchical algorithm from the plurality of pre-trained hierarchical algorithms based on comparing the respective metric data;

training, using a deep learning approach, the at least one hierarchical algorithm based on the input data to generate a new trained hierarchical algorithm; and

adding the new trained a hierarchical algorithm to the library.

2. The method of claim 1, wherein each of the plurality of pre-trained hierarchical algorithms are stored with features extracted from the input data used in training the hierarchical algorithm.

3. The method of claim 1, wherein training the at least one hierarchical algorithm includes training the at least one hierarchical algorithm for a predetermined number of iterations.

4. The method of claim 1, wherein the metric data includes an error rate, a peak signal-to-noise ratio, a distance metric, a similarity measure, or a structural similarity index.

5. The method of claim 1, wherein training the at least one hierarchical algorithm includes training the at least one hierarchical algorithm until a predetermined value of reconstruction error is reached.

6. The method of claim 1, wherein the input data is a section of video and the metric data is based on a style of the section of video.

7. The method of claim 1, wherein the hierarchical algorithm is a non-linear hierarchical algorithm.

8. The method of claim 1, wherein the hierarchical algorithm performs image enhancement, using super-resolution techniques.

9. The method of claim 1, wherein the hierarchical algorithm uses a spatio-temporal approach.

10. The method of claim 1, wherein each of the pre-trained hierarchical algorithms in the library has a library reference identifier.

11. A method for training neural networks, the method comprising the steps of:

receiving input data;

generating metric data from the input data, the metric data measuring quality of output data produced for the input data by a plurality of pre-trained neural networks stored in a library, wherein each of the plurality of pre-trained neural networks is associated with respective metric data and each is trained on different input data;

selecting at least one neural network from the plurality of pre-trained neural networks based on comparing the respective metric data;

training, using a deep learning approach, the at least one neural network based on the input data to generate a new trained neural network; and

adding the new trained neural network to the library.

12. The method of claim 11, further comprising the step of:

storing the new trained neural network with features generated from the input data.

13. The method of claim 11, wherein the metric data is based on generating the highest quality output data, and wherein the quality is defined using an error rate, a peak signal-to-noise ratio, a distance metric, a similarity measure, or a structural similarity index.

14. The method of claim 11, wherein the at least one neural network is a convolutional neural network.

15. The method of claim 11, wherein the metric data is a similarity measure, generating the metric data from the input data includes extracting features of the input data and selecting the at least one neural network includes:

using the extracted features to select the at least one pre-trained neural network from the plurality of neural networks as associated with features most similar to the features of the input data.

16. The method of claim 11, wherein each of the pre-trained neural networks in the library has a library reference identifier.

17. The method of claim 11, wherein the respective metric data measures how optimal the pre-trained neural network is for providing initialisation data for generating the new neural network.

18. The method of claim 11, wherein the input data comprises any of:

a sequence of frames of visual data and a region within a sequence of frames of visual data.

19. The method of claim 11, wherein the training occurs for a pre-determined number of iterations to limit computational time.

20. A computer program product embodied on a non-transitory storage medium and comprising instructions that, when executed, cause a system to train learned hierarchical algorithms, by performing the steps of:

receiving input data;

generating metric data from the input data, the metric data measuring quality of output data produced for the input data by a plurality of pre-trained hierarchical algorithms stored in a library, wherein each of the plurality of pre-trained hierarchical algorithms is associated with respective metric data and each is trained on different input data;

selecting at least one hierarchical algorithm from the plurality of pre-trained hierarchical algorithms based on comparing the respective metric data;

training, using a deep learning approach, the at least one hierarchical algorithm based on the input data to generate a new trained hierarchical algorithm; and

adding the new trained hierarchical algorithm to the library.