Enhancement of visual data

• US 10,523,955 B2
• Filed: 08/17/2017
• Issued: 12/31/2019
• Est. Priority Date: 02/19/2015
• Status: Expired due to Fees

• Current Assignee: Magic Pony Technology Limited (X Holdings Corp.)
• Original Assignee: Magic Pony Technology Limited (X Holdings Corp.)
• Inventors: Wang, Zehan, Bishop, Robert David, Shi, Wenzhe, Caballero, Jose, Aitken, Andrew Peter, Totz, Johannes
• Primary Examiner(s): Hsieh, Ping Y
• Application Number:
  US15/679,977
• Publication Number:
  US 20180130177A1
• Time in Patent Office: 866 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 3/4046 using neural networks
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/60 using machine learning, e.g. neural networks
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/142 Detection of scene cut or scene change
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/172 : the region being a picture, frame or field
• CPC Class Codes: H04N 19/176 : the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/177 : the unit being a group of pictures [GOP]
• CPC Class Codes: H04N 19/31 : in the temporal domain
• CPC Class Codes: H04N 19/33 : in the spatial domain
• CPC Class Codes: H04N 19/36 : Scalability techniques involving formatting the layers as a function of picture distortion after decoding, e.g. signal-to-noise [SNR] scalability
• CPC Class Codes: H04N 19/46 : Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/463 : by compressing encoding parameters before transmission
• CPC Class Codes: H04N 19/59 : involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/63 : using sub-band based transform, e.g. wavelets
• CPC Class Codes: H04N 19/80 : Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 19/86 : involving reduction of coding artifacts, e.g. of blockiness
• CPC Class Codes: H04N 19/87 : involving scene cut or scene change detection in combination with video compression
• CPC Class Codes: H04N 7/0117 : involving conversion of the spatial resolution of the incoming video signal for graphics images G09G2340/0407

Abstract

A method for enhancing at least a section of lower-quality visual data, the method comprising at least a section of the lower-quality visual data being received. A hierarchical algorithm is then selected from a plurality of hierarchical algorithms, wherein the step of selection is based on a predetermined metric and wherein the hierarchical algorithms were developed using a learned approach and at least one of the hierarchical algorithms is operable to increase the quality of the lower-quality visual data. The selected hierarchical algorithm is then used to increase the quality of the lower-quality visual data to create a higher-quality visual data.

Claims

1. A method for enhancing visual data, the method comprising steps of:

receiving at least a section of the visual data at an original quality;

selecting a hierarchical algorithm from a plurality of hierarchical algorithms stored in a library based on a similarity metric, wherein each hierarchical algorithm in the library is associated with respective features, the respective features for a hierarchical algorithm being features extracted from visual data used to train the hierarchical algorithm, wherein the selected hierarchical algorithm is most similar to the section based on the similarity metric applied to the respective features, and wherein the selected hierarchical algorithm is operable to increase the quality of the visual data; and

using the selected hierarchical algorithm to increase the quality of the visual data to create a higher-quality visual data.

2. The method of claim 1, wherein the visual data at the original quality is divided into smaller sections based on similarities between a plurality of sections of the visual data.

3. The method of claim 1, wherein the similarity metric is based on visual bag-of-words, histogram extraction, or scale invariant feature transform.

4. The method of claim 1, wherein the plurality of hierarchical algorithms is selected from the group including:

pre-trained hierarchical algorithms on example data; and

manually created hierarchical algorithms.

5. The method of claim 1, wherein, if a suitable hierarchical algorithm is not present in the plurality of hierarchical algorithms, a generic hierarchical algorithm is selected instead.

6. The method of claim 1, wherein the hierarchical algorithm is selected from a group including:

an algorithm trained using a learned approach, an algorithm that comprises a plurality of layers, a non-linear hierarchical algorithm, and an algorithm that uses a spatio-temporal approach.

7. The method of claim 1, wherein the visual data used to train the plurality of hierarchical algorithms is classified into a category depending on its content, wherein each hierarchical algorithm is associated with a different category.

8. The method of claim 1, wherein the plurality of hierarchical algorithms include different model types.

9. The method of claim 1, wherein the section is a frame of the visual data.

10. A method for increasing the quality of at least a section of a lower-quality visual data, the method comprising the steps of:

receiving the at least a section of the lower-quality visual data;

selecting an example-based algorithm from a plurality of example-based algorithms in a library, wherein the step of selecting is based on a similarity metric measuring similarity between content of the section and content of visual data used to train the example-based algorithm, the library storing data from which to determine the similarity metric, and wherein the example-based algorithms were formed using a learned approach and the example-based algorithms are operable to increase the quality of the lower-quality visual data; and

using the selected example-based algorithm to increase the quality of the lower-quality visual data to create a higher-quality visual data.

11. The method of claim 10, wherein the example-based algorithm is selected from the group including:

a non-linear hierarchical algorithm;

a dictionary algorithm;

a learned sparse algorithm; and

a collection of low-quality to high-quality mapping functions.

12. The method of claim 10, wherein the library includes example-based algorithms of different model types.

13. The method of claim 12, wherein the model type is based on a model that generates the highest quality upscaled visual data for the visual data used to train the example-based algorithm.

14. The method of claim 10, comprising the further steps of:

reducing the quality of an original visual data prior to transmission over a network at a first node to create a lower-quality visual data; and

transmitting the lower-quality visual data over the network from the first node to a second node,

wherein said method of claim 10 is carried out at the second node.

15. The method of claim 14, wherein the plurality of algorithms is stored in a library local to the first node.

16. The method of claim 10, wherein the content of the section of the lower-quality visual data are extracted as standardised features.

17. The method of claim 10, wherein the higher quality visual data has a higher resolution than the lower quality visual data or the higher quality visual data contains fewer artefacts than the lower-quality visual data.

18. The method of claim 10, wherein the section of the lower-quality visual data is selected from the group including:

a single section;

a sequence of sections; and

a region within a section or sequence of sections.

19. The method of claim 10, wherein enhancing the quality of visual data includes upscaling the quality of the visual data.

20. A computer program product embodied on a non-transitory storage medium and comprising instructions that, when executed, cause a system to enhance at least a section of lower-quality visual data using a hierarchical algorithm, by performing the steps of:

receiving at least a section of the lower-quality visual data;

selecting a hierarchical algorithm from a plurality of hierarchical algorithms in a library, each hierarchical algorithm having been trained using different visual data content, wherein the step of selection is based on a similarity metric measuring similarity between content of the section and the visual data content used to train the hierarchical algorithm and wherein the hierarchical algorithms were developed using a learned approach and at least one of the hierarchical algorithms is operable to increase the quality of the lower-quality visual data; and

using the selected hierarchical algorithm to increase the quality of the lower-quality visual data to create a higher-quality visual data.