Visual processing using temporal and spatial interpolation

• US 10,630,996 B2
• Filed: 08/18/2017
• Issued: 04/21/2020
• Est. Priority Date: 02/19/2015
• Status: Active Grant

• Current Assignee: Magic Pony Technology Limited (X Holdings Corp.)
• Original Assignee: Magic Pony Technology Limited (X Holdings Corp.)
• Inventors: Wang, Zehan, Bishop, Robert David, Shi, Wenzhe, Caballero, Jose, Aitken, Andrew Peter, Totz, Johannes
• Primary Examiner(s): Liew, Alex Kok S
• Application Number:
  US15/681,092
• Publication Number:
  US 20180130180A1
• Time in Patent Office: 977 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 3/4046 using neural networks
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/60 using machine learning, e.g. neural networks
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/142 Detection of scene cut or scene change
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/172 : the region being a picture, frame or field
• CPC Class Codes: H04N 19/176 : the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/177 : the unit being a group of pictures [GOP]
• CPC Class Codes: H04N 19/31 : in the temporal domain
• CPC Class Codes: H04N 19/33 : in the spatial domain
• CPC Class Codes: H04N 19/36 : Scalability techniques involving formatting the layers as a function of picture distortion after decoding, e.g. signal-to-noise [SNR] scalability
• CPC Class Codes: H04N 19/46 : Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/463 : by compressing encoding parameters before transmission
• CPC Class Codes: H04N 19/59 : involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/63 : using sub-band based transform, e.g. wavelets
• CPC Class Codes: H04N 19/80 : Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 19/86 : involving reduction of coding artifacts, e.g. of blockiness
• CPC Class Codes: H04N 19/87 : involving scene cut or scene change detection in combination with video compression
• CPC Class Codes: H04N 7/0117 : involving conversion of the spatial resolution of the incoming video signal for graphics images G09G2340/0407

Abstract

A method for enhancing at least a section of lower-quality visual data using a hierarchical algorithm, the method including receiving at least a plurality of neighbouring sections of lower-quality visual data. A plurality of input sections from the received plurality of neighbouring sections of lower quality visual data are selected and features are extracted from those plurality of input sections of lower-quality visual data. A target section based on the extracted features from the plurality of input sections of lower-quality visual data is then enhanced.

Claims

1. A method for enhancing a section of lower-quality visual data using a hierarchical algorithm, the method comprising the steps of:

receiving a plurality of neighbouring sections of lower-quality visual data;

selecting a plurality of input sections from the received plurality of neighbouring sections of lower quality visual data;

extracting features from the plurality of input sections;

generating at least one feature map based on the extracted features; and

enhancing a target section by upscaling the target section using a sub-pixel convolution layer of the hierarchical algorithm that aggregates the at least one feature map.

2. The method of claim 1, wherein the target section corresponds to one of the received plurality of neighbouring sections of lower-quality visual data.

3. The method of claim 1, wherein the target section does not correspond to one of the received plurality of neighbouring sections of lower-quality visual data.

4. The method of claim 1, wherein the lower-quality visual data includes a plurality of frames of video.

5. The method of claim 1, wherein a section of visual data includes at least one of a single frame of visual data, a sequence of frames of visual data, and a region within a frame or sequence of frames of visual data.

6. The method of claim 5, wherein at least one of the plurality of selected input sections and the target sections are frames of lower-quality visual data, and the at least one of the plurality of selected input sections occurs sequentially before the target section.

7. The method of claim 5, wherein at least one of the plurality of selected input sections and the target sections are frames of lower-quality visual data, and the at least one of the plurality of selected input sections occurs sequentially after the target section.

8. The method of claim 1, wherein the hierarchical algorithm includes a plurality of layers.

9. The method according to claim 8, wherein the layers are at least one of sequential, recurrent, recursive, branching, and merging.

10. The method of claim 1, wherein the extracting of the features is based on a predetermined extraction metric.

11. The method of claim 1, wherein the received plurality of neighbouring sections of lower-quality visual data are consecutive sections of lower-quality visual data.

12. The method of claim 1, wherein a predetermined algorithm metric used to determine the hierarchical algorithm to be selected.

13. The method of claim 12, wherein the predetermined algorithm metric is based on generating a highest quality version of the lower-quality visual data, wherein a quality is defined by at least one of an error rate, a bit error rate, a peak signal-to-noise ratio, and a structural similarity index.

14. The method of claim 1, wherein a standardised feature of the received plurality of neighbouring sections of lower-quality visual data are extracted and used to select the hierarchical algorithm from a library of algorithms.

15. The method of claim 1, wherein the hierarchical algorithm is at least one of:

developed using a machine learning technique,

a non-linear hierarchical algorithm, and

at least one convolutional neural network.

16. The method of claim 1, wherein the hierarchical algorithm is configured to be used to perform image enhancement using super-resolution techniques.

17. The method of claim 1, wherein the hierarchical algorithm uses a spatio-temporal approach.

18. The method of claim 1, wherein a higher-quality visual data is at a higher resolution than the lower-quality visual data.

19. The method of claim 1, wherein the lower-quality visual data includes a higher amount of artefacts than a higher-quality visual data.

20. The method of claim 1, wherein enhancing a quality of visual data includes upscaling the quality of the visual data.