Enhancing visual data using strided convolutions

• US 10,582,205 B2
• Filed: 08/18/2017
• Issued: 03/03/2020
• Est. Priority Date: 02/19/2015
• Status: Active Grant

• Current Assignee: Magic Pony Technology Limited (X Holdings Corp.)
• Original Assignee: Magic Pony Technology Limited (X Holdings Corp.)
• Inventors: Wang, Zehan, Bishop, Robert David, Shi, Wenzhe, Caballero, Jose, Aitken, Andrew Peter, Totz, Johannes
• Primary Examiner(s): Abdi, Amara
• Application Number:
  US15/680,753
• Publication Number:
  US 20180130178A1
• Time in Patent Office: 928 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 3/4046 using neural networks
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/60 using machine learning, e.g. neural networks
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/142 Detection of scene cut or scene change
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/172 : the region being a picture, frame or field
• CPC Class Codes: H04N 19/176 : the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/177 : the unit being a group of pictures [GOP]
• CPC Class Codes: H04N 19/31 : in the temporal domain
• CPC Class Codes: H04N 19/33 : in the spatial domain
• CPC Class Codes: H04N 19/36 : Scalability techniques involving formatting the layers as a function of picture distortion after decoding, e.g. signal-to-noise [SNR] scalability
• CPC Class Codes: H04N 19/46 : Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/463 : by compressing encoding parameters before transmission
• CPC Class Codes: H04N 19/59 : involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/63 : using sub-band based transform, e.g. wavelets
• CPC Class Codes: H04N 19/80 : Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 19/86 : involving reduction of coding artifacts, e.g. of blockiness
• CPC Class Codes: H04N 19/87 : involving scene cut or scene change detection in combination with video compression
• CPC Class Codes: H04N 7/0117 : involving conversion of the spatial resolution of the incoming video signal for graphics images G09G2340/0407

Abstract

A method for enhancing at least a section of lower-quality visual data using a hierarchical algorithm, the method comprises receiving at least one section of lower-quality visual data; and extracting a subset of features, from the at least one section of lower-quality visual data. A plurality of layers of reduced-dimension visual data from the extracted features are formed and enhanced to form at least one section of higher-quality visual data. The at least one section of higher-quality visual data corresponds to the at least one section of lower-quality visual data received.

Claims

1. A method for enhancing a section of low resolution visual data, the method comprising:

receiving at least one section of low-resolution visual data;

receiving at least one convolutional neural network based on the at least one section of low-resolution visual data;

extracting, using the at least one convolutional neural network, a subset of features from the at least one section of low-resolution visual data;

forming, using the at least one convolutional neural network, a plurality of feature maps of reduced-dimension visual data from the extracted subset of features; and

mapping, using the at least one convolutional neural network, the plurality of feature maps of reduced-dimension visual data to at least one section of high-resolution visual data using a sub-pixel convolution layer, wherein the at least one section of high-resolution visual data corresponds to the at least one section of low-resolution visual data.

2. The method of claim 1, wherein the extracted subset of features is based on at least one predetermined factor indicating a downscaling factor of the low-resolution visual data.

3. The method of claim 2, wherein a number of maps of the plurality of feature maps is based on the at least one predetermined factor.

4. The method of claim 1, wherein the at least one convolutional neural network differs for each section of visual data.

5. The method of claim 1, wherein the at least one convolutional neural network is selected from a library of convolutional neural networks.

6. The method of claim 5, wherein standardized features of the at least one section of received low-resolution visual data are extracted and used to select the at least one convolutional neural network from the library of convolutional neural networks.

7. The method of claim 5, wherein the at least one convolutional neural network to be selected from the library of convolutional neural networks is based on generating the highest resolution version of the low resolution visual data.

8. The method of claim 1, wherein the at least one convolutional neural network is developed using a learned approach.

9. The method of claim 8, wherein the learned approach includes a machine learning technique.

10. The method of claim 1, wherein the at least one convolutional neural network is non-linear.

11. The method of claim 1, wherein the at least one convolutional neural network performs image enhancement using a super-resolution technique.

12. The method of claim 1, wherein the at least one convolutional neural network uses a spatio-temporal approach.

13. The method of claim 1, wherein the high-resolution visual data is at a higher-resolution than the low-resolution visual data.

14. The method of claim 1, wherein the low-resolution visual data contains a higher amount of artefacts than the high-resolution visual data.

15. The method of claim 1, wherein the visual data includes a plurality of frames of video, or a plurality of images.

16. The method of claim 1, wherein in enhancing the at least one section visual data includes upscaling a quality of the at least one section visual data.

17. The method of claim 1, wherein the section of low resolution visual data includes at least one of a single frame of visual data, a sequence of frames of visual data, and a region within a frame or sequence of frames of visual data.

18. The method of claim 1, wherein the at least one section of low-resolution visual data includes a plurality of hierarchical layers that are at least one of sequential, recurrent, recursive, branching, and merging.

19. The method of claim 1, wherein the sub-pixel convolutional layer performs a convolution with a fractional stride.

20. The method of claim 19, wherein mapping the plurality of feature maps of reduced dimension visual data to at least one section of high-resolution visual data includes applying a periodic shuffling to an output of the convolution with the fractional stride.