Enhancing visual data using and augmenting model libraries

• US 10,499,069 B2
• Filed: 08/18/2017
• Issued: 12/03/2019
• Est. Priority Date: 02/19/2015
• Status: Expired due to Fees

• Current Assignee: Magic Pony Technology Limited (X Holdings Corp.)
• Original Assignee: Magic Pony Technology Limited (X Holdings Corp.)
• Inventors: Wang, Zehan, Bishop, Robert David, Shi, Wenzhe, Caballero, Jose, Aitken, Andrew Peter, Totz, Johannes
• Primary Examiner(s): Hsieh, Ping Y
• Application Number:
  US15/680,716
• Publication Number:
  US 20170345130A1
• Time in Patent Office: 837 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 3/4046 using neural networks
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/60 using machine learning, e.g. neural networks
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/142 Detection of scene cut or scene change
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/172 : the region being a picture, frame or field
• CPC Class Codes: H04N 19/176 : the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/177 : the unit being a group of pictures [GOP]
• CPC Class Codes: H04N 19/31 : in the temporal domain
• CPC Class Codes: H04N 19/33 : in the spatial domain
• CPC Class Codes: H04N 19/36 : Scalability techniques involving formatting the layers as a function of picture distortion after decoding, e.g. signal-to-noise [SNR] scalability
• CPC Class Codes: H04N 19/46 : Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/463 : by compressing encoding parameters before transmission
• CPC Class Codes: H04N 19/59 : involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/63 : using sub-band based transform, e.g. wavelets
• CPC Class Codes: H04N 19/80 : Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 19/86 : involving reduction of coding artifacts, e.g. of blockiness
• CPC Class Codes: H04N 19/87 : involving scene cut or scene change detection in combination with video compression
• CPC Class Codes: H04N 7/0117 : involving conversion of the spatial resolution of the incoming video signal for graphics images G09G2340/0407

Abstract

A method for enhancing visual data when communicating visual data over a network from a first node to a second node. The method at the first node comprises developing at least one modified hierarchical algorithm from a known hierarchical algorithm operable to substantially recreate at least one section of higher-quality visual data. References to one or more known hierarchical algorithms from which the modified hierarchical algorithms were developed are transmitted to the second node along with one or more modifications to the one or more known hierarchical algorithms operable to reproduce the one or more modified hierarchical algorithms from the known hierarchical algorithms. The second node is able to recreate substantially the higher-quality video using the modified hierarchical algorithm.

Claims

1. A method for enhancing a quality of visual data when communicating visual data over a network, the method comprising:

developing, at a first node of a network, a modified hierarchical algorithm from a known hierarchical algorithm, including training the known hierarchical algorithm based on visual data, the modified hierarchical algorithm configured to substantially recreate an original resolution version of the visual data from a lower-resolution version of the visual data;

transmitting, to a second node of the network, a reference to the known hierarchical algorithm from which the modified hierarchical algorithm was developed;

transmitting, to the second node, one or more modifications to the known hierarchical algorithm, the one or more modifications configured to reproduce the modified hierarchical algorithm from the known hierarchical algorithm; and

transmitting, to the second node, the lower-resolution version of the visual data.

2. The method of claim 1, wherein the visual data includes a single frame of visual data, a sequence of frames of visual data, or a region within a frame or sequence of frames of visual data.

3. The method of claim 1, wherein the modified hierarchical algorithm is configured to perform image enhancement using a super-resolution technique.

4. The method of claim 1, wherein the modified hierarchical algorithm includes a neural network having a plurality of layers.

5. The method of claim 1, wherein the training includes receiving the original resolution version of the visual data and the lower-resolution version of the visual data, and training the known hierarchical algorithm using at least one of the original resolution version of the visual data or the lower-resolution version of the visual data.

6. The method of claim 1, further comprising:

selecting the known hierarchical algorithm from a plurality of pre-trained hierarchical algorithms stored in a library.

7. A method for enhancing a quality of visual data when communicating visual data over a network, the method comprising:

receiving, from a first node of a network, a reference to a known hierarchical algorithm at a second node of the network;

receiving, from the first node, one or more modifications to the known hierarchical algorithm at the second node;

receiving, from the first node, a lower-resolution version of visual data at the second node;

reproducing, at the second node, a modified hierarchical algorithm based on the reference to the known hierarchical algorithm and the one or more modifications; and

substantially recreating, at the second node, an original resolution version of the visual data using the modified hierarchical algorithm and the lower-resolution version of the visual data.

8. The method of claim 7, further comprising:

sending, by the second node, a transmission request for the known hierarchical algorithm in response to the second node failing to identify the known hierarchical algorithm from the reference; and

receiving, from the first node, the known hierarchical algorithm in response to the transmission request.

9. The method of claim 7, further comprising:

storing, at the second node, the modified hierarchical algorithm in a library.

10. The method of claim 7, wherein the visual data includes a plurality of frames of video, or a plurality of images.

11. The method of claim 7, further comprising:

selecting, at the second node, the known hierarchical algorithm from a library storing a plurality of known hierarchical algorithm using the reference.

12. A system for enhancing visual data when communicating visual data over a network, the system comprising:

a first computing device, the first computing device representing wherein a first node of a network, the first computing device-s configured to cause at least one processor to;

develop a modified hierarchical algorithm from a known hierarchical algorithm, including training the known hierarchical algorithm based on visual data, the modified hierarchical algorithm configured to operable to substantially recreate an original resolution version of the visual data from a lower-resolution version of the visual data;

transmit, to a second computing device, a reference to the known hierarchical algorithm from which the modified hierarchical algorithm was developed, the second computing device representing a second node of the network;

transmit, to the second computing device, one or more modifications to the known hierarchical algorithm, the one or more modifications configured to reproduce the modified hierarchical algorithm from the known hierarchical algorithm; and

transmit, to the second computing device, the lower-resolution version of the visual data.

13. The system of claim 12, wherein the modified hierarchical algorithm is selected from a group including:

a non-linear hierarchical algorithm;

one or more convolutional neural networks; and

an algorithm trained using a learned approach.

14. The system of claim 12, wherein the modified hierarchical algorithm is stored in a library at the first computing device.

15. The system of claim 12, wherein the first computing device is configured to down-sample the visual data to the lower-resolution version.

16. The system of claim 12, wherein the lower-resolution version of the visual data has a higher amount of artefacts than the original resolution version of the visual data.

17. The system of claim 12, wherein the modified hierarchical algorithm includes a neural network having a plurality of layers.

18. The system of claim 12, wherein the second computing device is configured to cause at least one processor to:

reproduce the modified hierarchical algorithm based on the reference to the known hierarchical algorithm and the one or more modifications; and

substantially recreate the original resolution version of the visual data using the modified hierarchical algorithm and the lower-resolution version of the visual data.