Visual processing using temporal and spatial interpolation

• US 10,547,858 B2
• Filed: 08/17/2017
• Issued: 01/28/2020
• Est. Priority Date: 02/19/2015
• Status: Active Grant

• Current Assignee: Magic Pony Technology Limited (X Holdings Corp.)
• Original Assignee: Magic Pony Technology Limited (X Holdings Corp.)
• Inventors: Wang, Zehan, Bishop, Robert David, Shi, Wenzhe, Caballero, Jose, Aitken, Andrew Peter, Totz, Johannes
• Primary Examiner(s): Williams, Jeffery A
• Application Number:
  US15/679,660
• Publication Number:
  US 20170347110A1
• Time in Patent Office: 894 Days
• Field of Search: 37524001
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 3/4046 using neural networks
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/60 using machine learning, e.g. neural networks
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/142 Detection of scene cut or scene change
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/172 : the region being a picture, frame or field
• CPC Class Codes: H04N 19/176 : the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/177 : the unit being a group of pictures [GOP]
• CPC Class Codes: H04N 19/31 : in the temporal domain
• CPC Class Codes: H04N 19/33 : in the spatial domain
• CPC Class Codes: H04N 19/36 : Scalability techniques involving formatting the layers as a function of picture distortion after decoding, e.g. signal-to-noise [SNR] scalability
• CPC Class Codes: H04N 19/46 : Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/463 : by compressing encoding parameters before transmission
• CPC Class Codes: H04N 19/59 : involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/63 : using sub-band based transform, e.g. wavelets
• CPC Class Codes: H04N 19/80 : Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 19/86 : involving reduction of coding artifacts, e.g. of blockiness
• CPC Class Codes: H04N 19/87 : involving scene cut or scene change detection in combination with video compression
• CPC Class Codes: H04N 7/0117 : involving conversion of the spatial resolution of the incoming video signal for graphics images G09G2340/0407

Abstract

A method for enhancing at least a section of lower-quality visual data using a hierarchical algorithm, the method comprising receiving at least a plurality of neighbouring sections of lower-quality visual data. A plurality of input sections from the received plurality of neighbouring sections of lower quality visual data are selected and features are extracted from those plurality of input sections of lower-quality visual data. A target section based on the extracted features from the plurality of input sections of lower-quality visual data is then enhanced.

Claims

1. A method for upscaling at least a section of low-resolution video data using a convolutional neural network (CNN), the method comprising the steps of:

receiving at least three consecutive frames of low-resolution video data;

inputting the at least three consecutive frames of low-resolution video data into an initial layer of the CNN;

extracting, using a plurality of hidden convolutional layers of the CNN, low-resolution features from the at least three consecutive frames of low-resolution video data; and

enhancing, using a hidden convolutional layer of the CNN, the extracted low-resolution features from the three or more consecutive frames of low-resolution video data to generate a higher-resolution target section of video data corresponding to a middle frame of the at least three consecutive frames of low-resolution video data,

wherein the CNN is trained on training data including ground truth sections of video data with corresponding sequences of three or more consecutive frames of sub-sampled video data to reproduce ground truth sections of video data from the corresponding frames of sub-sampled video data.

2. The method according to claim 1, wherein the higher-resolution target section of video data corresponds to a single frame of the low-resolution video data.

3. The method according to claim 1, wherein the higher-resolution target section of video data does not correspond to one of the received frames of low-resolution video data.

4. The method according to claim 1, wherein at least one of the frames of low-resolution video data occurs sequentially before the target section.

5. The method according to claim 1, wherein at least one of the frames of low-resolution video data occurs sequentially after the target section.

6. The method according to claim 1, wherein the CNN includes a plurality of layers.

7. The method according to claim 6, wherein the plurality of layers includes one or more of a sequential layer, a recurrent layer, a recursive layer, a branching layer, or a merging layer.

8. The method according to claim 1, further comprising using a predetermined selection criterion to determine the hierarchical CNN to be used from a library of CNNs.

9. The method according to claim 8, wherein the predetermined selection criterion is based on generating the higher-resolution target section of video data the low-resolution video data, based on a measure of at least one of:

an error rate;

a bit error rate;

a peak signal-to-noise ratio;

or a structural similarity index.

10. The method according to claim 1, further comprising selecting the CNN from a library of CNNs based on standardised features extracted from the received frames of low-resolution video data.

11. The method according to claim 1, further comprising developing the CNN using a learned approach that includes machine learning techniques.

12. The method according to claim 1, further comprising performing image enhancement, using super-resolution techniques, with the CNN.

13. The method according to claim 1, wherein the lower-quality visual data contains a higher amount of artefacts than the higher-quality visual data.

15. The method of claim 1, wherein the at least three consecutive frames of low-resolution video data include multiple repetitions of a first frame or a last frame of video data.

14. A computer program product embodied on a non-transitory storage medium and comprising instructions that, when executed, cause a system to upscale at least a section of low-resolution video data using a CNN, by performing the steps of:

receiving at least three consecutive frames of low-resolution video data;

inputting the at least three consecutive frames of low-resolution video data into an initial layer of the CNN;

extracting, using a plurality of hidden convolutional layers of the CNN, low-resolution features from the at least three consecutive frames of low-resolution video data; and

enhancing, using a hidden convolutional layer of the CNN, the extracted low-resolution features from the three or more consecutive frames of low-resolution video data to generate a higher-resolution target section of video data corresponding to a middle frame of the at least three consecutive frames of low-resolution video data,

wherein the CNN is trained on training data including ground truth sections of video data with corresponding sequences of three or more consecutive frames of sub-sampled video data to reproduce ground truth sections of video data from the corresponding frames of sub-sampled video data.

16. The computer program product of claim 14, wherein the higher-resolution target section of video data corresponds to a single frame of the low-resolution video data.

17. The computer program product of claim 14, wherein the higher-resolution target section of video data does not correspond to one of the received frames of low-resolution video data.

18. The computer program product of claim 14, wherein at least one of the frames of low-resolution video data occurs sequentially before the target section.

19. The computer program product of claim 14, wherein at least one of the frames of low-resolution video data occurs sequentially after the target section.

20. The computer program product of claim 14, wherein the CNN includes a plurality of layers.

21. The computer program product of claim 20, wherein the plurality of layers includes one or more of a sequential layer, a recurrent layer, a recursive layer, a branching layer, or a merging layer.

22. The computer program product of claim 14, further comprising instructions that, when executed, cause the system to use a predetermined selection criterion to determine the CNN to be used from a library of CNNs.

23. The computer program product of claim 22, wherein the predetermined selection criterion is based on generating the higher-resolution target section of video data the low-resolution video data, based on a measure of at least one of:

an error rate;

a bit error rate;

a peak signal-to-noise ratio;

or a structural similarity index.

24. The computer program product of claim 14, further comprising instructions that, when executed, cause the system to select the CNN from a library of CNNs based on standardized features extracted from the received frames of low-resolution video data.

25. The computer program product of claim 14, further comprising instructions that, when executed, cause the system to develop the CNN using a learned approach that includes machine learning techniques.

26. The computer program product of claim 14, further comprising instructions that, when executed, cause the system to perform image enhancement, using super-resolution techniques, with the CNN.

27. The computer program product of claim 14, wherein the lower-quality visual data contains a higher amount of artefacts than the higher-quality visual data.

28. A system for upscaling at least a section of low-resolution video data using a CNN, the system comprising:

a least one processor;

memory storing instructions executable that, when executed the at least one processor, cause the system to;

receive at least three consecutive frames of low-resolution video data;

inputting the at least three consecutive frames of low-resolution video data into an initial layer of the CNN;

extract, using a plurality of hidden convolutional layers of the CNN, low-resolution features from the at least three consecutive frames of low-resolution video data; and

enhance, using a hidden convolutional layer of the CNN, the extracted low-resolution features from the three or more consecutive frames of low-resolution video data to generate a higher-resolution target section of video data corresponding to a middle frame of the at least three consecutive frames of low-resolution video data,

wherein the CNN is trained on training data including ground truth sections of video data with corresponding sequences of three or more consecutive frames of sub-sampled video data to reproduce ground truth sections of video data from the corresponding frames of sub-sampled video data.

29. The system of claim 28, wherein the higher-resolution target section of video data corresponds to a single frame of the low-resolution video data.

30. The system of claim 28, wherein the higher-resolution target section of video data does not correspond to one of the received frames of low-resolution video data.

31. The system of claim 28, wherein at least one of the frames of low-resolution video data occurs sequentially before the target section.

32. The system of claim 28, wherein at least one of the frames of low-resolution video data occurs sequentially after the target section.

33. The system of claim 28, wherein the CNN includes a plurality of layers.

34. The system of claim 33, wherein the plurality of layers includes one or more of a sequential layer, a recurrent layer, a recursive layer, a branching layer, or a merging layer.

35. The system of claim 28, wherein the instructions, when executed by the at least one processor, further cause the system to use a predetermined selection criterion to determine the CNN to be used from a library of CNNs.

36. The system of claim 35, wherein the predetermined selection criterion is based on generating the higher-resolution target section of video data the low-resolution video data, based on a measure of at least one of:

an error rate;

a bit error rate;

a peak signal-to-noise ratio;

or a structural similarity index.

37. The system of claim 28, wherein the instructions, when executed by the at least one processor, further cause the system to select the CNN from a library of CNNs based on standardized features extracted from the received frames of low-resolution video data.

38. The system of claim 28, wherein the instructions, when executed by the at least one processor, further cause the system to develop the CNN using a learned approach that includes machine learning techniques.

39. The system of claim 28, wherein the instructions, when executed by the at least one processor, further cause the system to perform image enhancement, using super-resolution techniques, with the CNN.

40. The system of claim 28, wherein the lower-quality visual data contains a higher amount of artefacts than the higher-quality visual data.