Image upsampling system, training method thereof and image upsampling method

• US 10,019,642 B1
• Filed: 03/02/2016
• Issued: 07/10/2018
• Est. Priority Date: 09/17/2015
• Status: Active Grant

• Current Assignee: BOE Technology Group Co., Ltd.
• Original Assignee: BOE Technology Group Co., Ltd.
• Inventors: Navarrete Michelini, Pablo, Zhang, Lijie, He, Jianmin
• Primary Examiner(s): Bhatnagar, Anand
• Application Number:
  US15/518,404
• Publication Number:
  US 20180197037A1
• Time in Patent Office: 860 Days
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution

Abstract

An image upsampling system, a training method thereof and an image upsampling method are provided, the feature images of an image are obtained by using the convolutional network, upsampling processing is performed on the images with the muxer layer to synthesize every n×n feature images in the input signal into a feature image with the resolution amplified by n×n times, in the upsampling procedure with the muxer layer, information of respective feature images in the input signal is recorded in the generated feature image(s) without loss; and thus, every time when the image passes through a muxer layer with an upsampling multiple of n, the image resolution can be increased by n×n times.

Claims

1. An image upsampling system, comprising:

at least one first convolutional network and at least one muxer layer that are cascaded;

wherein an signal input end of the image upsampling system is connected with a signal input end of a first convolutional network in the at least one first convolutional network, and a signal output end of the image upsampling system is connected with a signal output end of a last muxer layer in the at least one muxer layer;

a signal input end of every muxer layer in the at least one muxer layer is connected with a signal output end of a first convolutional network located in a stage prior to the muxer layer in the at least one first convolutional network, or connected with a signal output end of another muxer layer located in a stage prior to the muxer layer in the at least one muxer layer;

the first convolutional network is configured for converting an image input to its signal input end into a plurality of feature images and outputting the feature images to the signal input end of the muxer layer connected therewith;

the muxer layer is configured for synthesizing every n×

n feature images in the feature images input to its signal input end into a feature image whose resolution is n×

n times that of the input feature image and outputting the same; and

a number of feature images input to the muxer layer is a multiple of n×

n, n being an integer greater than one.

2. The image upsampling system according to claim 1, wherein a number of muxer layers is two or three.

3. The image upsampling system according to claim 2, further comprising:

a second convolutional network, whose signal input end is connected with a signal output end of the last muxer layer in the at least one muxer layer, and whose signal output end is connected with the signal output end of the image upsampling system;

wherein the second convolutional network is configured for optimizing picture quality of the feature images output by the muxer layer.

4. The image upsampling system according to claim 3, wherein, the first convolutional network and the second convolutional network include at least one convolution layer composed of a plurality of filter units.

5. The image upsampling system according to claim 1, wherein, a signal input end of each muxer layer is respectively connected with a signal output end of one corresponding first convolutional network in the at least one first convolutional network.

6. The image upsampling system according to claim 5, further comprising:

a second convolutional network, whose signal input end is connected with a signal output end of the last muxer layer in the at least one muxer layer, and whose signal output end is connected with the signal output end of the image upsampling system;

wherein the second convolutional network is configured for optimizing picture quality of the feature images output by the muxer layer.

7. The image upsampling system according to claim 6, wherein, the first convolutional network and the second convolutional network include at least one convolution layer composed of a plurality of filter units.

8. The image upsampling system according to claim 1, wherein, in the case where there are provided a plurality of muxer layers, the muxer layers have a same upsampling multiple.

9. The image upsampling system according to claim 8, further comprising:

a second convolutional network, whose signal input end is connected with a signal output end of the last muxer layer in the at least one muxer layer, and whose signal output end is connected with the signal output end of the image upsampling system;

wherein the second convolutional network is configured for optimizing picture quality of the feature images output by the muxer layer.

10. The image upsampling system according to claim 9, wherein, the first convolutional network and the second convolutional network include at least one convolution layer composed of a plurality of filter units.

11. The image upsampling system according to claim 1, wherein, the muxer layer has an upsampling multiple which is a prime number.

12. The image upsampling system according to claim 11, wherein, the muxer layer has an upsampling multiple which is 2.

13. The image upsampling system according to claim 1, wherein, the muxer layer is a self-adaptive interpolation filter.

14. The image upsampling system according to claim 1, further comprising:

a second convolutional network, whose signal input end is connected with a signal output end of the last muxer layer in the at least one muxer layer, and whose signal output end is connected with the signal output end of the image upsampling system;

wherein the second convolutional network is configured for optimizing picture quality of the feature images output by the muxer layer.

15. The image upsampling system according to claim 14, wherein, the first convolutional network and the second convolutional network include at least one convolution layer composed of a plurality of filter units.

16. A display device, comprising the image upsampling system according to claim 1.

17. A training method of the image upsampling system according to claim 1, comprising:

initializing respective parameters in the image upsampling system;

by using an original image signal as an output signal of the image upsampling system and using an image signal obtained by down-sampling the original image signal as an input signal of the image upsampling system, adjusting the respective parameters in the image upsampling system to allow the down-sampled image signal subjected to upsampling processing with the adjusted respective parameters to be the same as the original image signal.

18. The training method according to claim 17, wherein, initializing of the respective parameters in the image upsampling system includes:

initializing weights W_ij of respective filter units of respective convolution layers of the first convolutional network and the second convolutional network in the image upsampling system according to a formula below;

19. The training method according to claim 17, wherein, initializing of respective parameters in the image upsampling system includes:

initializing the weights W_ij of respective filter units of respective convolution layers of the first convolutional network and the second convolutional network in the image upsampling system according to a formula below;

20. A method for performing image upsampling with the image upsampling system according to claim 1, comprising:

converting, by a first convolutional network, an input image input to the first convolutional network into a plurality of feature images and outputting the feature images;

synthesizing, by a muxer layer, every n×

n feature images in the feature images input to the muxer layer into a feature image with a resolution amplified by n×

n times as larger as the input feature images, and outputting the same;

the number of feature images input to the muxer layer being a multiple of n×

n, and n being an integer greater than 1.