Image processing method and device

• US 10,395,167 B2
• Filed: 09/22/2017
• Issued: 08/27/2019
• Est. Priority Date: 01/25/2017
• Status: Active Grant

• Current Assignee: BOE Technology Group Co., Ltd.
• Original Assignee: BOE Technology Group Co., Ltd.
• Inventors: Liu, Hanwen, Navarrete Michelini, Pablo
• Primary Examiner(s): Thomas, Mia M
• Application Number:
  US15/712,654
• Publication Number:
  US 20180211157A1
• Time in Patent Office: 704 Days
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 3/4046 using neural networks
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 10/82 using neural networks

Abstract

An image processing method and an image processing device are provided. The image processing method includes steps of extracting a feature of an inputted first image by a first CNN, and reconstructing and outputting an image by a second CNN. The first CNN includes a plurality of first convolutional layers connected sequentially to each other and a plurality of first pooling layers each arranged between respective adjacent first convolutional layers, and each first convolutional layer is configured to generate and output a first convolutional feature. The second CNN includes a plurality of second convolutional layers connected sequentially to each other and a plurality of composite layers each arranged between respective adjacent second convolutional layers, and each composite layer is an up-sampling layer.

Claims

1. An image processing method, comprising:

extracting one or more features of an inputted first image by a first Convolutional Neural Network (CNN), the inputted first image being inputted to the first one of the first convolutional layers, wherein the first CNN comprises a plurality of first convolutional layers connected sequentially to each other and a plurality of first pooling layers each connected to and arranged between respective adjacent first convolutional layers, and each of the first convolutional layers is configured to generate and output a first convolutional feature; and

reconstructing the inputted first image and outputting the reconstructed image after reconstruction by a second CNN, wherein the second CNN comprises a plurality of second convolutional layers connected sequentially to each other and a plurality of second composite layers each connected to and arranged between respective adjacent second convolutional layers, and each of the second composite layers is an up-sampling layer, wherein

the number of the first convolutional layers is identical to the number of the second convolutional layers,

an outputted image from the last one of the first convolutional layers is applied to the first one of the second convolutional layers,

apart from the first one of the plurality of second convolutional layers, at least one of the second convolutional layers is configured to receive the first convolutional feature outputted from the corresponding first convolutional layer, and

an output from the second composite layer at an identical level started from the first one of the second convolutional lavers is combined with the first convolutional feature outputted from the corresponding first convolutional layer to acquire a final output image data.

2. The image processing method according to claim 1, wherein apart from the first one of the plurality of second convolutional layers, each of the second convolutional layers is configured to receive the first convolutional feature outputted from the respective first convolutional layer, and the output from the second composite layer at the identical level started from the first one of the second convolutional lavers is combined with the first convolutional feature outputted from the respective first convolutional layer to acquire the final output image data.

3. The image processing method according to claim 1, wherein each of the second composite layers is configured to acquire input images inputted to the second composite layer, and interleave pixel values of the input images to generate output images, wherein the number of the acquired images is identical to the number of the generated output images.

4. The image processing method according to claim 1, further comprising:

extracting a feature of an inputted third image by a third CNN, wherein

the third CNN comprises a plurality of third convolutional layers connected sequentially to each other and a plurality of third pooling layers each connected to and arranged between respective adjacent third convolutional layers, and each of the third convolution layers is configured to generate and output a third convolutional feature;

the number of the third convolutional layers is identical to the number of the first convolutional layers; and

apart from the first one of the plurality of second convolutional layers and one of the second convolutional layers that is connected to the first convolutional layer, at least one of the second convolutional layers is configured to receive the third convolutional feature outputted from the respective third convolutional layer, and an output from the second composite layer at the identical level started from the first one of the second convolutional layers is combined with the third convolutional feature outputted from the respective third convolutional layer to acquire the final output image data.

5. The image processing method according to claim 4, further comprising:

selecting and enabling at least one of the plurality of second convolutional layers to receive the output from the respective first convolutional layer or the respective third convolutional layer, and thereby selecting and enabling the first convolutional feature outputted from the respective first convolutional layer or the third convolutional feature outputted from the respective third convolutional layer to be combined with the output from the second composite layer at the identical level started from the first one of the second convolutional layers to acquire the final output image data.

6. The image processing method according to claim 5, wherein

the first convolutional feature is a content feature and the third convolutional feature is a style feature;

or

the first convolutional feature is the style feature and the third convolutional feature is the content feature.

7. The image processing method according to claim 4, further comprising:

selecting and enabling each of the plurality of second convolutional layers to receive the output from the respective first convolutional layer or the respective third convolutional layer, and thereby selecting and enabling the first convolutional feature outputted from the respective first convolutional layer or the third convolutional feature outputted from the respective third convolutional layer to be combined with the output from the second composite layer at the identical level started from the first one of the second convolutional lavers to acquire the final output image data.

8. The image processing method according to claim 4, wherein a structure of the first CNN is identical to a structure of the third CNN.

9. An image processing device, comprising:

a first Convolutional Neural Network (CNN) circuit configured to extract one or more features of an inputted first image by a first CNN, the inputted first image being inputted to the first one of the first convolutional layers, wherein the first CNN comprises a plurality of first convolutional layers connected sequentially to each other and a plurality of first pooling layers each connected to and arranged between respective adjacent first convolutional layers, and each of the first convolutional layers is configured to generate and output a first convolutional feature; and

a second CNN circuit configured to reconstruct the inputted first image and output the reconstructed image after reconstruction by a second CNN, wherein the second CNN comprises a plurality of second convolutional layers connected sequentially to each other and a plurality of second composite layers each connected to and arranged between respective adjacent second convolutional layers, and each of the second composite layers is an up-sampling layer, wherein

the number of the first convolutional layers is identical to the number of the second convolutional layers,

an outputted image from the last one of the first convolutional layers is applied to the first one of the second convolutional layers,

apart from the first one of the plurality of second convolutional layers, at least one of the second convolutional layers is configured to receive the first convolutional feature outputted from the corresponding first convolutional layer, and

an output from the second composite layer at an identical level started from the first one of the second convolutional layers is combined with the first convolutional feature outputted from the corresponding first convolutional layer to acquire a final output image data.

10. The image processing device according to claim 9, wherein apart from the first one of the plurality of second convolutional layers, each of the second convolutional layers is configured to receive the first convolutional feature outputted from the respective first convolutional layer, and the output from the second composite layer at the identical level started from the first one of the second convolutional layers is combined with the first convolutional feature outputted from the respective first convolutional layer to acquire the final output image data.

11. The image processing device according to claim 9, wherein each of the second composite layers is configured to acquire input images inputted to the second composite layer, and interleave pixel values of the input images to generate output images, wherein the number of the acquired input images is identical to the number of generated output images.

12. The image processing device according to claim 9, further comprising a third CNN circuit configured to extract a feature of an inputted third image by a third CNN, wherein

the third CNN comprises a plurality of third convolutional layers connected sequentially to each other and a plurality of third pooling layers each connected to and arranged between respective adjacent third convolutional layers, and each of the third convolution layers is configured to generate and output a third convolutional feature;

the number of the third convolutional layers is identical to the number of the first convolutional layers; and

apart from the first one of the plurality of second convolutional layers and one of the second convolutional layers that is connected to the first convolutional layer, at least one of the second convolutional layers is configured to receive the third convolutional feature outputted from the respective third convolutional layer, and an output from the second composite layer at the identical level started from the first one of the second convolutional layers is combined with the third convolutional feature outputted from the respective third convolutional layer to acquire the final output image data.

13. The image processing device according to claim 12, further comprising a selection circuit configured to select and enable at least one of the plurality of second convolutional layers to receive the output from the respective first convolutional layer or the respective third convolutional layer, and thereby select and enable the first convolutional feature outputted from the respective first convolutional layer or the third convolutional feature outputted from the respective third convolutional layer to be combined with the output from the second composite layer at the identical level started the first one of the second convolutional layers to acquire the final output image data.

14. The image processing device according to claim 13, wherein the selection circuit is further configured to select and enable each of the plurality of second convolutional layers to receive the output from the respective first convolutional layer or the respective third convolutional layer, and thereby select and enable the first convolutional feature outputted from the respective first convolutional layer or the third convolutional feature outputted from the respective third convolutional layer to be combined with the output from the second composite layer at the identical level started from the first one of the second convolutional layers to acquire the final output images data.

15. The image processing device according to claim 12, wherein a structure of the first CNN is identical to a structure of the third CNN.

16. An image processing device, comprising a processor and a memory configured to store therein a computer program, wherein the processor is configured to execute the computer program to:

extract one or more features of an inputted first image by a first Convolutional Neural Network (CNN), the inputted first image being inputted to the first one of the first convolutional layers, wherein the first CNN comprises a plurality of first convolutional layers connected sequentially to each other and a plurality of first pooling layers each connected to and arranged between respective adjacent first convolutional layers, and each of the first convolutional layers is configured to generate and output a first convolutional feature; and

reconstruct the inputted first images and outputting the reconstructed image after reconstruction by a second CNN, wherein the second CNN comprises a plurality of second convolutional layers connected sequentially to each other and a plurality of second composite layers each connected to and arranged between respective adjacent second convolutional layers, and each of the second composite layers is an up-sampling layer, wherein

the number of the first convolutional layers is identical to the number of the second convolutional layers,

an outputted image from the last one of the first convolutional layers is applied to the first one of the second convolutional layers,

apart from the first one of the plurality of second convolutional layers, at least one of the second convolutional layers is configured to receive the first convolutional feature outputted from the corresponding first convolutional layer, and

an output from the second composite layer at an identical level started from the first one of the second convolutional layers is combined with the first convolutional feature outputted from the corresponding first convolutional layer to acquire a final output image data.

17. The image processing device according to claim 16, wherein apart from the first one of the plurality of second convolutional layers, each of the second convolutional layers is configured to receive the first convolutional feature outputted from the respective first convolutional layer, and the output from the second composite layer at the identical level started from the first one of the second convolutional layers is combined with the first convolutional feature outputted from the respective first convolutional layer to acquire the output data.

18. The image processing device according to claim 16, wherein each of the second composite layers is configured to acquire input images inputted to the second composite layer, and interleave pixel values of the input images to generate output images, wherein the number of the acquired input images is identical to the number of the generated output images.

19. The image processing device according to claim 16, wherein the processor is further configured to extract a feature of an inputted third image by a third CNN, wherein

the third CNN comprises a plurality of third convolutional layers connected sequentially to each other and a plurality of third pooling layers each connected to and arranged between respective adjacent third convolutional layers, and each of the third convolution layers is configured to generate and output a third convolutional feature;

the number of the third convolutional layers is identical to the number of the first convolutional layers; and

apart from the first one of the plurality of second convolutional layers and one of the second convolutional layers that is connected to the first convolutional layer, at least one of the second convolutional layers is configured to receive the third convolutional feature outputted from the respective third convolutional layer, and an output from the second composite layer at the identical level started from the first one of the second convolutional layers is combined with the third convolutional feature outputted from the respective third convolutional layer to acquire the final output image data.

20. The image processing device according to claim 19, wherein the processor is further configured to select and enable at least one of the plurality of second convolutional layers to receive the output from the respective first convolutional layer or the respective third convolutional layer, and thereby select and enable the first convolutional feature outputted from the respective first convolutional layer or the third convolutional feature outputted from the respective third convolutional layer to be combined with the output from the second composite layer at the identical level started from the first one of the second convolutional layers to acquire the final output image data.