Real-time adjustment of hybrid DNN style transfer networks

• US 10,664,718 B1
• Filed: 07/11/2018
• Issued: 05/26/2020
• Est. Priority Date: 09/11/2017
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Rymkowski, Bartlomiej W., Rossi, Francesco
• Primary Examiner(s): Ahmed, Samir A
• Application Number:
  US16/032,909
• Time in Patent Office: 685 Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/046 Forward inferencing; Production systems
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 2207/20028 Bilateral filtering
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 3/4046 using neural networks
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks

Abstract

Artistic styles extracted from one or more source images may be applied to one or more target images, e.g., in the form of stylized images and/or stylized video sequences. The extracted artistic style may be stored as a plurality of layers in a neural network, which neural network may be further optimized, e.g., via the fusion of various elements of the network'"'"'s architectures. An optimized network architecture may be determined for each processing environment in which the network will be applied. The artistic style may be applied to the obtained images and/or video sequence of images using various optimization methods, such as the use of scalars to control the resolution of the unstylized and stylized images, temporal consistency constraints, as well as the use of dynamically adjustable or selectable versions of Deep Neural Networks (DNN) that are responsive to system performance parameters, such as available processing resources and thermal capacity.

Claims

1. An image processing method, comprising:

obtaining a first artistic style, wherein the artistic style is stored as a plurality of layers in a selected first neural network, and wherein the selected first neural network is configured to operate on images having multiple different resolutions;

obtaining a first set of device performance parameters for a first device at a first time;

obtaining a first target image captured at a second time that is later than the first time, wherein the first target image has a first resolution, and wherein the first target image was captured in a first capture mode;

scaling the first target image based on a configuration of selected first neural network to generate a first scaled target image;

applying at least a first part of the selected first neural network to the first target image to create a stylized version of the first target image;

applying at least a second part of the selected first neural network to the first scaled target image to create a stylized version of the first scaled target image;

combining the stylized versions of the first target image and the first scaled target image to generate an output stylized image; and

storing the output stylized version of the first target image in a memory.

2. The method of claim 1, further comprising:

obtaining a second set of device performance parameters for the first device at a third time that is later than the second time; and

modifying at least the second part of the selected first neural network based, at least in part, on the second set of device performance parameters.

3. The method of claim 2, further comprising:

obtaining a third set of device performance parameters for the first device at a fourth time that is later than the third time; and

modifying at least the first part of the selected first neural network based, at least in part, on the third set of device performance parameters.

4. The method of claim 1, wherein:

the first set of device performance parameters comprises a measurement of at least one of the following;

a device power state;

a device processing load state;

a device battery state; and

a device thermal state.

5. The method of claim 1, wherein:

the first part of the selected first neural network comprises a first number of convolutional layers; and

the second part of the selected first neural network comprises a second number of convolutional layers that is greater than the first number of convolutional layers.

6. The method of claim 1, wherein generating the output stylized image further comprises:

blending the combined stylized versions of the first target image and the first scaled target image with the first target image.

7. The method of claim 1, wherein the first capture mode comprises at least one of:

an image preview mode or an image record mode.

8. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:

obtain a first artistic style, wherein the artistic style is stored as a plurality of layers in a selected first neural network, and wherein the selected first neural network is configured to operate on images having multiple different resolutions;

obtain a first set of device performance parameters for a first device at a first time;

obtain a first target image captured at a second time that is later than the first time, wherein the first target image has a first resolution, and wherein the first target image was captured in a first capture mode;

scale the first target image based on a configuration of selected first neural network to generate a first scaled target image;

apply at least a first part of the selected first neural network to the first target image to create a stylized version of the first target image;

apply at least a second part of the selected first neural network to the first scaled target image to create a stylized version of the first scaled target image;

combine the stylized versions of the first target image and the first scaled target image to generate an output stylized image; and

store the output stylized version of the first target image in a memory.

9. The non-transitory program storage device of claim 8, further comprising instructions stored thereon to cause one or more processors to:

obtain a second set of device performance parameters for the first device at a third time that is later than the second time; and

modify at least the second part of the selected first neural network based, at least in part, on the second set of device performance parameters.

10. The non-transitory program storage device of claim 9, further comprising instructions stored thereon to cause one or more processors to:

obtain a third set of device performance parameters for the first device at a fourth time that is later than the third time; and

modify at least the first part of the selected first neural network based, at least in part, on the third set of device performance parameters.

11. The non-transitory program storage device of claim 8, wherein:

the first set of device performance parameters comprises a measurement of at least one of the following;

a device power state;

a device processing load state;

a device battery state; and

a device thermal state.

12. The non-transitory program storage device of claim 8, wherein:

the first part of the selected first neural network comprises a first number of convolutional layers; and

the second part of the selected first neural network comprises a second number of convolutional layers that is greater than the first number of convolutional layers.

13. The non-transitory program storage device of claim 8, wherein the instructions to cause the one or more processors to generate the output stylized image further comprise instructions to:

blend the combined stylized versions of the first target image and the first scaled target image with the first target image.

14. A device, comprising:

an image sensor;

a display screen;

a memory communicatively coupled to the image sensor;

one or more processors operatively coupled to the image sensor and the memory configured to execute instructions causing the one or more processors to;

obtain a first artistic style, wherein the artistic style is stored as a plurality of layers in a selected first neural network, and wherein the selected first neural network is configured to operate on images having multiple different resolutions;

obtain a first set of device performance parameters for the device at a first time;

obtain a first target image captured at a second time that is later than the first time, wherein the first target image has a first resolution, and wherein the first target image was captured in a first capture mode;

scale the first target image based on a configuration of selected first neural network to generate a first scaled target image;

apply at least a first part of the selected first neural network to the first target image to create a stylized version of the first target image;

apply at least a second part of the selected first neural network to the first scaled target image to create a stylized version of the first scaled target image;

combine the stylized versions of the first target image and the first scaled target image to generate an output stylized image; and

store the output stylized version of the first target image in the memory.

15. The device of claim 14, wherein the one or more processors are further configured to execute instructions causing the one or more processors to:

obtain a second set of device performance parameters for the device at a third time that is later than the second time; and

modify at least the second part of the selected first neural network based, at least in part, on the second set of device performance parameters.

16. The device of claim 15, wherein the one or more processors are further configured to execute instructions causing the one or more processors to:

obtain a third set of device performance parameters for the device at a fourth time that is later than the third time; and

modify at least the first part of the selected first neural network based, at least in part, on the third set of device performance parameters.

17. The device of claim 15, wherein:

the first set of device performance parameters comprises a measurement of at least one of the following;

a device power state;

a device processing load state;

a device battery state; and

a device thermal state.

18. The device of claim 14, wherein:

the first part of the selected first neural network comprises a first number of convolutional layers; and

the second part of the selected first neural network comprises a second number of convolutional layers that is greater than the first number of convolutional layers.

19. The device of claim 14, wherein the one or more processors are further configured to execute instructions causing the one or more processors to:

blend the combined stylized versions of the first target image and the first scaled target image with the first target image.

20. The device of claim 14, wherein the first capture mode comprises at least one of:

an image preview mode or an image record mode.