Real-time selection of DNN style transfer networks from DNN sets

• US 10,664,963 B1
• Filed: 07/11/2018
• Issued: 05/26/2020
• Est. Priority Date: 09/11/2017
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Rossi, Francesco, Wang, Xiaohuan C., Rymkowski, Bartlomiej W., Shi, Xiaojin, Zuliani, Marco, Marinichev, Alexey
• Primary Examiner(s): Mariam, Daniel G
• Application Number:
  US16/032,879
• Time in Patent Office: 685 Days
• Field of Search: 382156, 382284
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/046 Forward inferencing; Production systems
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 2207/20028 Bilateral filtering
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 3/4046 using neural networks
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks

Abstract

Artistic styles extracted from one or more source images may be applied to one or more target images, e.g., in the form of stylized images and/or stylized video sequences. The extracted artistic style may be stored as a plurality of layers in a neural network, which neural network may be further optimized, e.g., via the fusion of various elements of the network'"'"'s architectures. An optimized network architecture may be determined for each processing environment in which the network will be applied. The artistic style may be applied to the obtained images and/or video sequence of images using various optimization methods, such as the use of scalars to control the resolution of the unstylized and stylized images, temporal consistency constraints, as well as the use of dynamically adjustable or selectable versions of Deep Neural Networks (DNN) that are responsive to system performance parameters, such as available processing resources and thermal capacity.

Claims

1. An image processing method, comprising:

obtaining a first set of neural networks for applying a first artistic style, wherein each neural network in the first set comprises a plurality of layers, and wherein each neural network in the first set is configured to operate under a set of device performance parameters;

obtaining a first set of device performance parameters for a first device at a first time;

obtaining a first target image captured at a second time that is later than the first time, wherein the first target image has a first resolution, and wherein the first target image was captured in a first capture mode;

selecting a first one of the neural networks from the first set based, at least in part, on the;

first resolution, the first capture mode, and the first set of device performance parameters;

applying the selected first neural network to the first target image to create a stylized version of the first target image; and

storing the stylized version of the first target image in a memory.

2. The method of claim 1, further comprising:

obtaining a second target image captured at a third time that is later than the second time, wherein the second target image has a second resolution, wherein the second resolution is different than the first resolution, and wherein the second target image was captured in a second capture mode;

selecting a second one of the neural networks from the first set based, at least in part, on the;

second resolution, the second capture mode, and the first set of device performance parameters, wherein the second neural network is different from the first neural network;

applying the selected second neural network to the second target image to create a stylized version of the second target image; and

storing the stylized version of the second target image in the memory.

3. The method of claim 1, further comprising:

obtaining a second set of device performance parameters for the first device at a third time that is later than the second time, wherein the second set of device performance parameters are different than the first set of device performance parameters;

obtaining a second target image captured at a fourth time that is later than the third time, wherein the second target image has a second resolution, and wherein the second target image was captured in a second capture mode;

selecting a second one of the neural networks from the first set based, at least in part, on the;

second resolution, the second capture mode, and the second set of device performance parameters, wherein the second neural network is different from the first neural network;

applying the selected second neural network to the second target image to create a stylized version of the second target image; and

storing the stylized version of the second target image in the memory.

4. The method of claim 1, wherein:

the first set of device performance parameters comprises a measurement of at least one of the following;

a device power state;

a device processing load state;

a device battery state; and

a device thermal state.

5. The method of claim 1, wherein:

selecting a first one of the neural networks comprises consulting a pre-stored look-up table on the first device.

6. The method of claim 1, wherein each neural network in the first set is configured to operate under a different set of device performance parameters.

7. The method of claim 1, further comprising:

modifying one or more parameters of the selected first neural network before applying the selected first neural network to the first target image.

8. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:

obtain a first set of neural networks for applying a first artistic style, wherein each neural network in the first set comprises a plurality of layers, and wherein each neural network in the first set is configured to operate under a set of device performance parameters;

obtain a first set of device performance parameters for a first device at a first time;

obtain a first target image captured at a second time that is later than the first time, wherein the first target image has a first resolution, and wherein the first target image was captured in a first capture mode;

select a first one of the neural networks from the first set based, at least in part, on the;

first resolution, the first capture mode, and the first set of device performance parameters;

apply the selected first neural network to the first target image to create a stylized version of the first target image; and

store the stylized version of the first target image in a memory.

9. The non-transitory program storage device of claim 8, further comprising instructions stored thereon to cause one or more processors to:

obtain a second target image captured at a third time that is later than the second time, wherein the second target image has a second resolution, wherein the second resolution is different than the first resolution, and wherein the second target image was captured in a second capture mode;

select a second one of the neural networks from the first set based, at least in part, on the;

second resolution, the second capture mode, and the first set of device performance parameters, wherein the second neural network is different from the first neural network;

apply the selected second neural network to the second target image to create a stylized version of the second target image; and

store the stylized version of the second target image in the memory.

10. The non-transitory program storage device of claim 8, further comprising instructions stored thereon to cause one or more processors to:

obtain a second set of device performance parameters for the first device at a third time that is later than the second time, wherein the second set of device performance parameters are different than the first set of device performance parameters;

obtain a second target image captured at a fourth time that is later than the third time, wherein the second target image has a second resolution, and wherein the second target image was captured in a second capture mode;

select a second one of the neural networks from the first set based, at least in part, on the;

second resolution, the second capture mode, and the second set of device performance parameters, wherein the second neural network is different from the first neural network;

apply the selected second neural network to the second target image to create a stylized version of the second target image; and

store the stylized version of the second target image in the memory.

11. The non-transitory program storage device of claim 8, wherein the first set of device performance parameters comprises a measurement of at least one of the following:

a device power state;

a device processing load state;

a device battery state; and

a device thermal state.

12. The non-transitory program storage device of claim 8, wherein:

the instructions to select a first one of the neural networks further comprise instructions to consult a pre-stored look-up table on the first device.

13. The non-transitory program storage device of claim 8, wherein each neural network in the first set is configured to operate under a different set of device performance parameters.

14. A device, comprising:

an image sensor;

a display screen;

a memory communicatively coupled to the image sensor;

one or more processors operatively coupled to the image sensor and the memory configured to execute instructions causing the one or more processors to;

obtain a first set of neural networks for applying a first artistic style, wherein each neural network in the first set comprises a plurality of layers, and wherein each neural network in the first set is configured to operate under a set of device performance parameters;

obtain a first set of device performance parameters for the device at a first time;

obtain a first target image captured at a second time that is later than the first time, wherein the first target image has a first resolution, and wherein the first target image was captured in a first capture mode;

select a first one of the neural networks from the first set based, at least in part, on the;

first resolution, the first capture mode, and the first set of device performance parameters;

apply the selected first neural network to the first target image to create a stylized version of the first target image; and

store the stylized version of the first target image in the memory.

15. The device of claim 14, wherein the one or more processors are further configured to execute instructions causing the one or more processors to:

obtain a second target image captured at a third time that is later than the second time, wherein the second target image has a second resolution, wherein the second resolution is different than the first resolution, and wherein the second target image was captured in a second capture mode;

select a second one of the neural networks from the first set based, at least in part, on the;

second resolution, the second capture mode, and the first set of device performance parameters, wherein the second neural network is different from the first neural network;

apply the selected second neural network to the second target image to create a stylized version of the second target image; and

store the stylized version of the second target image in the memory.

16. The device of claim 14, wherein the one or more processors are further configured to execute instructions causing the one or more processors to:

obtain a second set of device performance parameters for the device at a third time that is later than the second time, wherein the second set of device performance parameters are different than the first set of device performance parameters;

obtain a second target image captured at a fourth time that is later than the third time, wherein the second target image has a second resolution, and wherein the second target image was captured in a second capture mode;

select a second one of the neural networks from the first set based, at least in part, on the;

second resolution, the second capture mode, and the second set of device performance parameters, wherein the second neural network is different from the first neural network;

apply the selected second neural network to the second target image to create a stylized version of the second target image; and

store the stylized version of the second target image in the memory.

17. The device of claim 14, wherein the first set of device performance parameters comprises a measurement of at least one of the following:

a device power state;

a device processing load state;

a device battery state; and

a device thermal state.

18. The device of claim 14, wherein:

the instructions to select a first one of the neural networks further comprise instructions to consult a pre-stored look-up table on the device.

19. The device of claim 14, wherein each neural network in the first set is configured to operate under a different set of device performance parameters.

20. The device of claim 14, wherein the one or more processors are further configured to execute instructions causing the one or more processors to:

modify one or more parameters of the selected first neural network before applying the selected first neural network to the first target image.