Style transfer-based image content correction

• US 10,198,839 B2
• Filed: 09/22/2016
• Issued: 02/05/2019
• Est. Priority Date: 09/22/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Rymkowski, Bartlomiej W., Zuliani, Marco
• Primary Examiner(s): Chawan, Sheela C
• Application Number:
  US15/273,674
• Publication Number:
  US 20180082407A1
• Time in Patent Office: 866 Days
• Field of Search: 382275, 382266, 382254, 382276, 382284, 382181, 382190, 382195, 382199, 382173, 345418, 345441, 345442, 345581, 345589, 345419, 345426, 345582, 345588, 345467, 345470, 345619, 345629, 345634, 345639, 345593, 434 81, 434 84, 434 94, 434365, 434371
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text

Abstract

Techniques are disclosed herein for applying an artistic style extracted from one or more source images, e.g., paintings, to one or more target images. The extracted artistic style may then be stored as a plurality of layers in a neural network. In some embodiments, two or more stylized target images may be combined and stored as a stylized video sequence. The artistic style may be applied to the target images in the stylized video sequence using various optimization methods and/or pixel- and feature-based regularization techniques in a way that prevents excessive content pixel fluctuations between images and preserves smoothness in the assembled stylized video sequence. In other embodiments, a user may be able to semantically annotate locations of undesired artifacts in a target image, as well as portion(s) of a source image from which a style may be extracted and used to replace the undesired artifacts in the target image.

Claims

1. An image processing method, comprising:

obtaining a first source image;

extracting an artistic style from at least a portion of the first source image, wherein the extracted artistic style is stored as a plurality of layers in a convolutional neural network;

obtaining a first target image comprising one or more undesired artifacts;

obtaining semantic information from the first target image corresponding to the one or more undesired artifacts;

applying, using the plurality of layers stored in the convolutional neural network, the extracted artistic style to the first target image in accordance with the semantic information to repair the one or more undesired artifacts, thereby creating a content corrected version of the first target image; and

storing the content corrected version of the first target image in a memory.

2. The method of claim 1, wherein:

the first source image and the first target image are the same image.

3. The method of claim 2, wherein:

the at least a portion of the first source image does not contain any of the one or more undesired artifacts.

4. The method of claim 1, wherein obtaining the semantic information further comprises:

obtaining an indication of a location within the first target image of each of the one or more undesired artifacts.

5. The method of claim 1, wherein:

applying the extracted artistic style to the first target image comprises utilizing an optimization process.

6. The method of claim 1, further comprising:

obtaining an indication of a location of the at least a portion of the first source image from which the artistic style is extracted.

7. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:

obtain a first source image;

extract an artistic style from at least a portion of the first source image, wherein the extracted artistic style is stored as a plurality of layers in a convolutional neural network;

obtain a first target image comprising one or more undesired artifacts;

obtain semantic information from the first target image corresponding to the one or more undesired artifacts;

apply, using the plurality of layers stored in the convolutional neural network, the extracted artistic style to the first target image in accordance with the semantic information to repair the one or more undesired artifacts, thereby creating a content corrected version of the first target image; and

store the content corrected version of the first target image in a memory.

8. The non-transitory program storage device of claim 7, wherein:

the first source image and the first target image are the same image.

9. The non-transitory program storage device of claim 7, wherein the instructions to obtain the semantic information further comprise instructions to:

obtain an indication of a location within the first target image of each of the one or more undesired artifacts.

10. The non-transitory program storage device of claim 7, wherein the instructions to apply the artistic style to the first target image further comprise instructions to:

utilize an optimization process.

11. The non-transitory program storage device of claim 10, further comprising instructions stored thereon to cause the one or more processors to:

obtain an indication of a location of the at least a portion of the first source image from which the artistic style is extracted.

12. A device, comprising:

a memory;

a user interface;

one or more processors operatively coupled to the memory and the user interface, wherein the one or more processors are configured to execute instructions causing the one or more processors to;

obtain a first source image;

extract an artistic style from at least a portion of the first source image, wherein the extracted artistic style is stored as a plurality of layers in a convolutional neural network;

obtain a first target image comprising one or more undesired artifacts;

obtain semantic information from the first target image corresponding to the one or more undesired artifacts;

apply, using the plurality of layers stored in the convolutional neural network, the extracted artistic style to the first target image in accordance with the semantic information to repair the one or more undesired artifacts, thereby creating a content corrected version of the first target image; and

store the content corrected version of the first target image in the memory.

13. The device of claim 12, wherein:

the first source image and the first target image are the same image.

14. The device of claim 13, wherein:

the at least a portion of the first source image does not contain any of the one or more undesired artifacts.

15. The device of claim 12, wherein the instructions to obtain the semantic information further comprise instructions to:

obtain an indication of a location within the first target image of each of the one or more undesired artifacts.

16. The device of claim 12, wherein the instructions to apply the artistic style to the first target image further comprise instructions to:

utilize an optimization process.

17. The device of claim 16, wherein the one or more processors are further configured to execute instructions causing the one or more processors to:

obtain an indication of a location of the at least a portion of the first source image from which the artistic style is extracted.