ARTISTIC STYLE TRANSFER FOR VIDEOS

• US 20180082715A1
• Filed: 09/22/2016
• Published: 03/22/2018
• Est. Priority Date: 09/22/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Rymkowski, Bartlomiej W., Zuliani, Marco
• Granted Patent: US 10,147,459 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals

Abstract

Techniques are disclosed herein for applying an artistic style extracted from one or more source images, e.g., paintings, to one or more target images. The extracted artistic style may then be stored as a plurality of layers in a neural network. In some embodiments, two or more stylized target images may be combined and stored as a stylized video sequence. The artistic style may be applied to the target images in the stylized video sequence using various optimization methods and/or pixel- and feature-based regularization techniques in a way that prevents excessive content pixel fluctuations between images and preserves smoothness in the assembled stylized video sequence. In other embodiments, a user may be able to semantically annotate locations of undesired artifacts in a target image, as well as portion(s) of a source image from which a style may be extracted and used to replace the undesired artifacts in the target image.

Claims

1. An image processing method, comprising:

obtaining a first source image;

extracting an artistic style from the first source image;

obtaining a first target image captured at a first time;

applying the artistic style to the first target image to create a stylized version of the first target image;

obtaining a second target image captured at a second time;

applying the artistic style to the second target image to create a stylized version of the second target image;

combining the stylized version of the first target image and the stylized version of the second target image to create a first stylized video sequence; and

storing the stylized video sequence in a memory.

2. The method of claim 1, wherein:

the artistic style is stored as a plurality of layers in a neural network.

3. The method of claim 1, further comprising:

determining a scene classification for the first target image,

wherein the first source image is obtained based on the determination of the scene classification for the first target image.

4. The method of claim 1, wherein:

applying the artistic style to the second target image comprises initializing an optimization process using the stylized version of the first target image.

5. The method of claim 1, wherein:

applying the artistic style to the second target image comprises initializing an optimization process using an interpolation of stylized versions of one or more images captured prior to the second target image and an initial stylized version of the second target image.

6. The method of claim 1, wherein:

applying the artistic style to the second target image further comprises initializing an optimization process using an interpolation of stylized versions of one or more images captured prior to the second target image, stylized versions of one or more images captured subsequently to the second target image, and an initial stylized version of the second target image.

7. The method of claim 1, wherein:

the first source image comprises an image of a painting.

8. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:

obtain a first source image;

extract an artistic style from the first source image;

obtain a first target image captured at a first time;

apply the artistic style to the first target image to create a stylized version of the first target image;

obtain a second target image captured at a second time;

apply the artistic style to the second target image to create a stylized version of the second target image;

combine the stylized version of the first target image and the stylized version of the second target image to create a first stylized video sequence; and

store the stylized video sequence in a memory.

9. The non-transitory program storage device of claim 8, wherein:

the artistic style is stored as a plurality of layers in a neural network.

10. The non-transitory program storage device of claim 8, further comprising instructions stored thereon to cause the one or more processors to:

determine a scene classification for the first target image,

wherein the first source image is obtained based on the determination of the scene classification for the first target image.

11. The non-transitory program storage device of claim 8, wherein the instructions to apply the artistic style to the second target image further comprise instructions to:

initialize an optimization process using the stylized version of the first target image.

12. The non-transitory program storage device of claim 8, wherein the instructions to apply the artistic style to the second target image further comprise instructions to:

initialize an optimization process using an interpolation of stylized versions of one or more images captured prior to the second target image and an initial stylized version of the second target image.

13. The non-transitory program storage device of claim 8, wherein the instructions to apply the artistic style to the second target image further comprise instructions to:

initialize an optimization process using an interpolation of stylized versions of one or more images captured prior to the second target image, stylized versions of one or more images captured subsequently to the second target image, and an initial stylized version of the second target image.

14. A device, comprising:

a memory;

one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions causing the one or more processors to;

obtain a first source image;

extract an artistic style from the first source image;

obtain a first target image captured at a first time;

apply the artistic style to the first target image to create a stylized version of the first target image;

obtain a second target image captured at a second time;

apply the artistic style to the second target image to create a stylized version of the second target image;

combine the stylized version of the first target image and the stylized version of the second target image to create a first stylized video sequence; and

store the stylized video sequence in the memory.

15. The device of claim 14, wherein:

the artistic style is stored as a plurality of layers in a neural network.

16. The device of claim 14, wherein the one or more processors are further configured to execute instructions causing the one or more processors to:

determine a scene classification for the first target image,

wherein the first source image is obtained based on the determination of the scene classification for the first target image.

17. The device of claim 14, wherein the instructions to apply the artistic style to the second target image further comprise instructions to:

initialize an optimization process using the stylized version of the first target image.

18. The device of claim 14, wherein the instructions to apply the artistic style to the second target image further comprise instructions to:

initialize an optimization process using an interpolation of stylized versions of one or more images captured prior to the second target image and an initial stylized version of the second target image.

19. The device of claim 14, wherein the instructions to apply the artistic style to the second target image further comprise instructions to:

initialize an optimization process using an interpolation of stylized versions of one or more images captured prior to the second target image, stylized versions of one or more images captured subsequently to the second target image, and an initial stylized version of the second target image.

20. The device of claim 14, wherein:

the first source image comprises an image of a painting.