Transferring motion between consecutive frames to a digital image

• US 10,445,921 B1
• Filed: 06/13/2018
• Issued: 10/15/2019
• Est. Priority Date: 06/13/2018
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Inc.
• Inventors: Li, Yijun, Fang, Chen, Yang, Jimei, Wang, Zhaowen, Lu, Xin
• Primary Examiner(s): Broome, Said
• Application Number:
  US16/007,898
• Time in Patent Office: 489 Days
• Field of Search: None
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation

Abstract

Transferring motion between consecutive frames to a digital image is leveraged in a digital medium environment. A digital image and at least a portion of the digital video are exposed to a motion transfer model. The portion of the digital video includes at least a first digital video frame and a second digital video frame that is consecutive to the first digital video frame. Flow data between the first digital video frame and the second digital image frame is extracted, and the flow data is then processed to generate motion features representing motion between the first digital video frame and the second digital video frame. The digital image is processed to generate image features of the digital image. Motion of the digital video is then transferred to the digital image by combining the motion features with the image features to generate a next digital image frame for the digital image.

Claims

1. A method implemented by at least one computing device, to transfer motion from a digital video to a digital image, the method comprising:

exposing, by the at least one computing device, the digital image and at least a portion of the digital video to a motion transfer model that combines an image feature extractor, a flow estimator, and a motion feature extractor, the portion of the digital video comprising at least a first digital video frame and a second digital video frame that is consecutive to the first digital video frame;

extracting, by the flow estimator, flow data between the first digital video frame and the second digital video frame;

processing, by the motion feature extractor, the flow data to generate motion features representing motion between the first digital video frame and the second digital video frame;

processing, by the image feature extractor, the digital image to generate image features of the digital image; and

transferring motion of the digital video to the digital image by combining the motion features with the image features to generate a next digital image frame for the digital image.

2. The method as described in claim 1, wherein the combining comprises:

concatenating the motion features with the image features to generate concatenated features in a high dimensional feature space; and

decoding, by a decoder, the concatenated features to generate the next digital image frame for the digital video in a pixel space.

3. The method as described in claim 1, further comprising combining the digital image with the next digital image frame to form an animated sequence of digital images.

4. The method as described in claim 3, wherein the animated sequence of digital images comprises a GIF.

5. The method as described in claim 1, wherein the flow data does not include content of the digital video.

6. The method as described in claim 1, wherein the image feature extractor is implemented as a first neural network and the motion feature extractor is implemented as a second neural network.

7. A method as described in claim 1, wherein the digital video includes content that is visually similar to the digital image.

8. The method as described in claim 1, further comprising generating an additional next digital image frame for the digital image by:

extracting, by the flow estimator, additional flow data between the second digital video frame and a third digital video frame;

processing, by the motion feature extractor, the additional flow data to generate additional motion features representing motion between the second digital video frame and the third digital video frame;

processing, by the image feature extractor, the generated next digital image frame to generate image features of the next digital image frame; and

combining the additional motion features with the additional image features to generate the additional next digital image frame for the digital image.

9. The method as described in claim 1, wherein the motion transfer model is trained, using machine learning, to transfer motion between two consecutive frames of a reference digital video to a still frame of the same reference video.

17. The system of claim 16 as described in claim 1, wherein the combining the motion features with the image features comprises:

concatenating the motion features with the image features to generate concatenated features in a high dimensional feature space; and

decoding, by a decoder, the concatenated features to generate the next digital image frame for the digital video in a pixel space.

19. The method as described in claim 1, wherein the flow data does not include content of the digital video.

20. The method as described in claim 1, wherein the image feature extractor is implemented as a first neural network and the motion feature extractor is implemented as a second neural network.

10. A method implemented by at least one computing device to generate a next digital image frame from at least two initial digital image frames, the method comprising:

exposing, by the at least one computing device, a first digital image frame and a second digital image frame that is consecutive to the first digital image frame to a motion transfer model that combines a motion predictor and a motion transferor;

generating, by the motion predictor and based on the first digital image frame and the second digital image frame, predicted motion between the second digital image frame and a next digital image frame; and

transferring, by the motion transferor, the predicted motion to the second digital image frame to generate the next digital image frame.

11. The method as described in claim 10, wherein the predicted motion is transferred to the second digital image frame to generate the next digital image frame by:

extracting, by a flow estimator of the motion predictor, flow data between the first digital video frame and the second digital image frame;

processing, by a first encoder of the motion predictor, the flow data to generate motion features representing motion between the first digital video frame and the second digital video frame;

processing, by a second encoder of the motion predictor, the second digital image frame to generate image features of the second digital image; and

generating the predicted motion between the second digital image frame and the next digital image frame by combining the motion features with the image features.

12. The method as described in claim 11, wherein the predicted motion is transferred to the second digital image frame to generate the third digital image frame by:

processing, by a third encoder of the motion transferor, the predicted motion between the second digital image frame and the third digital image frame to generate motion features representing motion between the second digital video frame and the third digital video frame;

processing, by a fourth encoder of the motion transferor, the second digital image frame to generate image features of the second digital image frame; and

generating the next digital image frame by combining the motion features with the image features of the second digital image frame.

13. The method as described in claim 12, wherein the combining the motion features with the image features of the second digital image frame to generate the next digital image frame comprises:

concatenating the motion features with the image features to generate concatenated features in a high dimensional feature space; and

decoding, by a decoder, the concatenated features to generate the next digital image frame in a pixel space.

14. The method as described in claim 10, wherein the first digital image frame and the second digital image frames comprise a portion of a digital video.

15. The method as described in claim 10, wherein the first digital image frame and the second digital image frames comprise at least a portion of a GIF.

16. A system comprising:

a motion transfer model that combines an image feature extractor, a flow estimator, and a motion feature extractor;

at least one processor; and

memory having stored thereon computer-readable instructions that are executable by the at least one processor to perform operations comprising;

exposing a digital image and at least a portion of a digital video to the motion transfer model, the portion of the digital video comprising at least a first digital video frame and a second digital video frame that is consecutive to the first digital video frame;

extracting, by the flow estimator, flow data between the first digital video frame and the second digital video frame;

processing, by the motion feature extractor, the flow data to generate motion features representing motion between the first digital video frame and the second digital video frame;

processing, by the image feature extractor, the digital image to generate image features of the digital image;

transferring motion of the digital video to the digital image by combining the motion features with the image features to generate a next digital image frame for the digital image; and

forming an animated sequence of digital images by combining the digital image with the next digital image frame in sequence.

18. The system as described in claim 16, wherein the animated sequence of digital images comprises a GIF.