Real-time synthetically generated video from still frames
First Claim
1. A system for generating synthetic video, the system comprising:
- one or more memory units for storing instructions; and
one or more processors configured to execute the instructions to perform operations comprising;
receiving video data comprising a sequence of image frames;
normalizing the sequence of image frames;
generating a sequence of difference images based on the normalized sequence of image frames;
generating an image sequence generator model based on the sequence of difference images;
training an autoencoder model using the sequence of difference images, the autoencoder model comprising an encoder model and a decoder model;
identifying a seed image frame;
generating a seed difference image based on the seed image frame by using the encoder model to perform a forward-step encoding on the seed image frame;
generating a sequence of synthetic difference images based on the seed difference image, wherein the sequence of synthetic difference images is generated by iteratively using the image sequence generator model to accept a previous synthetic difference image in the sequence as an input and return a subsequent synthetic difference image as an output, starting from the seed difference image;
generating a sequence of synthetic normalized image frames based on the sequence of synthetic difference images, by using the decoder model to perform at least one of forward-step or backward-step decoding; and
generating synthetic video by adding background to the synthetic normalized image frames.
1 Assignment
0 Petitions
Accused Products
Abstract
Systems and methods for generating synthetic video are disclosed. For example, a system may include a memory unit and a processor configured to execute the instructions to perform operations. The operations may include receiving video data, normalizing image frames, generating difference images, and generating an image sequence generator model. The operations may include training an autoencoder model using difference images, the autoencoder comprising an encoder model and a decoder model. The operations may include identifying a seed image frame and generating a seed difference image from the seed image frame. The operations may include generating, by the image sequence generator model, synthetic difference images based on the seed difference image. In some aspects, the operations may include using the decoder model to synthetic normalized image frames from the synthetic difference images. The operations may include generating synthetic video by adding background to the synthetic normalized image frames.
-
Citations
21 Claims
-
1. A system for generating synthetic video, the system comprising:
-
one or more memory units for storing instructions; and one or more processors configured to execute the instructions to perform operations comprising; receiving video data comprising a sequence of image frames; normalizing the sequence of image frames; generating a sequence of difference images based on the normalized sequence of image frames; generating an image sequence generator model based on the sequence of difference images; training an autoencoder model using the sequence of difference images, the autoencoder model comprising an encoder model and a decoder model; identifying a seed image frame; generating a seed difference image based on the seed image frame by using the encoder model to perform a forward-step encoding on the seed image frame; generating a sequence of synthetic difference images based on the seed difference image, wherein the sequence of synthetic difference images is generated by iteratively using the image sequence generator model to accept a previous synthetic difference image in the sequence as an input and return a subsequent synthetic difference image as an output, starting from the seed difference image; generating a sequence of synthetic normalized image frames based on the sequence of synthetic difference images, by using the decoder model to perform at least one of forward-step or backward-step decoding; and generating synthetic video by adding background to the synthetic normalized image frames. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19)
-
-
20. A method for generating synthetic video, the method comprising:
-
receiving video data comprising a sequence of image frames; normalizing the sequence of image frames; generating a sequence of difference images based on the normalized sequence of image frames; generating an image sequence generator model based on the sequence of difference images; training an autoencoder model using the sequence of difference images, the autoencoder model comprising an encoder model and a decoder model; identifying a seed image frame; generating a seed difference image based on the seed image frame by using the encoder model to perform a forward-step encoding on the seed image frame; generating a sequence of synthetic difference images based on the seed difference image, wherein the sequence of synthetic difference images is generated by iteratively using the image sequence generator model to accept a previous synthetic difference image in the sequence as an input and return a subsequent synthetic difference image as an output, starting from the seed difference image; generating a sequence of synthetic normalized image frames based on the sequence of synthetic difference images, by using the decoder model to perform at least one of forward-step or backward-step decoding; and generating synthetic video by adding background to the synthetic normalized image frames.
-
-
21. A system for generating synthetic video, the system comprising:
-
one or more memory units for storing instructions; and one or more processors configured to execute the instructions to perform operations comprising; receiving video data comprising a sequence of image frames; normalizing the sequence of image frames; generating a sequence of difference images based on the normalized sequence of image frames; generating an image sequence generator model based on the sequence of difference images; training an autoencoder model using the sequence of difference images, the autoencoder model comprising an encoder model and a decoder model; generating a seed difference image using a neural network model; generating a sequence of synthetic difference images based on the seed difference image, wherein the sequence of synthetic difference images is generated by iteratively using the image sequence generator model to accept a previous synthetic difference image in the sequence as an input and return a subsequent synthetic difference image as an output, starting from the seed difference image; generating a sequence of synthetic normalized image frames based on the sequence of synthetic difference images, by using the decoder model to perform at least one of forward-step or backward-step decoding; and generating synthetic video by adding background to the synthetic normalized image frames.
-
Specification