Video frame interpolation using a convolutional neural network

• US 10,491,856 B2
• Filed: 05/08/2018
• Issued: 11/26/2019
• Est. Priority Date: 03/15/2018
• Status: Active Grant

• Current Assignee: ETH Zurich, Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Schroers, Christopher, Meyer, Simone, Djelouah, Abdelaziz, Sorkine Hornung, Alexander, McWilliams, Brian, Gross, Markus
• Primary Examiner(s): Natnael, Paulos M
• Application Number:
  US15/974,576
• Publication Number:
  US 20190289257A1
• Time in Patent Office: 567 Days
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: H04N 7/0127 by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter
• CPC Class Codes: H04N 7/0135 involving interpolation processes interpolation-based image scaling G06T3/4007; interpolation for video coding H04N19/587, H04N19/59

Abstract

According to one implementation, a video processing system includes a computing platform having a hardware processor and a system memory storing a frame interpolation software code, the frame interpolation software code including a convolutional neural network (CNN) trained using a loss function having an image loss term summed with a phase loss term. The hardware processor executes the frame interpolation software code to receive first and second consecutive video frames including respective first and second images, and to decompose the first and second images to produce respective first and second image decompositions. The hardware processor further executes the frame interpolation software code to use the CNN to determine an intermediate image decomposition corresponding to an interpolated video frame for insertion between the first and second video frames based on the first and second image decompositions, and to synthesize the interpolated video frame based on the intermediate image decomposition.

Claims

1. A video processing system comprising:

a computing platform including a display, a hardware processor, and a system memory;

a frame interpolation software code stored in the system memory, the frame interpolation software code including a convolutional neural network (CNN) trained using a loss function having an image loss term summed with a phase loss term, the CNN having a plurality of convolutional processing blocks including a first subset of the plurality of convolutional processing blocks trained independently of a second subset of the plurality of convolutional processing blocks;

the hardware processor configured to execute the frame interpolation software code to;

receive a first video frame including a first image and a second video frame including a second image, the first and second video frames being consecutive;

decompose the first and second images to produce respective first and second image decompositions;

use the CNN to determine an intermediate image decomposition based on the first and second image decompositions, the intermediate image decomposition corresponding to an interpolated video frame for insertion between the first and second video frames;

synthesize the interpolated video frame based on the intermediate image decomposition; and

render a video sequence including the interpolated video frame inserted between the first and second video frames on the display.

2. The video processing system of claim 1, wherein the phase loss term of the loss function is weighted relative to the image loss term.

3. The video processing system of claim 1, wherein a weighting factor applied to the phase loss term is less than one.

4. The video processing system of claim 1, wherein the first and second images are decomposed using a complex-valued steerable pyramid to filter the first and second images.

5. The video processing system of claim 4, wherein each of the plurality of convolutional processing blocks corresponds respectively to a resolution level of the complex-valued steerable pyramid.

6. The video processing system of claim 5, wherein the CNN is configured to determine the intermediate image decomposition using the plurality of convolutional processing blocks in sequence, beginning with a convolutional processing block corresponding to a lowest resolution level of the complex-valued steerable pyramid and ending with a convolutional processing block corresponding to a highest resolution level of the complex-valued steerable pyramid.

7. The video processing system of claim 5, wherein convolutional processing blocks corresponding to lower resolution levels of the complex-valued steerable pyramid are trained independently of convolutional processing blocks corresponding to higher resolution levels of the complex-valued steerable pyramid.

8. The video processing system of claim 5, wherein the intermediate image decomposition is determined by the CNN level-by-level with respect to resolution levels of the complex-valued steerable pyramid, from a lowest resolution level to a highest resolution level, using the plurality of convolutional processing blocks in sequence.

9. The video processing system of claim 5, wherein an output of each convolutional processing block except a convolutional processing block corresponding to a highest resolution level of the complex-valued steerable pyramid is resized and provided as an input to a next one of the plurality of convolutional processing blocks in sequence.

10. A method for use by a video processing system including a display, a computing platform having a hardware processor, and a system memory storing a frame interpolation software code including a convolutional neural network (CNN) trained using a loss function having an image loss term summed with a phase loss term, the CNN having a plurality of convolutional processing blocks including a first subset of the plurality of convolutional processing blocks trained independently of a second subset of the plurality of convolutional processing blocks, the method comprising:

receiving, using the hardware processor, a first video frame including a first image and a second video frame including a second image, the first and second video frames being consecutive;

decomposing, using the hardware processor, the first and second images to produce respective first and second image decompositions;

using the hardware processor and the CNN to determine an intermediate image decomposition based on the first and second image decompositions, the intermediate image decomposition corresponding to an interpolated video frame for insertion between the first and second video frames;

synthesizing, using the hardware processor, the interpolated video frame based on the intermediate image decomposition; and

rendering, using the hardware processor, a video sequence including the interpolated video frame inserted between the first and second video frames on the display.

11. The method of claim 10, wherein the phase loss term of the loss function is weighted relative to the image loss term.

12. The method of claim 10, wherein a weighting factor applied to the phase loss term is less than one.

13. The method of claim 10, wherein decomposing the first and second images comprises filtering the first and second images using a complex-valued steerable pyramid.

14. The method of claim 13, wherein each of the plurality of convolutional processing blocks corresponds respectively to a resolution level of the complex-valued steerable pyramid.

15. The method of claim 14, wherein the CNN is configured to determine the intermediate image decomposition using the plurality of convolutional processing blocks in sequence, beginning with a convolutional processing block corresponding to a lowest resolution level of the complex-valued steerable pyramid and ending with a convolutional processing block corresponding to a highest resolution level of the complex-valued steerable pyramid.

16. The method of claim 14, wherein convolutional processing blocks corresponding to lower resolution levels of the complex-valued steerable pyramid are trained independently of convolutional processing blocks corresponding to higher resolution levels of the complex-valued steerable pyramid.

17. The method of claim 14, wherein the intermediate image decomposition is determined by the CNN level-by-level with respect to resolution levels of the complex-valued steerable pyramid, from a lowest resolution level to a highest resolution level, using the plurality of convolutional processing blocks in sequence.

18. The method of claim 14, wherein an output of each convolutional processing block except a convolutional processing block corresponding to a highest resolution level of the complex-valued steerable pyramid is resized and provided as an input to a next one of the plurality of convolutional processing blocks in sequence.