Video color propagation

• US 10,623,709 B2
• Filed: 08/31/2018
• Issued: 04/14/2020
• Est. Priority Date: 08/31/2018
• Status: Active Grant

• Current Assignee: ETH Zurich, Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Schroers, Christopher, Meyer, Simone, Cornillere, Victor, Gross, Markus, Djelouah, Abdelaziz
• Primary Examiner(s): Elahi, Shan E
• Application Number:
  US16/119,792
• Publication Number:
  US 20200077065A1
• Time in Patent Office: 592 Days
• Field of Search: 348 32
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: H04N 9/43 Conversion of monochrome picture signals to colour picture signals for colour picture display

Abstract

A video processing system includes a computing platform having a hardware processor and a memory storing a software code including a convolutional neural network (CNN). The hardware processor executes the software code to receive video data including a key video frame in color and a video sequence in gray scale, determine a first estimated colorization for each frame of the video sequence except the key video frame based on a colorization of a previous frame, and determine a second estimated colorization for each frame of the video sequence except the key video frame based on the key video frame in color. For each frame of the video sequence except the key video frame, the software code further blends the first estimated colorization with the second estimated colorization using a color fusion stage of the CNN to produce a colorized video sequence corresponding to the video sequence in gray scale.

Claims

1. A video processing system comprising:

a computing platform including a hardware processor and a system memory storing a software code including a convolutional neural network (CNN);

the hardware processor configured to execute the software code to;

receive a video data including a key video frame in color and a video sequence in gray scale beginning with the key video frame;

determine a first estimated colorization for each frame of the video sequence except the key video frame based on a colorization of a previous neighboring frame of the video sequence;

determine a second estimated colorization for each frame of the video sequence except the key video frame based on the key video frame in color; and

for each frame of the video sequence except the key video frame, blend the first estimated colorization for the each frame with the second estimated colorization for the each frame using a color fusion stage of the CNN to produce a colorized video sequence corresponding to the video sequence in gray scale.

2. The video processing system of claim 1, wherein the hardware processor is further configured to execute the software code to render the colorized video sequence on a display.

3. The video processing system of claim 1, wherein the first estimated colorization for each frame of the video sequence except the key video frame is determined based on the colorization of the previous neighboring frame of the video sequence using a local color propagation stage of the CNN.

4. The video processing system of claim 3, wherein the local color propagation stage of the CNN is trained using a warp loss function.

5. The video processing system of claim 3, wherein the second estimated colorization for each frame of the video sequence except the key video frame is determined based on the key video frame in color using a global color transfer stage of the CNN.

6. The video processing system of claim 5, wherein the second estimated colorization for each frame of the video sequence except the key video frame is determined using the global color transfer stage of the CNN further based on a comparison of a feature map of the each frame with a feature map of the key video frame in gray scale.

7. The video processing system of claim 5, wherein the first estimated colorization for each frame of the video sequence except the key video frame and the second estimated colorization for each frame of the video sequence except the key video frame are determined in parallel by the respective local color propagation stage and the global color transfer stage of the CNN.

8. The video processing system of claim 1, wherein the CNN is trained using a loss function including an image loss term and a warp loss term.

9. A method for use by a video processing system including a computing platform having a hardware processor and a system memory storing a software code including a convolutional neural network (CNN), the method comprising:

receiving, using the hardware processor, a video data including a key video frame in color and a video sequence in gray scale beginning with the key video frame;

determining, using the hardware processor, a first estimated colorization for each frame of the video sequence except the key video frame based on a colorization of a previous neighboring frame of the video sequence;

determining, using the hardware processor, a second estimated colorization for each frame of the video sequence except the key video frame based on the key video frame in color; and

for each frame of the video sequence except the key video frame, blend the first estimated colorization for the each frame with the second estimated colorization for the each frame, using the hardware processor and a color fusion stage of the CNN, to produce a colorized video sequence corresponding to the video sequence in gray scale.

10. The method of claim 9, further comprising rendering, using the hardware processor, the colorized video sequence on a display.

11. The method of claim 9, wherein determining the first estimated colorization for each frame of the video sequence except the key video frame based on the colorization of the previous neighboring frame of the video sequence is performed using a local color propagation stage of the CNN.

12. The method of claim 11, wherein the local color propagation stage of the CNN is trained using a warp loss function.

13. The method of claim 11, wherein determining the second estimated colorization for each frame of the video sequence except the key video frame based on the key video frame in color is performed using a global color transfer stage of the CNN.

14. The method of claim 13, wherein determining the second estimated colorization for each frame of the video sequence except the key video frame using the global color transfer stage of the CNN is further based on a comparison of a feature map of the each frame with a feature map of the key video frame in gray scale.

15. The method of claim 13, wherein determining the first estimated colorization for each frame of the video sequence except the key video frame and determining the second estimated colorization for each frame of the video sequence except the key video frame are performed in parallel by the respective local color propagation stage and the global color transfer stage of the CNN.

16. The method of claim 9, wherein the CNN is trained using a loss function including an image loss term and a warp loss term.

17. A video compression method for use by a computing platform having a hardware processor, the method comprising:

receiving, using the hardware processor, a video data including a key video frame in color and a video sequence in gray scale beginning with the key video frame;

determining, using the hardware processor, a first estimated colorization for each frame of the video sequence except the key video frame based on a colorization of a previous neighboring frame of the video sequence;

determining, using the hardware processor, a second estimated colorization for each frame of the video sequence except the key video frame based on the key video frame in color; and

for each frame of the video sequence except the key video frame, blend the first estimated colorization for the each frame with the second estimated colorization for the each frame, using the hardware processor and a color fusion stage of a convolutional neural network (CNN), to produce a colorized video sequence corresponding to the video sequence in gray scale.

18. The video compression method of claim 17, wherein determining the first estimated colorization for each frame of the video sequence except the key video frame based on the colorization of the previous neighboring frame of the video sequence is performed using a local color propagation stage of the CNN.

19. The video compression method of claim 18, wherein the local color propagation stage of the CNN is trained using a warp loss function.

20. The video compression method of claim 18, wherein determining the second estimated colorization for each frame of the video sequence except the key video frame based on the key video frame in color is performed using a global color transfer stage of the CNN.