End-to-end video and image compression

• US 10,623,775 B1
• Filed: 11/06/2017
• Issued: 04/14/2020
• Est. Priority Date: 11/04/2016
• Status: Active Grant

• Current Assignee: X Corp. (f/k/a Twitter, Inc.) (X Holdings Corp.)
• Original Assignee: X Corp. (f/k/a Twitter, Inc.) (X Holdings Corp.)
• Inventors: Theis, Lucas, Huszar, Ferenc, Wang, Zehan, Shi, Wenzhe
• Primary Examiner(s): Habib, Irfan
• Application Number:
  US15/804,584
• Time in Patent Office: 890 Days
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/082 modifying the architecture, e.g. adding, deleting or silencing nodes or connections
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/61 in combination with predictive coding

Abstract

A system (e.g., an auto-encoder system) includes an encoder, a decoder and a learning module. The encoder generates compressed video data using a lossy compression algorithm, the lossy compression algorithm being implemented using a trained neural network with at least one convolution, generate at least one first parameter based on the compressed video data, and communicate the compressed video data and the model to at least one device configured to decode the compressed video data using an inverse algorithm based on the lossy compression algorithm. The decoder generates decoded video data based on the compressed video data using the inverse algorithm and the model, and generate at least one second parameter based on the decoded video data. The learning module trains the model using the at least one first parameter and the at least one second parameter.

Claims

1. An auto-encoder comprising:

an encoder configured to;

receive video data to be compressed,

select a model from a plurality of models based on the video data to be compressed, the model being based on at least one neural network, the at least one neural network configured to implement a lossy compression algorithm, the at least one neural network having been trained based on a type of the video data to be compressed,

generate compressed video data using the selected model,

generate at least one first parameter based on the compressed video data, and

communicate the compressed video data and the model to at least one device configured to decode the compressed video data using an inverse algorithm based on the lossy compression algorithm;

a decoder configured to;

generate decoded video data based on the compressed video data using the inverse algorithm and the model,

generate at least one second parameter based on the decoded video data, and

train the model using the at least one first parameter and the at least one second parameter.

2. The auto-encoder of claim 1, wherein the encoder is configured to quantize the compressed video data using rounding.

3. The auto-encoder of claim 1, wherein the encoder is configured to downsample the compressed video data using a stride in a convolution.

4. The auto-encoder of claim 1, wherein the decoder is configured to upsample the compressed video data using zero-padding and reorganization of coefficients in a mask used in a convolution.

5. The auto-encoder of claim 1, wherein the encoder is configured to use a continuous, differentiable approximation of a probability distribution to assign a number of bits to a representation based on a frequency of the representation.

6. The auto-encoder of claim 1, wherein an independent Gaussian Scale Mixture (GSM) is used to model a distribution of coefficients and estimate a bit rate for the model used in the auto-encoder.

7. The auto-encoder of claim 1, wherein

model coefficients associated with the model are trained using incremental training, and

the incremental training uses a binary mask.

8. The auto-encoder of claim 1, wherein model coefficients associated with the model correspond to elements in a mask used in a convolution.

9. The auto-encoder of claim 1, wherein the encoder is further configured to perform at least one of intra-frame prediction and inter-frame prediction.

10. A method comprising:

receiving video data to be compressed,

selected a model from a plurality of models based on the video data to be compressed, the model being based on at least one neural network, the at least one neural network configured to implement a lossy compression algorithm, the at least one neural network having been trained based on a type of the video data to be compressed,

generating first compressed video data using the selected model;

generating at least one first parameter based on the compressed video data;

communicating the compressed video data and the model to at least one device configured to decode the compressed video data using an inverse algorithm based on the lossy compression algorithm;

generating decoded video data based on the compressed video data using the inverse algorithm and the model;

generating at least one second parameter based on the decoded video data;

training the model using the at least one first parameter and the at least one second parameter; and

generating second compressed video data using the trained model.

11. The method of claim 10, wherein the compressed video data is quantized using rounding.

12. The method of claim 10, wherein the compressed video data is downsampled using a stride in a convolution.

13. The method of claim 10, wherein the compressed video data is upsampled using zero-padding and reorganization of coefficients in a mask used in a convolution.

14. The method of claim 10, wherein a continuous, differentiable approximation of a probability distribution is used to assign a number of bits to a representation based on a frequency of the representation.

15. The method of claim 10, wherein an independent Gaussian Scale Mixture (GSM) is used to model a distribution of coefficients and to estimate a bit rate for the model.

16. The method of claim 10, wherein

model coefficients associated with the model are trained using incremental training, and

the incremental training uses a binary mask.

17. The method of claim 10, wherein model coefficients associated with the model correspond to elements in a mask used in a convolution.

18. The method of claim 10, further comprising at least one of:

performing intra-frame prediction on input video data; and

performing inter-frame prediction on the input video data.

19. The method of claim 10, wherein the model is selected based on a type of image or video to be encoded.

20. A non-transitory computer readable medium having code segments stored thereon, the code segments, when executed by a processor cause the processor to:

receive video data to be compressed;

select a model from a plurality of models based on the video data to be compressed, the model being based on at least one neural network, the at least one neural network configured to implement a lossy compression algorithm, the at least one neural network having been trained based on a type of the video data to be compressed;

generate first compressed video data using the selected model;

generate at least one first parameter based on the compressed video data;

communicate the compressed video data and the model to at least one device configured to decode the compressed video data using an inverse algorithm based on the lossy compression algorithm;

generate decoded video data based on the compressed video data using the inverse algorithm and the model;

generate at least one second parameter based on the decoded video data;

train the model using the at least one first parameter and the at least one second parameter; and

generate second compressed video data using the trained model.