COMPRESSING IMAGES USING NEURAL NETWORKS

• US 20170230675A1
• Filed: 12/30/2016
• Published: 08/10/2017
• Est. Priority Date: 02/05/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Wierstra, Daniel Pieter, Gregor, Karol, Besse, Frederic Olivier
• Granted Patent: US 10,432,953 B2
• Time in Patent Office: Days
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 9/002 using neural networks
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for compressing images using neural networks. One of the methods includes receiving an image; processing the image using an encoder neural network, wherein the encoder neural network is configured to receive the image and to process the image to generate an output defining values of a first number of latent variables that each represent a feature of the image; generating a compressed representation of the image using the output defining the values of the first number of latent variables; and providing the compressed representation of the image for use in generating a reconstruction of the image.

Claims

1. A method comprising:

receiving an image;

processing the image using an encoder neural network, wherein the encoder neural network is configured to receive the image and to process the image to generate an output defining values of latent variables that each represent a feature of the image;

generating a lossy compressed representation of the image using a first number of the latent variables that is less than all of the latent variables that have values that are defined by the output; and

providing the lossy compressed representation of the image for use in generating a reconstruction of the image.

2. The method of claim 1, wherein generating the lossy compressed representation of the image comprises:

discretizing the values of the first number of latent variables; and

compressing a sequence of the discretized values using arithmetic encoding.

3. The method of claim 2, further comprising:

determining a respective probability for each of the first number of latent variables by sampling from a prior distribution parameterized by a hidden state of a generative neural network, wherein compressing the sequence of the discretized values comprises;

compressing the sequence of the discretized values and their respective probabilities using arithmetic encoding.

4. The method of claim 1, wherein, for each latent variable, the output parametrizes a distribution from which the value of the latent variable is selected.

5. The method of claim 1, wherein the encoder neural network comprises a plurality of neural network layers, and wherein each of the plurality of neural network layers is configured to generate a respective layer output defining a value of a respective latent variable that represents a feature of the image.

6. The method of claim 5, wherein the latent variables in the first number of latent variables are less than all of the latent variables that have values that are defined by the layer outputs of the plurality of neural network layers.

7. The method of claim 6, wherein the plurality of neural network layers are arranged in a hierarchy, and wherein the first number of latent variables are the latent variables that have values that are defined by the layer outputs of a predetermined number of highest levels in the hierarchy.

8. The method of claim 1, wherein the features that are arranged in a hierarchy from least abstract representation of the image to most abstract representation of the image, and wherein the first number of latent variables are the latent variables that represent features at a predetermined number of highest levels in the hierarchy.

9. The method of claim 1, further comprising:

generating the reconstruction of the image from the lossy compressed representation of the image, comprising;

selecting a value of the latent variables that are not in the first number of latent variables randomly from a prior distribution; and

generating the reconstruction of the image by conditioning a generative neural network on the values of the first number of latent variables and the randomly selected values of the latent variables that are not in the first number of latent variables.

10. The method of claim 9, wherein the generative neural network and the encoder neural network have been jointly trained as a variational auto-encoder.

11. The method of claim 9, wherein the generative neural network and the encoder neural network are recurrent neural networks.

12. The method of claim 1, wherein providing the lossy compressed representation comprises transmitting the lossy compressed representation over a data communication network to a decoder system.

13. The method of claim 1, wherein providing the lossy compressed representation comprises storing the lossy compressed representation in a memory and, in response to a request to retrieve the image from the memory, retrieving the lossy compressed representation from the memory and providing the lossy compressed representation for use in reconstructing the image.

14. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

receiving an image;

processing the image using an encoder neural network, wherein the encoder neural network is configured to receive the image and to process the image to generate an output defining values of latent variables that each represent a feature of the image;

generating a lossy compressed representation of the image using a first number of the latent variables that is less than all of the latent variables that have values that are defined by the output; and

providing the lossy compressed representation of the image for use in generating a reconstruction of the image.

15. The system of claim 14, wherein generating the lossy compressed representation of the image comprises:

discretizing the values of the first number of latent variables; and

compressing a sequence of the discretized values using arithmetic encoding.

16. The system of claim 15, the operations further comprising:

determining a respective probability for each of the first number of latent variables by sampling from a prior distribution parameterized by a hidden state of a generative neural network, wherein compressing the sequence of the discretized values comprises;

compressing the sequence of the discretized values and their respective probabilities using arithmetic encoding.

17. The system of claim 14, wherein the features that are arranged in a hierarchy from least abstract representation of the image to most abstract representation of the image, and wherein the first number of latent variables are the latent variables that represent features at a predetermined number of highest levels in the hierarchy.

18. The system of claim 14, the operations further comprising:

generating the reconstruction of the image from the lossy compressed representation of the image, comprising;

selecting a value of the latent variables that are not in the first number of latent variables randomly from a prior distribution; and

generating the reconstruction of the image by conditioning a generative neural network on the values of the first number of latent variables and the randomly selected values of the latent variables that are not in the first number of latent variables.

19. The system of claim 18, wherein the generative neural network and the encoder neural network have been jointly trained as a variational auto-encoder.

20. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving an image;

processing the image using an encoder neural network, wherein the encoder neural network is configured to receive the image and to process the image to generate an output defining values of latent variables that each represent a feature of the image;

generating a lossy compressed representation of the image using a first number of the latent variables that is less than all of the latent variables that have values that are defined by the output; and

providing the lossy compressed representation of the image for use in generating a reconstruction of the image.