Image compression and decompression using embeddings

• US 10,652,565 B1
• Filed: 10/12/2017
• Issued: 05/12/2020
• Est. Priority Date: 10/12/2017
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Zhang, Jia Bi, Olgiati, Andrea, Wang, Meng
• Primary Examiner(s): Roz, Mark
• Application Number:
  US15/782,725
• Time in Patent Office: 943 Days
• Field of Search: None
• CPC Class Codes: G06F 18/213 Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06T 9/002 using neural networks
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/463 by compressing encoding parameters before transmission

Abstract

A processing device receives a representation of an image, wherein the image has a first size and the representation has a second size that is smaller than the first size, the representation having been generated from the image by a first portion of a first trained machine learning model. The processing device processes the representation of the image using a second portion of the trained machine learning model to generate a reconstruction of the image and then outputs the reconstruction of the image.

Claims

1. A method comprising:

receiving an image at a first computing device, the image having a first size;

processing, by the first computing device, the image using a first portion of a first trained machine learning model to generate a representation of the image, the representation having a second size that is smaller than the first size;

sending the representation of the image to a second computing device comprising a second portion of the first trained machine learning model, wherein the first trained machine learning model was separated into the first portion and the second portion after training of the first trained machine learning model was performed;

processing, by the second computing device, the representation using the second portion of the first trained machine learning model to generate a reconstructed image; and

processing the reconstructed image using a generative adversarial network (GAN) to generate an improved version of the reconstructed image;

wherein a middle layer of the first trained machine learning model is a dimensionality reduction layer that generates the representation of the image, wherein the first trained machine learning model was trained while the first portion and the second portion were combined, and wherein the first trained machine learning model was separated at the dimensionality reduction layer into the first portion and the second portion after the training of the first trained machine learning model was performed.

2. The method of claim 1, further comprising performing the following prior to processing the image using the first portion of the first trained machine learning model:

processing the image using a second trained machine learning model that is trained to classify images;

determining that the image has a first image classification; and

selecting the first trained machine learning model from a plurality of trained machine learning models, wherein each of the plurality of trained machine learning models is associated with a different image classification and wherein the first trained machine learning model is associated with the first image classification.

3. The method of claim 2, wherein the image has a first image quality, wherein the reconstructed image comprises a second image quality that is lower than the first image quality, and wherein the improved version of the reconstructed image approximately has the first image quality, the method further comprising:

selecting the GAN from a plurality of GANs, wherein each GAN of the plurality of GANs is associated with a different image classification and wherein the GAN is associated with the first image classification.

19. The method of claim 1, further comprising:

adjusting, after the first trained machine learning model is trained, a hyperparameter of the first trained machine learning model, wherein the hyperparameter controls an amount of information included in representations of images generated by the first portion of the first trained machine learning model.

4. A system comprising:

a first computing device comprising;

a processing device; and

a memory to store computer executable instructions that, if executed, cause the processing device to;

receive an image, the image having a first size;

process the image using a first portion of a first trained machine learning model to generate a representation of the image, the representation having a second size that is smaller than the first size; and

send the representation of the image to a second computing device comprising a second portion of the first trained machine learning model, wherein the first trained machine learning model was separated into the first portion and the second portion after training of the first trained machine learning model was performed, wherein the second portion of the first trained machine learning model is to generate a reconstruction of the image from the representation;

wherein a middle layer of the first trained machine learning model is a dimensionality reduction layer that generates the representation of the image, wherein the first trained machine learning model was trained while the first portion and the second portion were combined, and wherein the first trained machine learning model was separated at the dimensionality reduction layer into the first portion and the second portion after the training of the first trained machine learning model was performed.

5. The system of claim 4, wherein the instructions, if executed, further cause the processing device to:

receive a training dataset comprising a first plurality of images having a first image classification;

train a first machine learning model using the training dataset to generate the first trained machine learning model;

separate the first trained machine learning model into the first portion and the second portion after training of the first trained machine learning model is complete; and

transmit the second portion to the second computing device.

6. The system of claim 4, wherein the first trained machine learning model comprises a convolutional autoencoder, wherein the first portion comprises an encoder component of the convolutional autoencoder, and wherein the second portion comprises a decoder component of the convolutional autoencoder.

7. The system of claim 4, wherein the instructions, if executed, further cause the processing device to perform the following prior to processing the image using the first portion of the first trained machine learning model:

process the image using a second trained machine learning model that is trained to classify images;

determine that the image has a first image classification; and

select the first trained machine learning model from a plurality of trained machine learning models, wherein each of the plurality of trained machine learning models is associated with a different image classification and wherein the first trained machine learning model is associated with the first image classification.

8. The system of claim 7, wherein the second computing device is further to:

select a generative adversarial network (GAN) from a plurality of GANs, wherein each GAN of the plurality of GANs is associated with a different image classification and wherein the GAN is associated with the first image classification; and

process the reconstruction of the image using the GAN to generate an improved version of the reconstruction.

9. The system of claim 4, further comprising the second computing device, wherein the second computing device is to:

receive the representation of the image;

determine that the representation of the image has been generated from the first portion of the first trained machine learning model; and

process the representation of the image using the second portion of the first trained machine learning model to generate the reconstruction of the image.

10. The system of claim 9, wherein the reconstruction of the image comprises one or more defects, and wherein the second computing device is further to:

determine that the reconstruction of the image was generated by the second portion of the first trained machine learning model; and

process the reconstruction using a generative adversarial network (GAN) to generate an improved version of the reconstruction of the image, wherein the improved version of the reconstruction of the image lacks at least one defect of the one or more defects, and wherein the GAN is associated with the first trained machine learning model.

11. The system of claim 4, wherein the instructions, if executed, further cause the processing device to:

adjust a hyperparameter of the first trained machine learning model, wherein the hyperparameter controls an amount of information included in representations of images generated by the first portion of the first trained machine learning model;

process the image using the first portion of the first trained machine learning model to generate a new representation of the image, the new representation having a third size that is lower than the second size; and

send the new representation of the image to the second computing device comprising a second portion of the first trained machine learning model, wherein the second portion of the first trained machine learning model is to generate a new reconstruction of the image from the new representation, and wherein a first delta between the image and the new reconstruction is greater than a second delta between the image and the reconstruction.

12. The system of claim 4, wherein the instructions, if executed, further cause the processing device to:

store the representation of the image in a data store;

receive a request for the image from a requestor;

retrieve the representation of the image from the data store;

determine that the representation of the image has been generated from the first portion of the first trained machine learning model;

process the representation of the image using the second portion of the first trained machine learning model to generate the reconstruction of the image; and

provide the reconstruction of the image to the requestor.

13. A non-transitory computer-readable storage device storing computer-executable instructions that, if executed by a processing device, cause the processing device to:

receive, by the processing device, a representation of an image, wherein the image has a first size and the representation has a second size that is smaller than the first size, the representation having been generated from the image by a first portion of a first trained machine learning model;

process, by the processing device, the representation of the image using a second portion of the first trained machine learning model to generate a reconstruction of the image, wherein the first trained machine learning model was separated into the first portion and the second portion after training of the first trained machine learning model was performed; and

output the reconstruction of the image;

wherein a middle layer of the first trained machine learning model is a dimensionality reduction layer that generates the representation of the image, wherein the first trained machine learning model was trained while the first portion and the second portion were combined, and wherein the first trained machine learning model was separated at the dimensionality reduction layer into the first portion and the second portion after the training of the first trained machine learning model was performed.

14. The non-transitory computer-readable storage device of claim 13, wherein the instructions, if executed, further cause the processing device to:

process the reconstruction of the image using a generative adversarial network (GAN) to generate an improved version of the reconstruction.

15. The non-transitory computer-readable storage device of claim 14, wherein the image has a first image quality, wherein the reconstruction of the image comprises a second image quality that is lower than the first image quality, and wherein the improved version of the reconstruction approximately has the first image quality.

16. The non-transitory computer-readable storage device of claim 14, wherein the first trained machine learning model is one of a plurality of trained machine learning models, each of the plurality of trained machine learning models being associated with a different image classification, and wherein the instructions, if executed, further cause the processing device to:

determine that the image has a first image classification; and

select the GAN from a plurality of GANs, wherein each GAN of the plurality of GANs is associated with a different image classification and wherein the GAN is associated with the first image classification.

17. The non-transitory computer-readable storage device of claim 13, wherein the instructions, if executed, further cause the processing device to:

receive the image;

process the image using the first portion of the first trained machine learning model to generate the representation of the image; and

store the representation of the image.

18. The non-transitory computer-readable storage device of claim 13, wherein the representation of the image is received from a remote computing device that comprises the first portion of the first trained machine learning model.