Producing Higher-Quality Samples Of Natural Images

• US 20170365038A1
• Filed: 06/15/2017
• Published: 12/21/2017
• Est. Priority Date: 06/16/2016
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Denton, Emily, Chintala, Soumith, Szlam, Arthur David, Fergus, Robert D.
• Granted Patent: US 10,319,076 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/60 using machine learning, e.g. neural networks

Abstract

In one embodiment, a method includes accessing a plurality of generative adversarial networks (GANs) that are each applied to a particular level k of a Laplacian pyramid. Each GAN may comprise a generative model G_k and a discriminative model D_k. At each level k, the generative model G_k may take as input a noise vector z_k and may output a generated image {tilde over (h)}_k. At each level k, the discriminative model D_k may take as input either the generated image {tilde over (h)}_k or a real image h_k, and may output a probability that the input was the real image h_k. The method may further include generating a sample image Ĩ_k from the generated images {tilde over (h)}_k, wherein the sample image is based on the probabilities outputted by each of the discriminative models D_k and the generated images {tilde over (h)}_k. The method may further include providing the sample image Ĩ_k for display.

Claims

1. A method comprising, by one or more computing devices:

accessing a plurality of generative adversarial networks (GANs) that are each applied to a particular level k of a Laplacian pyramid, wherein each GAN comprises a generative model G_k and a discriminative model D_k, wherein, at each level k;

the generative model G_k takes as input a noise vector z_k and outputs a generated image {tilde over (h)}_k; and

the discriminative model D_k takes as input either the generated image {tilde over (h)}_k or a real image h_k drawn from a set of training data, and outputs a probability that the input was the real image h_k;

generating a sample image Ĩ

_k from the generated images {tilde over (h)}_k, wherein the sample image is based at least in part on the probabilities outputted by each of the discriminative models D_k at each level k of the Laplacian pyramid and the generated images {tilde over (h)}_k; and

providing the sample image Ĩ

_k for display on a display screen of a client device of a user of a communications network.

2. The method of claim 1, wherein generating the sample image Ĩ

_k from the generated images {tilde over (h)}_k further comprises;

applying an upsampling operator u(.) to at least one of the generated images {tilde over (h)}_k; and

combining the upsampled image {tilde over (h)}_k with an image I_{k 1} generated by a generative model G_{k 1}, wherein the image I_{k 1} was generated at level k 1 in the Laplacian pyramid.

3. The method of claim 2, wherein the sample image Ĩ

_k is generated using the equation Ĩ

_k=u(Ĩ

_{k 1}) {tilde over (h)}_k.

4. The method of claim 1, wherein the generated images {tilde over (h)}_k are generated in a coarse-to-fine fashion.

5. The method of claim 1, wherein each level k in the Laplacian pyramid corresponds to a generated image {tilde over (h)}_k that comprises a particular number of pixels, wherein as k increases, the number of pixels in {tilde over (h)}_k decreases.

6. The method of claim 1, wherein the generative model G_k and the discriminative model D_k each take an additional vector of information as input.

7. The method of claim 6, wherein the additional vector of information is a generated image l_k created by a conditional GAN model.

8. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

access a plurality of generative adversarial networks (GANs) that are each applied to a particular level k of a Laplacian pyramid, wherein each GAN comprises a generative model G_k and a discriminative model D_k, wherein, at each level k;

the generative model G_k takes as input a noise vector z_k and outputs a generated image {tilde over (h)}_k; and

the discriminative model D_k takes as input either the generated image {tilde over (h)}_k or a real image h_k drawn from a set of training data, and outputs a probability that the input was the real image h_k;

generate a sample image Ĩ

_k from the generated images {tilde over (h)}_k, wherein the sample image is based at least in part on the probabilities outputted by each of the discriminative models D_k at each level k of the Laplacian pyramid and the generated images {tilde over (h)}_k; and

provide the sample image Ĩ

_k for display on a display screen of a client device of a user of a communications network.

9. The media of claim 8, wherein generating the sample image Ĩ

_k from the generated images {tilde over (h)}_k further comprises;

applying an upsampling operator u(.) to at least one of the generated images {tilde over (h)}_k; and

combining the upsampled image {tilde over (h)}_k with an image I_{k 1} generated by a generative model G_{k 1}, wherein the image I_{k 1} was generated at level k 1 in the Laplacian pyramid.

10. The media of claim 9, wherein the sample image Ĩ

_k is generated using the equation Ĩ

_k=u(Ĩ

_{k 1}) {tilde over (h)}_k.

11. The media of claim 8, wherein the generated images {tilde over (h)}_k are generated in a coarse-to-fine fashion.

12. The media of claim 8, wherein each level k in the Laplacian pyramid corresponds to a generated image {tilde over (h)}_k that comprises a particular number of pixels, wherein as k increases, the number of pixels in {tilde over (h)}_k decreases.

13. The media of claim 8, wherein the generative model G_k and the discriminative model D_k each take an additional vector of information as input.

14. The media of claim 13, wherein the additional vector of information is a generated image l_k created by a conditional GAN model.

15. A system comprising:

one or more processors; and

one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to;

access a plurality of generative adversarial networks (GANs) that are each applied to a particular level k of a Laplacian pyramid, wherein each GAN comprises a generative model G_k and a discriminative model D_k, wherein, at each level k;

the generative model G_k takes as input a noise vector z_k and outputs a generated image {tilde over (h)}_k; and

the discriminative model D_k takes as input either the generated image {tilde over (h)}_k or a real image h_k drawn from a set of training data, and outputs a probability that the input was the real image h_k;

generate a sample image Ĩ

_k from the generated images {tilde over (h)}_k, wherein the sample image is based at least in part on the probabilities outputted by each of the discriminative models D_k at each level k of the Laplacian pyramid and the generated images {tilde over (h)}_k; and

provide the sample image Ĩ

_k for display on a display screen of a client device of a user of a communications network.

16. The system of claim 15, wherein generating the sample image Ĩ

_k from the generated images {tilde over (h)}_k further comprises;

applying an upsampling operator u(.) to at least one of the generated images {tilde over (h)}_k; and

combining the upsampled image {tilde over (h)}_k with an image I_{k 1} generated by a generative model G_{k 1}, wherein the image I_{k 1} was generated at level k 1 in the Laplacian pyramid.

17. The system of claim 16, wherein the sample image Ĩ

_k is generated using the equation Ĩ

_k=u(Ĩ

_{k 1}) {tilde over (h)}_k.

18. The system of claim 15, wherein the generated images {tilde over (h)}_k are generated in a coarse-to-fine fashion.

19. The system of claim 15, wherein each level k in the Laplacian pyramid corresponds to a generated image {tilde over (h)}_k that comprises a particular number of pixels, wherein as k increases, the number of pixels in {tilde over (h)}_k decreases.

20. The system of claim 15, wherein the generative model G_k and the discriminative model D_k each take an additional vector of information as input.