Image generation using neural networks

• US 9,940,551 B1
• Filed: 06/17/2016
• Issued: 04/10/2018
• Est. Priority Date: 06/17/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Mordvintsev, Alexander
• Primary Examiner(s): WOLDEMARIAM, AKLILU K
• Application Number:
  US15/185,655
• Time in Patent Office: 662 Days
• Field of Search: 382118, 382152, 382190, 382100, 382165, 382254, 382156, 382178, 382310, 345419, 345440, 345630
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 18/2148 characterised by the process organisation or structure, e.g. boosting cascade
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for image generation using neural networks. In one of the methods, an initial image is received. Data defining an objective function is received, and the objective function is dependent on processing of a neural network trained to identify features of an image. The initial image is modified to generate a modified image by iteratively performing the following: a current version of the initial image is processed using the neural network to generate a current objective score for the current version of the initial image using the objective function; and the current version of the initial image is modified to increase the current objective score by enhancing a feature detected by the processing.

Claims

1. A method performed by data processing apparatuses, the method comprising:

receiving an initial image;

receiving data defining an objective function that is dependent on processing of a neural network trained to identify features of an image, wherein the neural network comprises a plurality of neurons, and wherein the objective function includes terms associated with outputs generated by a subset of the neurons of the neural network during processing of an input image by the neural network; and

modifying the initial image to generate a modified image by iteratively performing the following;

processing a current version of the initial image using the neural network to generate a current objective score for the current version of the initial image using the objective function, wherein the objective score is calculated based on the outputs generated by the subset of the neurons of the neural network during processing of the input image by the neural network; and

modifying the current version of the initial image to increase the current objective score by performing an iteration of gradient descent against the current version of the initial image to enhance a feature detected by the processing, wherein performing the iteration of gradient descent comprises backpropagating gradients through a plurality of layers of the neural network to determine an adjustment to the current version of the initial image.

2. The method of claim 1, wherein modifying the initial image to generate the modified image by iteratively performing the following further comprises permuting the current version of the initial image.

3. The method of claim 2, wherein permuting the current version of the initial image comprises jittering the current version of the initial image.

4. The method of claim 2, wherein permuting the current version of the initial image comprises resizing the current version of the initial image.

5. The method of claim 1, wherein the neural network is organized into a series of layers, and wherein the objective function has term associated with one of the series of layers.

6. The method of claim 1, wherein processing a current version of the initial image using the neural network to generate a current objective score for the current version of the initial image using the objective function comprises:

boosting gradient values in areas of low frequency.

7. A non-transitory, computer-readable medium storing instructions operable when executed to cause at least one processor to perform operations comprising:

receiving an initial image;

receiving data defining an objective function that is dependent on processing of a neural network trained to identify features of an image, wherein the neural network comprises a plurality of neurons, and wherein the objective function includes terms associated with outputs generated by a subset of the neurons of the neural network during processing of an input image by the neural network; and

modifying the initial image to generate a modified image by iteratively performing the following;

processing a current version of the initial image using the neural network to generate a current objective score for the current version of the initial image using the objective function, wherein the objective score is calculated based on the outputs generated by the subset of the neurons of the neural network during processing of the input image by the neural network; and

modifying the current version of the initial image to increase the current objective score by performing an iteration of gradient descent against the current version of the initial image to enhance a feature detected by the processing, wherein performing the iteration of gradient descent comprises backpropagating gradients through a plurality of layers of the neural network to determine an adjustment to the current version of the initial image.

8. The medium of claim 7, wherein modifying the initial image to generate the modified image by iteratively performing the following further comprises permuting the current version of the initial image.

9. The medium of claim 8, wherein permuting the current version of the initial image comprises jittering the current version of the initial image.

10. The medium of claim 8, wherein permuting the current version of the initial image comprises jittering the current version of the initial image.

11. The medium of claim 7, wherein the neural network is organized into a series of layers, and wherein the objective function has term associated with one of the series of layers.

12. The medium of claim 7, wherein processing a current version of the initial image using the neural network to generate a current objective score for the current version of the initial image using the objective function comprises:

boosting gradient values in areas of low frequency.

13. A system comprising:

one or more processors configured to execute computer program instructions; and

memory encoded with computer program instructions that, when executed by one or more processors, cause a computer device to perform operations comprising;

receiving an initial image;

receiving data defining an objective function that is dependent on processing of a neural network trained to identify features of an image, wherein the neural network comprises a plurality of neurons, and wherein the objective function includes terms associated with outputs generated by a subset of the neurons of the neural network during processing of an input image by the neural network; and

modifying the initial image to generate a modified image by iteratively performing the following;

processing a current version of the initial image using the neural network to generate a current objective score for the current version of the initial image using the objective function, wherein the objective score is calculated based on the outputs generated by the subset of the neurons of the neural network during processing of the input image by the neural network; and

modifying the current version of the initial image to increase the current objective score by performing an iteration of gradient descent against the current version of the initial image to enhance a feature detected by the processing, wherein performing the iteration of gradient descent comprises backpropagating gradients through a plurality of layers of the neural network to determine an adjustment to the current version of the initial image.

14. The system of claim 13, wherein modifying the initial image to generate the modified image by iteratively performing the following further comprises permuting the current version of the initial image.

15. The system of claim 14, wherein permuting the current version of the initial image comprises jittering the current version of the initial image.

16. The system of claim 14, wherein permuting the current version of the initial image comprises jittering the current version of the initial image.

17. The system of claim 13, wherein the neural network is organized into a series of layers, and wherein the objective function has term associated with one of the series of layers.

18. The system of claim 13, wherein processing a current version of the initial image using the neural network to generate a current objective score for the current version of the initial image using the objective function comprises:

boosting gradient values in areas of low frequency.