Method of training a neural network

• US 20180211164A1
• Filed: 01/23/2017
• Published: 07/26/2018
• Est. Priority Date: 01/23/2017
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia, Inc.)
• Original Assignee: FotoNation Limited (Adeia, Inc.)
• Inventors: BAZRAFKAN, Shabab, LEMLEY, Joe
• Granted Patent: US 10,915,817 B2
• Time in Patent Office: Days
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning

Abstract

Training a target neural network comprises providing a first batch of samples of a given class to respective instances of a generative neural network, each instance providing a variant of the sample in accordance with the parameters of the generative network. Each variant produced by the generative network is compared with another sample of the class to provide a first loss function for the generative network. A second batch of samples is provided to the target neural network, at least some of the samples comprising variants produced by the generative network. A second loss function is determined for the target neural network by comparing outputs of instances of the target neural network to one or more targets for the neural network. The parameters for the target neural network are updated using the second loss function and the parameters for the generative network are updated using the first and second loss functions.

Claims

1. A method of training a target neural network comprising:

a) providing a first batch of samples of a given class to respective instances of a generative neural network, each instance of said generative neural network providing a variant of said sample in accordance with the parameters of said generative neural network;

b) comparing each variant produced by said generative neural network with another sample of said class to provide a first loss function for said generative neural network;

c) providing a second batch of samples to said target neural network at least some of said samples comprising variants produced by said generative neural network;

d) determining a second loss function for said target neural network by comparing outputs of instances of said target neural network to one or more targets for said neural network;

e) updating the parameters for said target neural network using said second loss function;

f) updating the parameters for said generative neural network using said first and second loss functions; and

g) repeating steps a) to f) for successive batches of samples.

2. A method according to claim 1 wherein said second batch of samples comprises a proportion of variants produced by said generative neural network to samples from said given class.

3. A method according to claim 2 wherein said proportion is 50:

50.

4. A method according to claim 2 wherein said proportion varies from batch to batch.

5. A method according to claim 4 wherein said portion of variants increases with successive repetitions of steps a) to f).

6. A method according to claim 2 wherein said second batch of samples comprises a portion of variants produced by said generative neural network being a function of the first loss function for said generative neural network.

7. A method according to claim 1 wherein said target neural network is a multi-class classifier and said method further comprises:

providing a batch of samples of a given second class to respective instances of a second generative neural network, each instance of said second generative neural network providing a variant of said sample in accordance with the parameters of said second generative neural network;

comparing each variant produced by said second generative neural network with another sample of said second class to provide a loss function for said second generative neural network;

wherein said second batch of samples provided to said target neural network further comprises said variants produced by said second generative neural network; and

updating the parameters for said second generative neural network using said loss function for said second generative neural network and said second loss function for said target neural network.

8. A method according to claim 1 wherein a combined loss function for said first generative neural network is α

L_A+β

L_B where L_A is said first loss function and L_B is said second loss function and said combined loss function is used to update the parameters of said generative neural network.

9. A method according to claim 8 where α

=0.3 and β

=0.7.

10. A method according to claim 8 where α

and β

change with successive repetitions of steps a) to f).

11. A method according to claim 1 wherein said target neural network comprises a fully-connected layer providing said at least one output.

12. A method according to claim 1 wherein each sample comprises an image comprising one or more channels.

13. A method according to claim 12 wherein said channels comprise one or more of image planes or processed versions of image planes.

14. A method according to claim 12 wherein said target neural comprises a gender classifier for indicating a gender of a subject of an image.

15. A method according to claim 1 wherein the balance of samples provided to the target neural network are taken from said samples from said class used to provide said first loss function for said generative neural network.

16. A method according to claim 1 comprising providing pairs of samples from said first batch of samples of a given class to respective instances of a generative neural network, each instance of said generative neural network combining said pairs of samples and providing a variant of said samples in accordance with the parameters of said generative neural network.

17. A computer program product comprising a computer readable medium on which computer executable instructions are stored which when executed on a computing device are arranged to perform the method of claim 1.

18. A target neural network produced according to the method of claim 1.