System and method for addressing overfitting in a neural network

• US 9,406,017 B2
• Filed: 08/30/2013
• Issued: 08/02/2016
• Est. Priority Date: 12/24/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Hinton, Geoffrey E., Krizhevsky, Alexander, Sutskever, Ilya, Srivastva, Nitish
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Figueroa, Kevin W
• Application Number:
  US14/015,768
• Publication Number:
  US 20140180986A1
• Time in Patent Office: 1,068 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/24133 Distances to prototypes
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/082 modifying the architecture, e.g. adding, deleting or silencing nodes or connections
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent

Abstract

A system for training a neural network. A switch is linked to feature detectors in at least some of the layers of the neural network. For each training case, the switch randomly selectively disables each of the feature detectors in accordance with a preconfigured probability. The weights from each training case are then normalized for applying the neural network to test data.

Claims

1. A computer-implemented method comprising:

obtaining a plurality of training cases; and

training a neural network having a plurality of layers on the plurality of training cases, each of the layers including one or more feature detectors, each of the feature detectors having a corresponding set of weights, and a subset of the feature detectors being associated with respective probabilities of being disabled during processing of each of the training cases, wherein training the neural network on the plurality of training cases comprises, for each of the training cases respectively;

determining one or more feature detectors to disable during processing of the training case, comprising determining whether to disable each of the feature detectors in the subset based on the respective probability associated with the feature detector,

disabling the one or more feature detectors in accordance with the determining, and

processing the training case using the neural network with the one or more feature detectors disabled to generate a predicted output for the training case.

2. The method of claim 1, wherein training the neural network further comprises:

adjusting the weights of each of feature detectors to generate trained values for each weight in the set of weights corresponding to the feature detector.

3. The method of claim 2, further comprising:

normalizing the trained weights for each of the feature detectors in the subset, wherein normalizing the trained weights comprises multiplying the trained values of the weights for each of the one or more feature detectors in the subset by a respective probability of the feature detector not being disabled during processing of each of the training cases.

4. The method of claim 1, wherein the subset includes feature detectors in a first layer of the plurality of layers and feature detectors in one or more second layers of the plurality of layers, wherein the feature detectors in the first layer are associated with a first probability and wherein the feature detectors in the one or more second layers are associated with a second, different probability.

5. The method of claim 4, wherein the first layer is an input layer of the neural network and the one or more second layers are hidden layers of the neural network.

6. The method of claim 4, wherein the first layer and the one or more second layers are hidden layers of the neural network.

7. The method of claim 1, wherein determining one or more feature detectors to disable during processing of the training case comprises:

determining to disable the same feature detectors that were disabled during processing of a preceding training case.

8. The method of claim 1, wherein the neural network is one of a feedforward neural network, a convolutional neural network, a recurrent neural network, an auto-encoder, or a Boltzmann machine.

9. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

obtaining a plurality of training cases; and

training a neural network having a plurality of layers on the plurality of training cases, each of the layers including one or more feature detectors, each of the feature detectors having a corresponding set of weights, and a subset of the feature detectors being associated with respective probabilities of being disabled during processing of each of the training cases, wherein training the neural network on the plurality of training cases comprises, for each of the training cases respectively;

determining one or more feature detectors to disable during processing of the training case, comprising determining whether to disable each of the feature detectors in the subset based on the respective probability associated with the feature detector,

disabling the one or more feature detectors in accordance with the determining, and

processing the training case using the neural network with the one or more feature detectors disabled to generate a predicted output for the training case.

10. The system of claim 9, wherein training the neural network further comprises:

adjusting the weights of each of the feature detectors to generate trained values for each weight in the set of weights corresponding to the feature detector.

11. The system of claim 10, the operations further comprising:

normalizing the trained weights for each of the feature detectors in the subset, wherein normalizing the trained weights comprises multiplying the trained values of the weights for each of the one or more feature detectors in the subset by a respective probability of the feature detector not being disabled during processing of each of the training cases.

12. The system of claim 9, wherein the subset includes feature detectors in a first layer of the plurality of layers and feature detectors in one or more second layers of the plurality of layers, wherein the feature detectors in the first layer are associated with a first probability and wherein the feature detectors in the one or more second layers are associated with a second, different probability.

13. The system of claim 12, wherein the first layer is an input layer of the neural network and the one or more second layers are hidden layers of the neural network.

14. The system of claim 12, wherein the first layer and the one or more second layers are hidden layers of the neural network.

15. The system of claim 9, wherein determining one or more feature detectors to disable during processing of the training case comprises:

determining to disable the same feature detectors that were disabled during processing of a preceding training case.

16. The system of claim 9, wherein the neural network is one of a feedforward neural network, a convolutional neural network, a recurrent neural network, an auto-encoder, or a Boltzmann machine.

17. A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

obtaining a plurality of training cases; and

training a neural network having a plurality of layers on the plurality of training cases, each of the layers including one or more feature detectors, each of the feature detectors having a corresponding set of weights, and a subset of the feature detectors being associated with respective probabilities of being disabled during processing of each of the training cases, wherein training the neural network on the plurality of training cases comprises, for each of the training cases respectively;

determining one or more feature detectors to disable during processing of the training case, comprising determining whether to disable each of the feature detectors in the subset based on the respective probability associated with the feature detector,

disabling the one or more feature detectors in accordance with the determining, and

processing the training case using the neural network with the one or more feature detectors disabled to generate a predicted output for the training case.

18. The computer storage medium of claim 17, wherein training the neural network further comprises:

adjusting the weights of each of the feature detectors to generate trained values for each weight in the set of weights corresponding to the feature detector.

19. The computer storage medium of claim 18, the operations further comprising:

normalizing the trained weights for each of the feature detectors in the subset, wherein normalizing the trained weights comprises multiplying the trained values of the weights for each of the one or more feature detectors in the subset by a respective probability of the feature detector not being disabled during processing of each of the training cases.

20. The computer storage medium of claim 17, wherein the subset includes feature detectors in a first layer of the plurality of layers and feature detectors in one or more second layers of the plurality of layers, wherein the feature detectors in the first layer are associated with a first probability and wherein the feature detectors in the one or more second layers are associated with a second, different probability.

21. The computer storage medium of claim 20, wherein the first layer is an input layer of the neural network and the one or more second layers are hidden layers of the neural network.

22. The computer storage medium of claim 20, wherein the first layer and the one or more second layers are hidden layers of the neural network.

23. The computer storage medium of claim 17, wherein determining one or more feature detectors to disable during processing of the training case comprises:

determining to disable the same feature detectors that were disabled during processing of a preceding training case.

24. The computer storage medium of claim 17, wherein the neural network is one of a feedforward neural network, a convolutional neural network, a recurrent neural network, an auto-encoder, or a Boltzmann machine.