Novel Quadratic Regularization For Neural Network With Skip-Layer Connections

• US 20140279771A1
• Filed: 03/12/2013
• Published: 09/18/2014
• Est. Priority Date: 03/12/2013
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: GOLOVASHKIN, DMITRY, ABOYOUN, PATRICK, SASHIKANTH, VAISHNAVI
• Granted Patent: US 9,047,566 B2
• Time in Patent Office: Days
• US Class Current: 706/16
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology

Abstract

According to one aspect of the invention, target data comprising observations is received. A neural network comprising input neurons, output neurons, hidden neurons, skip-layer connections, and non-skip-layer connections is used to analyze the target data based on an overall objective function that comprises a linear regression part, the neural network'"'"'s unregularized objective function, and a regularization term. An overall optimized first vector value of a first vector and an overall optimized second vector value of a second vector are determined based on the target data and the overall objective function. The first vector comprises skip-layer weights for the skip-layer connections and output neuron biases, whereas the second vector comprises non-skip-layer weights for the non-skip-layer connections.

Claims

1. A method comprising:

receiving target data comprising a plurality of observations, wherein each observation in said plurality of observation comprises (a) one or more values of one or more input neurons in a neural network and (b) one or more corresponding values of one or more output neurons in the neural network, and wherein the neural network comprises one or more layers of hidden neurons, one or more skip-layer connections between the one or more input neurons and the one or more output neurons, and one or more non-skip-layer connections between neurons of the neural network;

determining an overall objective function that comprises (a) a first part corresponding to linear regression, (b) a second part corresponding to the neural network'"'"'s unregularized objective function, and (c) a third part corresponding to a regularization of an unregularized objective function of the neural network; and

determining, based at least in part on said target data and said overall objective function, an overall optimized first vector value of a first vector and an overall optimized second vector value of a second vector, wherein said first vector comprises a set of skip-layer weights for the one or more skip-layer connections and a set of output neuron biases, and wherein said second vector comprises a set of non-skip-layer weights for the one or more non-skip-layer connections;

wherein the method is performed by one or more computing devices.

2. The method of claim 1, wherein said first part of said objective function comprises a L2 norm as multiplied by a complement of an alpha factor, and wherein said second part of said objective function comprises an unregularized neural network objective function as multiplied by said alpha factor.

3. The method of claim 1, wherein said third part of said objective function represents a quadratic regularization expression with the one or more non-skip-layer weights as arguments, as multiplied by a beta factor, and wherein said determining, based at least in part on said target data and said overall objective function, a first vector value of a first vector and a second vector value of a second vector comprises:

initializing said first vector to an incumbent first vector value determined with only said first part of said objective function;

initializing said second vector to an incumbent second vector value within bound box constraints represented by a lower bound L and an upper bound U;

estimating, based on said incumbent first vector value of said first vector and said incumbent second vector value of said second vector, a minimum eigenvalue of a Hessian matrix comprising second derivatives of said unregularized objective function in said second part of said overall objective function;

determining, based at least in part on said minimum eigenvalue of said Hessian matrix, an upper bound for said beta factor in said third part of said objective function;

determining a plurality of vector value combinations of said first vector and said second vector by minimizing, based on said target data without de-trending preprocessing, said overall objective function for a plurality of distinct factor value combinations of said alpha factor and said beta factor;

wherein each distinct factor value combination in the plurality of distinct factor value combinations comprises a distinct combination of an alpha factor value of said alpha factor between zero (0) and one (1), inclusive, and a beta factor value of said beta factor between zero (0) and said upper bound of said beta factor, inclusive;

wherein each vector value combination in the plurality of vector values combinations comprises a first vector value of said first vector and a second vector value of said second vector; and

selecting, based on one or more selection criteria, a particular vector value combination among the plurality of vector value combinations as an overall optimized vector value combination, wherein said overall optimized vector value combination comprises said overall optimized first vector value and said overall optimized second vector value.

4. The method of claim 3, wherein said minimum eigenvalue of said Hessian matrix is numerically estimated without an explicit functional form of said Hessian matrix in terms of weights and biases of the neural network.

5. The method of claim 3, wherein said minimum eigenvalue of said Hessian matrix is computed from an explicit functional form of said Hessian matrix in terms of weights and biases of the neural network.

6. The method of claim 1, wherein said unregularized objective function in said second part of said objective function is explicitly derived from (a) a configuration of the neural network and (b) a plurality of activation functions associated with the one or more hidden neurons and with the one or more output neurons.

7. The method of claim 1, wherein said one or more non-skip layer weights were projected into bound box constraints represented by a lower bound vector L and an upper bound vector U.

8. The method of claim 1, wherein said neural network is a feed-forward neural network.

9. The method of claim 1, wherein at least one of a plurality of activation functions in said neural network is one of a tan, bipolarSigmoid, cos, gaussian, gompertz, linear, sigmoid, reciprocal, sin, square, or tan h.

10. The method of claim 1, wherein said overall optimized first vector value and said overall optimized second vector value corresponds to an overall minimum value of said unregularized objective function among a plurality of values of said unregularized objective function, and wherein each value in the plurality of values of said unregularized objective function is obtained through minimizing said overall objective function.

11. One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause performance of a method for evaluating reporting window functions, the method comprising:

receiving target data comprising a plurality of observations, wherein each observation in said plurality of observation comprises (a) one or more values of one or more input neurons in a neural network and (b) one or more corresponding values of one or more output neurons in the neural network, and wherein the neural network comprises one or more layers of hidden neurons, one or more skip-layer connections between the one or more input neurons and the one or more output neurons, and one or more non-skip-layer connections between neurons of the neural network;

determining an overall objective function that comprises (a) a first part corresponding to linear regression, (b) a second part corresponding to an unregularized objective function of the neural network, and (c) a third part corresponding to a regularization of said unregularized objective function; and

determining, based at least in part on said target data and said overall objective function, an overall optimized first vector value of a first vector and an overall optimized second vector value of a second vector, wherein said first vector comprises a set of skip-layer weights for the one or more skip-layer connections and a set of output neuron biases, and wherein said second vector comprises a set of non-skip-layer weights for the one or more non-skip-layer connections.

12. The one or more media of claim 11, wherein said first part of said objective function comprises a L2 norm as multiplied by a complement of an alpha factor, and wherein said second part of said objective function comprises an unregularized neural network objective function as multiplied by said alpha factor.

13. The one or more media of claim 11, wherein said third part of said objective function represents a quadratic regularization expression with the one or more non-skip-layer weights as arguments, as multiplied by a beta factor, and wherein said determining, based at least in part on said target data and said overall objective function, a first vector value of a first vector and a second vector value of a second vector comprises:

initializing said first vector to an incumbent first vector value determined with only said first part of said objective function;

initializing said second vector to an incumbent second vector value within bound box constraints represented by a lower bound L and an upper bound U;

estimating, based on said incumbent first vector value of said first vector and said incumbent second vector value of said second vector, a minimum eigenvalue of a Hessian matrix comprising second derivatives of said unregularized objective function in said second part of said overall objective function;

determining, based at least in part on said minimum eigenvalue of said Hessian matrix, an upper bound for said beta factor in said third part of said objective function;

determining a plurality of vector value combinations of said first vector and said second vector by minimizing, based on said target data without de-trending preprocessing, said overall objective function for a plurality of distinct factor value combinations of said alpha factor and said beta factor;

wherein each distinct factor value combination in the plurality of distinct factor value combinations comprises a distinct combination of an alpha factor value of said alpha factor between zero (0) and one (1), inclusive, and a beta factor value of said beta factor between zero (0) and said upper bound of said beta factor, inclusive;

wherein each vector value combination in the plurality of vector values combinations comprises a first vector value of said first vector and a second vector value of said second vector; and

selecting, based on one or more selection criteria, a particular vector value combination among the plurality of vector value combinations as an overall optimized vector value combination, wherein said overall optimized vector value combination comprises said overall optimized first vector value and said overall optimized second vector value.

14. The one or more media of claim 13, wherein said minimum eigenvalue of said Hessian matrix is numerically estimated without an explicit functional form of said Hessian matrix in terms of weights and biases of the neural network.

15. The one or more media of claim 13, wherein said minimum eigenvalue of said Hessian matrix is computed from an explicit functional form of said Hessian matrix in terms of weights and biases of the neural network.

16. The one or more media of claim 11, wherein said unregularized objective function in said second part of said objective function is explicitly derived from (a) a configuration of the neural network and (b) a plurality of activation functions associated with the one or more hidden neurons and with the one or more output neurons.

17. The one or more media of claim 11, wherein said one or more non-skip layer weights were projected into bound box constraints represented by a lower bound vector L and an upper bound vector U.

18. The one or more media of claim 11, wherein said neural network is a feed-forward neural network.

19. The one or more media of claim 11, wherein at least one of a plurality of activation functions in said neural network is one of a tan, bipolarSigmoid, cos, gaussian, gompertz, linear, sigmoid, reciprocal, sin, square, or tan h.

20. The one or more media of claim 11, wherein said overall optimized first vector value and said overall optimized second vector value corresponds to an overall minimum value of said unregularized objective function among a plurality of values of said unregularized objective function, and wherein each value in the plurality of values of said unregularized objective function is obtained through minimizing said overall objective function.