Minimizing Global Error in an Artificial Neural Network

• US 20150088795A1
• Filed: 09/22/2014
• Published: 03/26/2015
• Est. Priority Date: 09/23/2013
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Aboyoun, Patrick, Golovashkin, Dmitry
• Granted Patent: US 10,068,170 B2
• Time in Patent Office: Days
• US Class Current: 706/21
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

Computer systems, machine-implemented methods, and stored instructions are provided for minimizing an approximate global error in an artificial neural network that is configured to predict model outputs based at least in part on one or more model inputs. A model manager stores the artificial neural network model. The model manager may then minimize an approximate global error in the artificial neural network model at least in part by causing evaluation of a mixed integer linear program that determines weights between artificial neurons in the artificial neural network model. The mixed integer linear program accounts for piecewise linear activation functions for artificial neurons in the artificial neural network model. The mixed integer linear program comprises a functional expression of a difference between actual data and modeled data, and a set of one or more constraints that reference variables in the functional expression.

Claims

1. A method comprising:

storing an artificial neural network model that is configured to predict one or more outputs based at least in part on one or more inputs, wherein the artificial neural network model comprises an input layer, one or more intermediate layers, and an output layer; and

minimizing an approximate global error in the artificial neural network model at least in part by causing evaluation of a mixed integer linear program that determines one or more weights between two or more artificial neurons in the artificial neural network model, wherein the mixed integer linear program accounts for one or more piecewise linear activation functions for one or more artificial neurons in the artificial neural network model;

wherein the method is performed by one or more computing devices.

2. The method of claim 1, further comprising replacing, in the artificial neural network model, at least one non-linear activation function with at least one piecewise linear step function.

3. The method of claim 1, further comprising replacing, in the artificial neural network model, at least one non-linear activation function with at least one continuous piecewise linear function.

4. The method of claim 1, further comprising replacing, in the artificial neural network model, at least one non-linear activation function with at least one piecewise linear function that includes three or more segments.

5. The method of claim 1, further comprising replacing, in the artificial neural network model, all of a plurality of non-linear activation functions with corresponding piecewise linear functions that approximate the non-linear activation functions.

6. The method of claim 1, further comprising:

creating the artificial neural network model based on known outputs, and

after minimizing the approximate global error in the artificial neural network model, using the artificial neural network model to predict one or more unknown outputs based at least in part on one or more known inputs.

7. The method of claim 1, wherein the one or more piecewise linear activation functions are non-differentiable and non-usable with an alternative gradient approach that, if used, would minimize local error in the artificial neural network model by iteratively improving the one or more weights.

8. The method of claim 1, wherein causing evaluation of the mixed integer linear program comprises branching a candidate set of weights into candidate sub-sets of weights and determining upper and lower bounds for the approximate global error using the candidate sub-sets of weights.

9. The method of claim 1, wherein causing evaluation of the mixed integer linear program comprises using one or more of a branch and cut technique, a cutting plane technique, a branch and price technique, a branch and bound technique, or a Lipschitzian optimization technique.

10. The method of claim 1, wherein the mixed integer linear program comprises a functional expression of a difference between actual data and modeled data, and a set of one or more constraints that reference two or more variables in the functional expression.

11. One or more non-transitory storage media storing instructions which, when executed by one or more processors, cause:

storing an artificial neural network model that is configured to predict one or more outputs based at least in part on one or more inputs, wherein the artificial neural network model comprises an input layer, one or more intermediate layers, and an output layer; and

minimizing an approximate global error in the artificial neural network model at least in part by causing evaluation of a mixed integer linear program that determines one or more weights between two or more artificial neurons in the artificial neural network model, wherein the mixed integer linear program accounts for one or more piecewise linear activation functions for one or more artificial neurons in the artificial neural network model.

12. The one or more non-transitory storage media of claim 11, the instructions further comprising instructions for replacing, in the artificial neural network model, at least one non-linear activation function with at least one piecewise linear step function.

13. The one or more non-transitory storage media of claim 11, the instructions further comprising instructions for replacing, in the artificial neural network model, at least one non-linear activation function with at least one continuous piecewise linear function.

14. The one or more non-transitory storage media of claim 11, the instructions further comprising instructions for replacing, in the artificial neural network model, at least one non-linear activation function with at least one piecewise linear function that includes three or more segments.

15. The one or more non-transitory storage media of claim 11, the instructions further comprising instructions for replacing, in the artificial neural network model, all of a plurality of non-linear activation functions with corresponding piecewise linear functions that approximate the non-linear activation functions.

16. The one or more non-transitory storage media of claim 11, the instructions further comprising instructions for:

creating the artificial neural network model based on known outputs, and

after minimizing the approximate global error in the artificial neural network model, using the artificial neural network model to predict one or more unknown outputs based at least in part on one or more known inputs.

17. The one or more non-transitory storage media of claim 11, wherein the one or more piecewise linear activation functions are non-differentiable and non-usable with an alternative gradient approach that, if used, would minimize local error in the artificial neural network model by iteratively improving the one or more weights.

18. The one or more non-transitory storage media of claim 11, wherein causing evaluation of the mixed integer linear program comprises branching a candidate set of weights into candidate sub-sets of weights and determining upper and lower bounds for the approximate global error using the candidate sub-sets of weights.

19. The one or more non-transitory storage media of claim 11, wherein causing evaluation of the mixed integer linear program comprises using one or more of a branch and cut technique, a cutting plane technique, a branch and price technique, a branch and bound technique, or a Lipschitzian optimization technique.

20. The one or more non-transitory storage media of claim 11, wherein the mixed integer linear program comprises a functional expression of a difference between actual data and modeled data, and a set of one or more constraints that reference two or more variables in the functional expression.