Hybrid neural network generation system and method

• US 7,340,440 B2
• Filed: 12/08/2006
• Issued: 03/04/2008
• Est. Priority Date: 04/06/2001
• Status: Expired due to Term

• Current Assignee: SAS Institute Incorporated
• Original Assignee: SAS Institute Incorporated
• Inventors: Hartmann, Wolfgang Michael, Brocklebank, John C., Goodnight, James Howard
• Primary Examiner(s): Hirl; Joseph P
• Application Number:
  US11/636,322
• Publication Number:
  US 20070094176A1
• Time in Patent Office: 452 Days
• Field of Search: 706/19, 706/15, 706/14, 706/12, 704/270
• US Class Current: 706/19
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

A computer-implemented method and system for building a neural network is disclosed. The neural network predicts at least one target based upon predictor variables defined in a state space. First, an input data set is retrieved that includes the predictor variables and at least one target associated with the predictor variables for each observation. In the state space, a number of points is inserted in the state space based upon the values of the predictor variables. The number of points is less than the number of observations. A statistical measure is determined that describes a relationship between the observations and the inserted points. Weights and activation functions of the neural network are determined using the statistical measure.

Claims

1. An artificial neural network that predicts at least one target based upon observations defined in a state space, comprising:

a first stage that contains a first activation function type, wherein the first stage is predictive of the target, wherein residuals result from predictions by the first stage of the target; and

a second stage that contains a second activation function type, wherein the second stage is predictive of the residuals resulting from the predictions by the first stage;

wherein the first activation function type is a different function type than the second activation function type;

whereby the second activation function type in the second stage provides greater accuracy in predicting the target than if the first activation function type were used in the second stage to provide predictions with respect to the target.

2. The artificial neural network of claim 1 wherein the first activation function type is a nonlinear function.

3. The artificial neural network of claim 2 wherein the first activation function type includes at least one parameter that is optimized using a nonlinear optimization technique.

4. The artificial neural network of claim 3 wherein the nonlinear optimization technique is the Levenberg-Marquardt optimization.

5. The artificial neural network of claim 1 further comprising an orthogonal layer that calculates an uncorrelated input data set from the observations.

6. The artificial neural network of claim 5 wherein the orthogonal layer determines principal components using principal component analysis.

7. The artificial neural network of claim 6 wherein the orthogonal layer computes a subset of the principal components to determine the input to the first stage.

8. The artificial neural network of claim 7 wherein the subset of principal components is determined by maximizing a measure of association among a set of principal components and the target.

9. The artificial neural network of claim 8 wherein the subset of principal components includes the principal components for which a sum of square errors of the linear regression is minimum.

10. The artificial neural network of claim 8 wherein the subset of principal components includes the principal components for which an accuracy rate is maximum.

11. The artificial neural network of claim 1 wherein the first activation function type is determined from a statistical measure of the observations and the target.

12. The artificial neural network of claim 1 wherein the statistical measure is a frequency table populated by the observations.

13. The artificial neural network of claim 1 wherein a function type includes a square activation function type, a sine activation function type, a tanh activation function type, arctan activation function type, a logistic activation function type, an exp activation function type, a cos activation function type, or a Gaussian activation function type;

whereby a sine activation function type is a different function type than a square activation function type.

14. The artificial neural network of claim 1 wherein the first activation function type differs from the second activation function type based upon a differing type of function and not based upon parameters being different between the first activation function type and the second activation function type.

15. The artificial neural network of claim 1 wherein the first stage contains parameters of activation functions that were determined by optimizing the activation functions with respect to a predetermined objective function.

16. The artificial neural network of claim 1 further comprising:

a third stage that contains a third activation function type, wherein the third stage is predictive of the residuals resulting from predictions by the second stage.

17. The artificial neural network of claim 16 wherein the third activation function type is a different function type than the second activation function type.

18. The artificial neural network of claim 17 wherein the third activation function type is a different function type than the first activation function type.

19. The artificial neural network of claim 18 wherein the third activation function type differs from the second activation function type based upon a differing type of function and not based upon parameters being different between activation functions that are in different stages of the artificial neural network.

20. The artificial neural network of claim 1 wherein the neural network is constructed in a layer-by-layer stepwise fashion such that the first layer of the neural network is completed before the second layer of the neural network is determined;

wherein the type of activation function for the second layer is unknown until after the first layer has been completed.

21. A method for using an artificial neural network that predicts at least one target based upon observations defined in a state space, comprising:

using a first stage that contains a first activation function type, wherein the first stage is predictive of the target, wherein residuals result from predictions by the first stage of the target; and

using a second stage that contains a second activation function type, wherein the second stage is predictive of the residuals resulting from the predictions by the first stage;

wherein the first activation function type is a different function type than the second activation function type;

whereby the second activation function type in the second stage provides greater accuracy in predicting the target than if the first activation function type were used in the second stage to provide predictions with respect to the target.

22. A computer-readable medium containing computer software comprising program code for carrying out a method for using an artificial neural network that predicts at least one target based upon observations defined in a state space, said method comprising:

using a first stage that contains a first activation function type, wherein the first stage is predictive of the target, wherein residuals result from predictions by the first stage of the target; and

using a second stage that contains a second activation function type, wherein the second stage is predictive of the residuals resulting from the predictions by the first stage;

wherein the first activation function type is a different function type than the second activation function type;

whereby the second activation function type in the second stage provides greater accuracy in predicting the target than if the first activation function type were used in the second stage to provide predictions with respect to the target.