System and method for hyper-parameter analysis for multi-layer computational structures

• US 10,534,994 B1
• Filed: 11/11/2015
• Issued: 01/14/2020
• Est. Priority Date: 11/11/2015
• Status: Active Grant

• Current Assignee: Cadence Design Systems Incorporated (Cadence Group)
• Original Assignee: Cadence Design Systems Incorporated (Cadence Group)
• Inventors: Kaul, Piyush, Hijazi, Samer Lutfi, Casas, Raul Alejandro, Kumar, Rishi, Mao, Xuehong, Rowen, Christopher
• Primary Examiner(s): Smith, Paulinho E
• Application Number:
  US14/938,370
• Time in Patent Office: 1,525 Days
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent

Abstract

The present disclosure relates to a computer-implemented method for analyzing one or more hyper-parameters for a multi-layer computational structure. The method may include accessing, using at least one processor, input data for recognition. The input data may include at least one of an image, a pattern, a speech input, a natural language input, a video input, and a complex data set. The method may further include processing the input data using one or more layers of the multi-layer computational structure and performing matrix factorization of the one or more layers. The method may also include analyzing one or more hyper-parameters for the one or more layers based upon, at least in part, the matrix factorization of the one or more layers.

Claims

1. A computer-implemented method for analyzing hyper-parameters of a multi-layer computational structure comprising:

accessing, using at least one processor, input data for recognition, wherein the input data includes at least one of an image, a pattern, a speech input, a natural language input, a video input, and a complex data set;

processing the input data using one or more layers of the multi-layer computational structure;

performing matrix factorization of the one or more layers of the multi-layer computational structure;

analyzing one or more hyper-parameters of the one or more layers based upon, at least in part, the matrix factorization of the one or more layers;

training one or more filters of the one or more layers; and

converting one or more trained filters of the one or more layers to a plurality vectors.

2. The computer-implemented method of claim 1, further comprising:

generating a covariance matrix using the plurality of vectors.

3. The computer-implemented method of claim 2, wherein performing matrix factorization includes determining an amount of energy retained for one or more basis weight values of the one or more layers based upon, at least in part, the covariance matrix.

4. The computer-implemented method of claim 3, wherein analyzing one or more hyper-parameters of the one or more layers is further based upon, at least in part, whether the amount of energy retained exceeds an energy threshold.

5. The computer-implemented method of claim 4, further comprising:

receiving a complexity target and adjusting the energy threshold until the complexity target is achieved.

6. The computer-implemented method of claim 1, wherein analyzing one or more hyper-parameters of the one or more layers is performed iteratively for each of the one or more layers.

7. The computer-implemented method of claim 6, wherein analyzing one or more hyper-parameters of the one or more layers includes at least one of estimating, changing, and reducing a number of feature maps.

8. The computer-implemented method of claim 1, wherein the multi-layer computational structure is one or more of a neural network with weights, a convolutional neural network, a deep belief network, a recurrent neural network and an autoencoder.

9. The computer-implemented method of claim 1, wherein analyzing one or more hyper-parameters of the one or more layers is further based upon, at least in part, one or more of balancing a computational load between the one or more layers of the multi-layer computational structure, reducing over-fitting, estimating an implementation cost, and improving a detection performance.

10. The computer-implemented method of claim 1, wherein the one or more hyper-parameters includes one or more of a number of feature maps for each of the one or more layers and a number of weights for each of the one or more layers.

11. The computer-implemented method of claim 1, further comprising:

Retraining the one or more filters of the one of more layers, based upon, at least in part, the analyzing of the one or more hyper-parameters of the one or more layers.

12. The computer-implemented method of claim 11, wherein retraining the one or more layers is performed iteratively.

13. The computer-implemented method of claim 11, wherein retraining one or more layers is one or more of a partial retraining of each layer of the one or more layers and a complete retraining of each layer of the one or more layers.

14. The computer-implemented method of claim 1, wherein the multi-layer computational structure includes at least one of:

one or more pooling layers, one or more non-linear functions, one or more convolution layers with uniform filters and one or more convolutional layers with non-uniform filters.

15. The computer-implemented method of claim 1, wherein the multi-layer computational structure includes a plurality of hybrid layers wherein the feature maps of each of the plurality of hybrid layers is associated with one or more different feature maps of one or more previous layers.

16. A system for analyzing the optimal number of feature maps for a multi-layer computational structure comprising:

a computing device having at least one processor configured to receive input data for recognition, wherein the input data includes at least one of an image, a pattern, a speech input, a natural language input, a video input, and a complex data set, the at least one processor further configured to process the input data using one or more layers of the multi-layer computational structure, the at least one processor further configured to perform matrix factorization of the one or more layers, and the at least one processor further configured to analyze one or more hyper-parameters for the one or more layers based upon, at least in part, the matrix factorization of the one or more layers, wherein the at least one processor is further configured to train one or more filters from the one or more layers and wherein the at least one processor is further configured to retrain the one or more filters of the one or more layers, based upon, at least in part, the analyzing of the one or more hyper-parameters of the one or more layers.

17. The system of claim 16, wherein the at least one processor is further configured to convert the one or more filters to a plurality vectors.

18. The system of claim 17, wherein the at least one processor is further configured to generate a covariance matrix using the plurality of vectors.

19. The system of claim 18, wherein performing matrix factorization includes determining an amount of energy retained for one or more basis weight values of the one or more layers based upon, at least in part, the covariance matrix.

20. The system of claim 19, wherein analyzing one or more hyper-parameters for the one or more layers is further based upon, at least in part, whether the amount of energy retained exceeds an energy threshold.

21. The system of claim 20, wherein the at least one processor is further configured to receive a complexity target and adjusting the energy threshold until the complexity target is achieved.

22. The computer-implemented method of claim 16, wherein analyzing one or more hyper-parameters for the one or more layers is performed iteratively for each of the one or more layers.

23. The computer-implemented method of claim 22, wherein analyzing one or more hyper-parameters includes at least one of estimating, changing, and reducing a number of feature maps.

24. The system of claim 16, wherein the multi-layer computational structure is one or more of a neural network with weights, a convolutional neural network, a deep belief network, a recurrent neural network and an autoencoder.

25. The system of claim 16, wherein analyzing the optimal number of feature maps for the one or more layers is further based upon, at least in part, one or more of balancing a computational load between the one or more layers of the multi-layer computational structure, reducing overfitting, and improving detection performance.

26. The system of claim 16, wherein the one or more hyper-parameters includes one or more of a number of feature maps for each of the one or more layers and a number of weights for each of the one or more layers.

27. The system of claim 16, wherein retraining the one or more layers is performed iteratively.

28. The system of claim 16, wherein retraining one or more layers is one or more of a partial retraining of each layer of the one or more layers and a complete retraining of each layer of the one or more layers.

29. The system of claim 16, wherein the multi-layer computational structure includes at least one of:

one or more pooling layers, one or more non-linear functions, one or more convolution layers with uniform filters and one or more convolutional layers with non-uniform filters.

30. The system of claim 16, wherein the multi-layer computational structure includes a plurality of hybrid layers wherein the feature maps of each of the plurality of hybrid layers is associated with one or more different feature maps of one or more previous layers.

31. A non-transitory computer-readable storage medium for analyzing hyper-parameters of a multi-layer computational structure, the computer-readable storage medium having stored thereon instructions that when executed by a machine result in the following operations:

accessing input data for recognition, wherein the input data includes at least one of an image, a pattern, a speech input, a natural language input, a video input, and a complex data set;

processing the input data using one or more layers of the multi-layer computational structure;

performing matrix factorization of the one or more layers;

analyzing one or more hyper-parameters of the one or more layers based upon, at least in part, the matrix factorization of the one or more layers; and

wherein the multi-layer computational structure includes a plurality of hybrid layers wherein the feature maps of each of the plurality of hybrid layers is associated with one or more different feature maps of one or more previous layers.

32. The computer-readable storage medium of claim 31, wherein operations further comprise:

training one or more filters from the one or more layers.

33. The computer-readable storage medium of claim 32, wherein operations further comprise:

converting the one or more filters of the first layer to a plurality of vectors.

34. The computer-readable storage medium of claim 33, wherein operations further comprise:

generating a covariance matrix using the plurality of vectors.

35. The computer-readable storage medium of claim 34, wherein performing matrix factorization includes determining an amount of energy retained for one or more basis weight values of the one or more layers based upon, at least in part, the covariance matrix.

36. The computer-readable storage medium of claim 35, wherein analyzing one or more hyper-parameters for the one or more layers is further based upon, at least in part, whether the amount of energy retained exceeds an energy threshold.

37. The computer-readable storage medium of claim 36, wherein operations further comprise:

receiving a complexity target and adjusting the energy threshold until the complexity target is achieved.

38. The computer-readable storage medium of claim 32, wherein operations further comprise:

retraining the one or more filters of the one or more layers, based upon, at least in part, the analyzing of the one or more hyper-parameters of the one or more layers.

39. The computer-readable storage medium of claim 38, wherein the retraining one or more layers is performed iteratively.

40. The computer-readable storage medium of claim 38, wherein retraining one or more layers is one or more of a partial retraining of each layer of the one or more layers and a complete retraining of each layer of the one or more layers.

41. The computer-readable storage medium of claim 31, wherein analyzing one or more hyper-parameters for the one or more layers is performed iteratively for each of the one or more layers.

42. The computer-readable storage medium of claim 41, wherein analyzing one or more hyper-parameters includes at least one of estimating, changing, and reducing a number of feature maps.

43. The computer-readable storage medium of claim 31, wherein the multi-layer computational structure is one or more of a neural network, a convolutional neural network, a deep belief network, a recurrent neural network and an autoencoder.

44. The computer-readable storage medium of claim 31, wherein analyzing the optimal number of feature maps for the one or more layers is further based upon, at least in part, one or more of balancing a computational load between the one or more layers of the multi-layer computational structure, reducing overfitting, and improving detection performance.

45. The computer-readable storage medium of claim 31, wherein the one or more hyper-parameters includes one or more of a number of feature maps for each of the one or more layers and a number of weights for each of the one or more layers.

46. The computer-readable storage medium of claim 31, wherein the multi-layer computational structure includes at least one of:

one or more pooling layers, one or more non-linear functions, one or more convolution layers with uniform filters and one or more convolutional layers with non-uniform filters.