DATA SPLITTING BY GRADIENT DIRECTION FOR NEURAL NETWORKS

0Associated
Cases 
0Associated
Defendants 
0Accused
Products 
0Forward
Citations 
0
Petitions 
1
Assignment
First Claim
1. A method of generating an improved neural network, the method comprising:
 splitting first training data into N groups of training data, where N>
1, based on similarity of a gradient direction of an error cost function with respect to the first training data; and
training a base neural network with the first training data, wherein the base neural network comprises N subnetwork portions, and wherein each of the N subnetwork portions is trained on a respective one of the N groups of training data.
1 Assignment
0 Petitions
Accused Products
Abstract
Systems and methods improve the performance of a network that has converged such that the gradient of the network and all the partial derivatives are zero (or close to zero) by splitting the training data such that, on each subset of the split training data, some nodes or arcs (i.e., connections between a node and previous or subsequent layers of the network) have individual partial derivative values that are different from zero on the split subsets of the data, although their partial derivatives averaged over the whole set of training data is close to zero. The present system and method can create a new network by splitting the candidate nodes or arcs that diverge from zero and then trains the resulting network with each selected node trained on the corresponding cluster of the data. Because the direction of the gradient i s different for each of the nodes or arcs that are split, the nodes and their arcs in the new network will train to be different. Therefore, the new network is not at a stationary point.
0 Citations
No References
No References
51 Claims
 1. A method of generating an improved neural network, the method comprising:
splitting first training data into N groups of training data, where N>
1, based on similarity of a gradient direction of an error cost function with respect to the first training data; andtraining a base neural network with the first training data, wherein the base neural network comprises N subnetwork portions, and wherein each of the N subnetwork portions is trained on a respective one of the N groups of training data.  View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24)
 25. A computer system for generating an improved neural network, the computer system comprising one or more computers, wherein the one or more computers comprise at least one processor and associated memory, wherein the associated memory stores software that when executed by the at least one processor, causes the at least one processor to:
split first training data into N groups of training data, where N>
1, based on similarity of a gradient direction of an error cost function with respect to the first training data; andtrain a base neural network with the first training data, wherein the base neural network comprises N subnetwork portions, and wherein each of the N subnetwork portions is trained on a respective one of the N groups of training data.  View Dependent Claims (26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48)
 49. A computer system for generating an improved neural network, the computer system comprising:
a first set of one or more processing cores for pretraining a base neural network to a desired performance level; and a second set of one or more processing cores for; splitting first training data into N groups of training data, where N>
1, based on similarity of a gradient direction of an error cost function with respect to the first training data; andtraining the base neural network with the first training data, wherein the base neural network comprises N subnetwork portions, and wherein each of the N subnetwork portions is trained on a respective one of the N groups of training data.  View Dependent Claims (50, 51)
1 Specification
This PCT application claims priority to U.S. provisional patent application Ser. No. 62/516,785, filed Jun. 8, 2017, entitled “Data and Node Splitting by Gradient Direction,” with the same inventor as identified above, and which is incorporated herein by reference in its entirety.
Machine learning is a process implemented by computers to selflearn algorithms that can make predictions on data through building models from sample data inputs. There are many types of machine learning system, such as artificial neural networks (ANNs), decision trees, support vector machines (SVMs), and others. These systems first have to be trained on some of the sample inputs before making meaningful predictions with new data. For example, an ANN typically consists of multiple layers of neurons. Each neuron is connected with many others, and links can be enforcing or inhibitory in their effect on the activation state of connected neurons. Each individual neural unit may have a summation function which combines the values of all its inputs together. There may be a threshold function or limiting function on each connection and on the neuron itself, such that the signal must surpass the limit before propagating to other neurons. The weight for each respective input to a node can be trained by back propagation of the partial derivative of an error cost function, with the estimates being accumulated over the training data samples. A large, complex ANN can have millions of connections between nodes, and the weight for each connection has to be learned.
ANNs are trained on training data until they converge to a minimum in the error cost function. Once ANNs are trained to convergence, no techniques currently exist to further improve the performance of the network without making changes to the structure of the network. Furthermore, there is no systematic way to make changes in the network that will improve performance. This can be problematic because there is a constant desire to make ANNs perform more efficiently, especially as they grow larger and more complex.
In one general aspect, the present invention is directed to a computerimplemented systems and methods for improving the performance of a network that has converged such that the gradient of the network and all the partial derivatives are zero (or close to zero). In one embodiment, the method comprises splitting training data for the network into N groups of training data, where N>1, based on similarity of gradient direction. Then, the neural network is trained (or retrained), where each of N subnetwork portions of the neural network is trained on a respective one of the N groups of training data. The training data may be split such that, on each subset of the split training data, some nodes or arcs (i.e., connections between a node and previous or subsequent layers of the network) have individual partial derivative values that are different from zero on the split subsets of the data, although their partial derivatives averaged over the whole set of training data is close to zero. In particular embodiments of the present invention, a new, improved network is created by splitting one or more candidate nodes or arcs in the neural network that diverge from zero and then training the resulting network with each selected node trained on the corresponding group (or cluster) of the data.
Because the direction of the gradient is different for each of the nodes or arcs that are split, the nodes and their arcs in the new network will train to be different. Therefore, the new network is not at a stationary point, i.e., is not at a minimum. When the new network is trained from an initial point that matches the previously trained minimum, it can result in a reduction in the error cost function, thereby improving the performance of the network. The splitting of the data by gradient similarity is crucial. The node splitting by itself would not be nearly as effective because the gradients would still be close to zero.
These and other benefits from embodiments of the present invention will be apparent from the description that follows.
Various embodiments of the present invention are described herein by way of example in conjunction with the following figures, wherein:
The present disclosure describes a system and method for improving the performance and robustness of a neural network after that network has already been trained to convergence at or near a minimum in its error cost function. Neural networks are typically trained by an iterative process such as stochastic gradient descent. In stochastic gradient descent, the gradient is estimated on small batches of data called “minibatches.” The estimate of the gradient on each minibatch is only a random sample estimate of the gradient on the full set of training data. Therefore, the iterative training process does not fully converge in the mathematical sense, but merely gets close to a minimum in the cost function, with a small random fluctuation from one minibatch to another.
Even the gradient on the full set of training data is only an estimate of the gradient of the true function, so the error on the training data may not be the same as the error on independent validation data. Therefore, in iterative training of neural networks, there is usually a stopping criterion that stops the iterative training when further iterations appear to be providing no consistent improvement in performance. Thus, the gradient of the error cost function, which would be zero at the minimum of the error cost function, may merely be close to zero but not exactly zero at the point at which the training is terminated.
An iterative training process that has been terminated by such a stopping rule is said to be “converged,” although it is not fully converged in the mathematical sense. For the purpose of continued training, there is little practical difference between an iterative process that has mathematically converged to a stationary point, with a gradient of zero, and an iterative process with a stopping rule that has been stopped at a point close to such a stationary point. A slight random perturbation to a process that is exactly at a stationary point produces the same situation as a stopping rule, a point that is close to a local minimum. However, in both cases, continued learning will be slow because the magnitude of the gradient is very small and estimates of the gradient are noisy.
The present system and method can make improvements to the alreadytrained neural network even if the minimum that the neural network has been trained to is the global minimum. The present system and method selectively changes the structure of the trained neural network to create a new network or ensemble with a lower error cost and with greater robustness.
After the base network is trained, the process evaluates 102 each node in the base network as a candidate for splitting and then selects 103 the best candidates for splitting. Various embodiments of systems and methods for evaluating 102 and selecting 103 candidates for splitting will be discussed in further detail with reference to
Various embodiments can utilize any one of these norms or a combination thereof as the criterion for ranking the nodes for selection. In one embodiment, all of the nodes in the network are ranked according to the chosen technique and the nodes with the highest norms or all those with norms over a particular threshold value are selected. In another embodiment, only nodes in a single layer are evaluated, ranked, and selected at a time, since the further training of nodes in one layer will affect the partial derivatives of nodes and arcs in other layers. In yet another embodiment, only a single node is selected per application of the process.
Once a node or set of nodes has been selected, the selected data examples are clustered into two or more clusters. One vector to be clustered is obtained for each training example. For example, the vector could be the set of partial derivatives with respect to a set of nodes <∂aC(m)/∂a_{j}>, where m is for a single example and j ranges over a set of nodes. As another example, the vector could be the gradient with respect to the outgoing arcs or the incoming arcs for one or more selected nodes. For example, the vector for clustering can be a vector created by first computing a gradient vector for all the arcs leaving each selected node and then forming a longer vector by concatenating the vectors created for each of the selected nodes. In another illustrative embodiment, clustering is done separately for each selected node.
In an illustrative embodiment, the vectors are normalized to have unit length, because the direction of the gradient determines the subsequent further training to a larger degree than the length of the vector.
It is not essential that the clusters be well separated; rather, it is only necessary that at least two cluster centers be different enough to produce a difference during the further training of the network as modified. In other words, a data item or example does not have to be assigned to any of the data clusters, or it could be assigned to multiple clusters. Because the average gradient across all of the data is zero and the selected nodes have gradient norms that are significantly different from zero, it is easy to find two or more clusters that are significantly different, because their norms are nonzero but their average is zero. Any suitable clustering algorithm that generates sufficiently different clusters may be used. For example, in various embodiments, Kmeans clustering may be used. As another example, the vectors can be modeled as being drawn from a mixture of Gaussian distributions. The Gaussian mixture model can be trained, for example, by the wellknown expectationmaximization (EM) algorithm and each mixture component is a cluster.
As an illustrative embodiment of this invention, the clustering of the concatenated gradient vectors may be done by supervised or unsupervised training of an autoencoder with a softmax vector as its bottleneck layer, as shown in
The illustrative embodiment shown in
In the illustrative embodiment, the node splitting and subsequent training is determined by splitting the training data into the multiple groups, such as into the clusters by a clustering algorithm. More details about ways to split training data can be found in U.S. provisional application Ser. No. 62/518,302, entitled “Robust AntiAdversarial Machine Learning,” filed Jun. 12, 2017 and U.S. provisional application Ser. No. 62/623,773, entitled “SelfOrganizing Partially Ordered Networks,” filed Jan. 30, 2018, both of which are incorporated herein by reference in their entirety.
In various embodiments, each selected node is split into multiple nodes, one new node for each cluster, as illustrated in
Then, for some amount of subsequent training on the new network with the new nodes 202a, 202b, for each item of training data, all but one node in each new node set is dropped out of the backpropagation computation in various embodiments of the preset invention. The one node that is not dropped is the one corresponding to the cluster for the current data input item. For example, with reference to the example in
The gradient estimate for each new node, and for each arc connected to that node will only include data from a given cluster. Because the clustering groups together data with similar gradients, and it separates into different clusters data that have different gradient directions, the vector of weights for arcs connected to a given new nodes will move in subsequent training in a different direction from any of the other nodes in a set of new nodes. In other words, although the network with the new node splits is initialized with weights that are equivalent to the original network before the splits, in the new network this configuration of weights is not near a stationary point. In fact, the new weights among the arcs connected to the new modes will quickly diverge from each other.
In one illustrative embodiment, the different data selection for the node associated with each cluster is done only for the first epoch of further training. After that, all the data is used for back propagation for all the nodes, except for normal dropout, if any.
In another illustrative embodiment, the entire base network is cloned, creating an ensemble with a copy the original network for each cluster. Each copy of the network is then trained just on data from the corresponding cluster. For example, as illustrated in
In other embodiments, different subnetworks of a neural network could be trained with the different groups of data. For example,
For clarification purposes, with reference to
In some embodiments, a portion of the data from other clusters is mixed in with the data from the corresponding cluster.
As the new network is initialized identically to the old network, the new network'"'"'s performance is initially identical to the base network. But, unlike the base network, the new network is not at a stationary point in the further training, i.e., is not converged at or near a local minimum for its training set. It is the splitting of the data that has been selected for different gradient directions that causes the training process to no longer be at a stationary point. The new network has nodes and arcs whose gradients are significantly different from zero. Therefore, the stochastic gradient descent in the further training will lead to reduction in the error cost function C. Because the new network is initialized to match the original network, the subsequent training gives improved performance over the previous network that had already been trained to convergence.
There are methods well known to those skilled in the art of optimization by gradient descent to increase the likelihood that a training process will converge to a local minimum with especially good performance. For example, a simple technique is to independently run iterative training from many different randomly selected starting points and then choose the one with the best performance. One illustrative embodiment of the invention starts the data and node splitting process with such a chosen bestperformance network.
According to various embodiments, therefore, two innovations can combine to achieve an effect that superficially might seem to be impossible: a systematic process for improving the performance of a network that is already achieving its optimum performance. The node selection process provides a guide for selecting changes to the network to an expanded network that is capable of better performance than the optimum performance of the original network. The data splitting and specialized subnetwork portion training (e.g., nondropped out nodes) allows a gradient descent process to start with performance matching the previous optimum performance, yet have gradients with magnitudes that are substantially different from zero.
The node splitting by itself would not be as effective because the gradients of the error cost function would still be close to zero.
The data clustering and splitting has the further beneficial effect that the nodes in a new set of split nodes will tend to learn different features from each other. This beneficial effect also occurs in an illustrative embodiment in which the node splitting is used to create an ensemble of new networks.
The process of evaluating, splitting, and further training the network can be utilized iteratively to continue to improve the network as long as there are some nodes or arcs whose gradients are significantly different from zero on at least some data examples. Once the gradients of the individual data examples are all close to zero, the network will be stable and robust, because any small change in the input or in any of the activations of inner nodes will produce only a small change in the output as the partial derivatives will not only be zero when averaged across the training data, but also for each individual data example.
The illustrative systems and methods described herein, including the system for splitting a network according to the direction of the gradient described in
Each processor could have one or multiple cores. The cores could be CPU, graphical processing unit (GPU) cores, and/or AI accelerators, for example. For example, in an embodiment with multiple CPU cores, one set of cores could execute the program instructions for training the base neural network, another set for evaluating each node for gradient spread, and so on. GPU cores operate in parallel and, hence, can typically process data more efficiently that a collection of CPU cores, but all the cores execute the same code at one time. An AI accelerator is a class of microprocessor designed to accelerate artificial neural networks, which have hundreds or thousands of parallel, relatively lowprecision, processing units (or cores). The memory unit(s) may comprise computer memory that is accessible by the processing cores, such as RAM, ROM, processor registers or processor cache, for example.
As shown in
Embodiments of the present invention can be used to improve many different types of machine learning systems, including deep neural networks, in a variety of applications. For example, embodiments of the present invention can improve recommender systems, speech recognition systems, and classification systems, including image and diagnostic classification systems, to name but a few examples.
In one general aspect, therefore, the present invention is directed to methods and systems for generating an improved neural network. A method of generating an improved neural network according to embodiments of the present invention may comprise splitting first training data into N groups of training data, where N>1, based on similarity of gradient direction; and training a base neural network with the first training data, wherein the base neural network comprises N subnetwork portions, and wherein each of the N subnetwork portions is trained on a respective one of the N groups of training data. A computer system for generating an improved neural network according to embodiments of the present invention may comprise one or more computers, where the one or more computers comprise at least one processor and associated memory, where the associated memory stores software that when executed by the at least one processor, causes the at least one processor to: split first training data into N groups of training data, where N>1, based on similarity of gradient direction; and train a base neural network with the first training data, wherein the base neural network comprises N subnetwork portions, and wherein each of the N subnetwork portions is trained on a respective one of the N groups of training data.
According to various implementations, prior to training the base neural network, a first node of the base neural network is split into a node set that comprises N new nodes, such that one new node in the node set corresponds to one and only one of each of the N groups of training data. In that case, the step of training the base neural network comprises, for each data item in the first training data that belongs to one of the N groups of training data, dropping from the training for that data item all nodes in the node set that do not correspond to the group for the data item. In such an embodiment, the first node, prior to splitting, may comprise one or more incoming arcs and one or more outgoing arcs, with each of the incoming and outgoing arcs of the first node, prior to splitting, has a respective weight. In that case, splitting the first node into the new node set may comprise: having each new node in the new node set have the same number of incoming and outgoing arcs as the first node prior to splitting; and initializing the incoming and outgoing arcs of each new node in the new node set with equivalent weights as the first node prior to splitting, such that after the training, the weights for the incoming and outgoing arcs of each new node N in the new node set diverge. Also, training the base neural network may comprise, after dropping nodes for the first training data, training the base neural network, with the new node set, without dropping nodes for second training data that is different from the first training data.
In yet other implementations, prior to training the base neural network, an ensemble of N ensemble member networks is generated, where each of the N ensemble members is identical to the base neural network, and such that one ensemble member corresponds to one and only one of each of the N groups of training data. In that case, the step of training the base neural network may comprise training the ensemble, wherein, for each data item in the first training data that belongs to one of the N groups of training data, training the ensemble comprises dropping from the training for that data item all ensemble members in the ensemble that do not correspond to the group for the data item. In such embodiments, training the ensemble may comprise, after dropping ensemble members for the first training data, training the ensemble without dropping ensemble members for second training data that is different from the first training data
In various implementations, prior to training the base neural network with the first training data, the base neural network is pretraining to a desired performance level, such as to convergence, which may comprise iteratively pretraining the base neural network through gradient descent until a gradient of an applicable error cost function is below a threshold minimum.
In addition, splitting the first training data into N groups may comprise clustering the first training data into N clusters. This may be done by generating a vector for each training example in the first training data and clustering the vectors into N clusters. The clustering may also be performed with a clustering algorithm and/or by training an autoencoder with a bottleneck layer.
The software for the various machine learning systems described herein, such as the software modules shown in
The examples presented herein are intended to illustrate potential and specific implementations of the present invention. It can be appreciated that the examples are intended primarily for purposes of illustration of the invention for those skilled in the art. No particular aspect or aspects of the examples are necessarily intended to limit the scope of the present invention. Further, it is to be understood that the figures and descriptions of the present invention have been simplified to illustrate elements that are relevant for a clear understanding of the present invention, while eliminating, for purposes of clarity, other elements. While various embodiments have been described herein, it should be apparent that various modifications, alterations, and adaptations to those embodiments may occur to persons skilled in the art with attainment of at least some of the advantages. The disclosed embodiments are therefore intended to include all such modifications, alterations, and adaptations without departing from the scope of the embodiments as set forth herein.