STAGED TRAINING OF NEURAL NETWORKS FOR IMPROVED TIME SERIES PREDICTION PERFORMANCE

• US 20200133977A1
• Filed: 12/26/2019
• Published: 04/30/2020
• Est. Priority Date: 02/05/2016
• Status: Active Grant

• Current Assignee: SAS Institute Incorporated
• Original Assignee: SAS Institute Incorporated
• Inventors: Bequet, Henry Gabriel Victor, Rioux, Jacques, Izquierdo, John Alejandro, Chen, Huina, Du, Juan
• Granted Patent: US 10,740,395 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/9014 hash tables
• CPC Class Codes: G06F 16/903 Querying for retrieval from the web G06F16/953
• CPC Class Codes: G06F 16/90344 by using string matching techniques
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: H04L 41/16 using machine learning or artificial intelligence
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

An apparatus includes a processor to: train a first neural network of a chain to generate first configuration data including first trained parameters, wherein the chain performs an analytical function generating a set of output values from a set of input values, each neural network has inputs to receive the set of input values and outputs to output a portion of the set of output values, and the neural networks are ordered from the first at the head to a last neural network at the tail, and are interconnected so that each neural network additionally receives the outputs of a preceding neural network; train, using the first configuration data, a next neural network in the chain ordering to generate next configuration data including next trained parameters; and use at least the first and next configuration data and data indicating the interconnections to instantiate the chain to perform the analytical function.

Claims

1. An apparatus comprising a processor and a storage to store instructions that, when executed by the processor, cause the processor to perform operations comprising:

train a first neural network of a chain of neural networks to generate a first portion of multiple portions of time series data that corresponds to a temporally earliest subrange of time of multiple subranges of time within a full range of time that is covered by the time series data, wherein;

the chain comprises a set of neural networks ordered to start with the first neural network at a head of the chain and to end with a last neural network at a tail of the chain;

each neural network of the chain comprises external inputs, additional inputs and outputs;

each neural network of the chain generates a portion of the multiple portions of the time series data at the outputs of the neural network from input data values provided at the external inputs of the neural network;

each portion of the multiple portions of the time series data corresponds to a subrange of the multiple subranges; and

the set of neural networks is interconnected within the chain such that each neural network, except the first neural network at the head of the chain, receives, at the additional inputs of the neural network, a portion of the multiple portions of the time series data that is generated at the outputs of a preceding neural network in the ordering of neural networks within the chain;

retrieve, from the first neural network, a first neural network configuration data comprising hyperparameters and first trained parameters learned by the first neural network from the training of the first neural network;

train, using at least the first neural network configuration data, a next neural network in the ordering of neural networks within the chain to generate a next portion of the multiple portions that corresponds to a next subrange of time of the multiple subranges of time that temporally follows the earliest subrange;

retrieve, from the next neural network, a next neural network configuration data comprising the hyperparameters and next trained parameters learned by the next neural network from the training of the next neural network; and

use at least the first neural network configuration data and the next neural network configuration data to instantiate the chain.

2. The apparatus of claim 1, wherein:

each neural network of the chain has the same quantities of external inputs, additional inputs and outputs;

during the training of each neural network of the chain, each additional input of the neural network that is not used to receive a portion of time series data from an output of a preceding neural network in the ordering of neural networks within the chain is provided with a null input.

following instantiation of the chain, the processor is caused to operate the chain to generate the time series data from the set of input data values; and

during operation of the chain, each additional input of each neural network of the chain that is not used to receive a portion of a time series data from an output of a preceding neural network in the ordering of neural networks within the chain is provided with a null input.

3. The apparatus of claim 1, wherein the processor is caused to sequentially train each neural network in the chain via backpropagation following the ordering of the chain from the first neural network to the last neural network, wherein:

each neural network is trained using a training data set comprising sets of input values and corresponding sets of output values;

each set of output values is generated as time series data from the corresponding set of input values through use of non-neuromorphic processing;

the sets of input values of the training data are provided to the external inputs of each neural network, and separate portions of the corresponding sets of output values are provided to the outputs of each neural network;

each of the separate portions of the corresponding sets of output values that are provided to the outputs of each neural network corresponds to the portion of the time series data that is to be generated by the neural network; and

during the training of each neural network in the chain other than the first neural network at the head of the chain, the preceding neural network is operated to generate, from each set of input values of the training data, a corresponding portion of time series data that is provided at the additional inputs of the neural network.

4. The apparatus of claim 3, wherein the processor is caused to perform operations comprising:

analyze the neural network training data to identify a portion of the output data values across the sets of output values of the neural network training data that shows a relatively high degree of correlation; and

derive a manner of dividing the time series data into the multiple portions of the time series data that are each output by one of the of the neural networks in the chain based, at least in part, on the identified portion of the output data values that shows the relatively high degree of correlation.

5. The apparatus of claim 1, wherein:

the multiple portions of time series data are implemented using the hyperparameters, and are based on a division of the full range of time into the multiple subranges of time that was derived prior to the training of the first neural network; and

the processor is caused, following the training of the first neural network and before the use of the first neural network configuration data to train the next neural network, to perform operations comprising;

test the first neural network using a testing data set comprising sets of input values and corresponding sets of output values;

analyze results of the testing of the first neural network to determine whether a degree of accuracy of the first neural network in generating the first portion of time series data meets a first threshold degree of accuracy; and

in response to determination that the degree of accuracy of the first neural network does not meet the first threshold degree of accuracy, perform operations comprising;

derive a different division of the full range of time into the multiple subranges of time;

alter the hyperparameters to implement the multiple portions of time series data based on the different division of the full range of time; and

repeat the training and testing of the first neural network following the alteration of the hyperparameters.

6. The apparatus of claim 5, wherein, in response to a determination that the degree of accuracy of the first neural network does meet the first threshold degree of accuracy, the processor is caused to perform operations comprising:

use the first neural network configuration data to train the next neural network;

test the next neural network using the test data set;

analyze the results of the testing of the next neural network to determine whether a degree of accuracy of the next neural network in generating the next portion of time series data meets a second threshold degree of accuracy; and

in response to a determination that the degree of accuracy of the next neural network does not meet the second threshold degree of accuracy, perform operations comprising;

derive the different division of the full range of time into the multiple subranges of time;

alter the hyperparameters to implement the multiple portions of time series data based on the different division of the full range of time; and

repeat the training and testing of at least the first neural network following the alteration of the hyperparameters.

7. The apparatus of claim 1, wherein:

the hyperparameters specify at least a quantity of artificial neurons within each neural network of the chain and a quantity of layers of artificial neurons within each neural network of the chain;

the quantity of layers includes an input layer of artificial neurons connected to the external inputs and the additional inputs; and

the quantity of layers includes an output layer of artificial neurons connected to the outputs.

8. The apparatus of claim 7, wherein the chain comprises a type of chain selected from a group consisting of:

a single-link chain, wherein;

each neural network in the chain, except the first neural network at the head of the chain, receives the portion of the multiple portions of time series data that is generated at the outputs of the immediately preceding neural network in the ordering of neural networks within the chain; and

the quantity of additional inputs enables each neural network, except the first neural network at the head of the chain, to receive all of the outputs of the immediately preceding neural network at its additional inputs; and

a multi-link chain, wherein;

each neural network in the chain, except the first neural network at the head of the chain, receives all of the multiple portions of time series data that are generated at the outputs of all of the preceding neural networks in the ordering of neural networks within the chain; and

the quantity of additional inputs enables the last neural network to receive all of the outputs of all of the other neural networks in the chain.

9. The apparatus of claim 1, wherein:

the trained parameters of the first neural network configuration data comprise weights and biases that represent what was learned by the first neural network during training; and

the processor is caused to train the set of neural networks sequentially in an order that follows the ordering of neural networks in the chain from the first neural network at the head of the chain to the last neural network at the tail of the chain.

10. The apparatus of claim 1, comprising a plurality of neuromorphic devices communicatively coupled to the processor, wherein the processor is caused to perform operations comprising:

derive the portion of the multiple portions of time series data that is output by each neural network from the input data values during operation of the chain to generate time series data based on at least one of;

a quantity of artificial neurons within each neuromorphic device of the plurality of neuromorphic devices;

a maximum quantity of layers that each neuromorphic device of the plurality of neuromorphic devices is able to support;

a maximum quantity of inputs that each neuromorphic device of the plurality of neuromorphic devices is able to support;

or

a maximum quantity of outputs that each neuromorphic device of the plurality of neuromorphic devices is able to support;

provide the first neural network configuration data to at least a first neuromorphic device of the plurality of neuromorphic devices to instantiate the first neural network;

provide the next neural network configuration data to at least a second neuromorphic device of the plurality of neuromorphic devices to instantiate the second neural network; and

provide a last neural network configuration data to at least a third neuromorphic device of the plurality of neuromorphic devices to instantiate the last neural network.

11. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause a processor to perform operations comprising:

train a first neural network of a chain of neural networks to generate a first portion of multiple portions of time series data that corresponds to a temporally earliest subrange of time of multiple subranges of time within a full range of time that is covered by the time series data, wherein;

the chain comprises a set of neural networks ordered to start with the first neural network at a head of the chain and to end with a last neural network at a tail of the chain;

each neural network of the chain comprises external inputs, additional inputs and outputs;

each neural network of the chain generates a portion of the multiple portions of the time series data at the outputs of the neural network from input data values provided at the external inputs of the neural network;

each portion of the multiple portions of the time series data corresponds to a subrange of the multiple subranges; and

the set of neural networks is interconnected within the chain such that each neural network, except the first neural network at the head of the chain, receives, at the additional inputs of the neural network, a portion of the multiple portions of the time series data that is generated at the outputs of a preceding neural network in the ordering of neural networks within the chain;

retrieve, from the first neural network, a first neural network configuration data comprising hyperparameters and first trained parameters learned by the first neural network from the training of the first neural network;

train, using at least the first neural network configuration data, a next neural network in the ordering of neural networks within the chain to generate a next portion of the multiple portions that corresponds to a next subrange of time of the multiple subranges of time that temporally follows the earliest subrange;

retrieve, from the next neural network, a next neural network configuration data comprising the hyperparameters and next trained parameters learned by the next neural network from the training of the next neural network; and

use at least the first neural network configuration data and the next neural network configuration data to instantiate the chain.

12. The computer-program product of claim 11, wherein:

each neural network of the chain has the same quantities of external inputs, additional inputs and outputs;

during the training of each neural network of the chain, each additional input of the neural network that is not used to receive a portion of time series data from an output of a preceding neural network in the ordering of neural networks within the chain is provided with a null input.

following instantiation of the chain, the processor is caused to operate the chain to generate the time series data from the set of input data values; and

during operation of the chain, each additional input of each neural network of the chain that is not used to receive a portion of a time series data from an output of a preceding neural network in the ordering of neural networks within the chain is provided with a null input.

13. The computer-program product of claim 11, wherein the processor is caused to sequentially train each neural network in the chain via backpropagation following the ordering of the chain from the first neural network to the last neural network, wherein:

each neural network is trained using a training data set comprising sets of input values and corresponding sets of output values;

each set of output values is generated as time series data from the corresponding set of input values through use of non-neuromorphic processing;

the sets of input values of the training data are provided to the external inputs of each neural network, and separate portions of the corresponding sets of output values are provided to the outputs of each neural network;

each of the separate portions of the corresponding sets of output values that are provided to the outputs of each neural network corresponds to the portion of the time series data that is to be generated by the neural network; and

during the training of each neural network in the chain other than the first neural network at the head of the chain, the preceding neural network is operated to generate, from each set of input values of the training data, a corresponding portion of time series data that is provided at the additional inputs of the neural network.

14. The computer-program product of claim 13, wherein the processor is caused to perform operations comprising:

analyze the neural network training data to identify a portion of the output data values across the sets of output values of the neural network training data that shows a relatively high degree of correlation; and

derive a manner of dividing the time series data into the multiple portions of the time series data that are each output by one of the of the neural networks in the chain based, at least in part, on the identified portion of the output data values that shows the relatively high degree of correlation.

15. The computer-program product of claim 11, wherein:

the multiple portions of time series data are implemented using the hyperparameters, and are based on a division of the full range of time into the multiple subranges of time that was derived prior to the training of the first neural network; and

the processor is caused, following the training of the first neural network and before the use of the first neural network configuration data to train the next neural network, to perform operations comprising;

test the first neural network using a testing data set comprising sets of input values and corresponding sets of output values;

analyze results of the testing of the first neural network to determine whether a degree of accuracy of the first neural network in generating the first portion of time series data meets a first threshold degree of accuracy; and

in response to determination that the degree of accuracy of the first neural network does not meet the first threshold degree of accuracy, perform operations comprising;

derive a different division of the full range of time into the multiple subranges of time;

alter the hyperparameters to implement the multiple portions of time series data based on the different division of the full range of time; and

repeat the training and testing of the first neural network following the alteration of the hyperparameters.

16. The computer-program product of claim 15, wherein, in response to a determination that the degree of accuracy of the first neural network does meet the first threshold degree of accuracy, the processor is caused to perform operations comprising:

use the first neural network configuration data to train the next neural network;

test the next neural network using the test data set;

analyze the results of the testing of the next neural network to determine whether a degree of accuracy of the next neural network in generating the next portion of time series data meets a second threshold degree of accuracy; and

in response to a determination that the degree of accuracy of the next neural network does not meet the second threshold degree of accuracy, perform operations comprising;

derive the different division of the full range of time into the multiple subranges of time;

alter the hyperparameters to implement the multiple portions of time series data based on the different division of the full range of time; and

repeat the training and testing of at least the first neural network following the alteration of the hyperparameters.

17. The computer-program product of claim 11, wherein:

the hyperparameters specify at least a quantity of artificial neurons within each neural network of the chain and a quantity of layers of artificial neurons within each neural network of the chain;

the quantity of layers includes an input layer of artificial neurons connected to the external inputs and the additional inputs; and

the quantity of layers includes an output layer of artificial neurons connected to the outputs.

18. The computer-program product of claim 17, wherein the chain comprises a type of chain selected from a group consisting of:

a single-link chain, wherein;

each neural network in the chain, except the first neural network at the head of the chain, receives the portion of the multiple portions of time series data that is generated at the outputs of the immediately preceding neural network in the ordering of neural networks within the chain; and

the quantity of additional inputs enables each neural network, except the first neural network at the head of the chain, to receive all of the outputs of the immediately preceding neural network at its additional inputs; and

a multi-link chain, wherein;

each neural network in the chain, except the first neural network at the head of the chain, receives all of the multiple portions of time series data that are generated at the outputs of all of the preceding neural networks in the ordering of neural networks within the chain; and

the quantity of additional inputs enables the last neural network to receive all of the outputs of all of the other neural networks in the chain.

19. The computer-program product of claim 11, wherein:

the trained parameters of the first neural network configuration data comprise weights and biases that represent what was learned by the first neural network during training; and

the processor is caused to train the set of neural networks sequentially in an order that follows the ordering of neural networks in the chain from the first neural network at the head of the chain to the last neural network at the tail of the chain.

20. The computer-program product of claim 11, wherein the processor is caused to perform operations comprising:

derive the portion of the multiple portions of time series data that is output by each neural network from the input data values during operation of the chain to generate time series data based on at least one of;

a quantity of artificial neurons within each neuromorphic device of a plurality of neuromorphic devices communicatively coupled to the processor;

a maximum quantity of layers that each neuromorphic device of the plurality of neuromorphic devices is able to support;

a maximum quantity of inputs that each neuromorphic device of the plurality of neuromorphic devices is able to support;

or

a maximum quantity of outputs that each neuromorphic device of the plurality of neuromorphic devices is able to support;

provide the first neural network configuration data to at least a first neuromorphic device of the plurality of neuromorphic devices to instantiate the first neural network;

provide the next neural network configuration data to at least a second neuromorphic device of the plurality of neuromorphic devices to instantiate the second neural network; and

provide a last neural network configuration data to at least a third neuromorphic device of the plurality of neuromorphic devices to instantiate the last neural network.

21. A computer-implemented method comprising:

training, by a processor, a first neural network of a chain of neural networks to generate a first portion of multiple portions of time series data that corresponds to a temporally earliest subrange of time of multiple subranges of time within a full range of time that is covered by the time series data, wherein;

the chain comprises a set of neural networks ordered to start with the first neural network at a head of the chain and to end with a last neural network at a tail of the chain;

each neural network of the chain comprises external inputs, additional inputs and outputs;

each neural network of the chain generates a portion of the multiple portions of the time series data at the outputs of the neural network from input data values provided at the external inputs of the neural network;

each portion of the multiple portions of the time series data corresponds to a subrange of the multiple subranges; and

the set of neural networks is interconnected within the chain such that each neural network, except the first neural network at the head of the chain, receives, at the additional inputs of the neural network, a portion of the multiple portions of the time series data that is generated at the outputs of a preceding neural network in the ordering of neural networks within the chain;

retrieving, from the first neural network, a first neural network configuration data comprising hyperparameters and first trained parameters learned by the first neural network from the training of the first neural network;

training, by the processor and using at least the first neural network configuration data, a next neural network in the ordering of neural networks within the chain to generate a next portion of the multiple portions that corresponds to a next subrange of time of the multiple subranges of time that temporally follows the earliest subrange;

retrieving, from the next neural network, a next neural network configuration data comprising the hyperparameters and next trained parameters learned by the next neural network from the training of the next neural network; and

using, by the processor, at least the first neural network configuration data and the next neural network configuration data to instantiate the chain.

22. The computer-implemented method of claim 21, wherein:

each neural network of the chain has the same quantities of external inputs, additional inputs and outputs;

during the training of each neural network of the chain, each additional input of the neural network that is not used to receive a portion of time series data from an output of a preceding neural network in the ordering of neural networks within the chain is provided with a null input.

the method comprises, following instantiation of the chain, operating the chain to generate the time series data from the set of input data values; and

during operation of the chain, each additional input of each neural network of the chain that is not used to receive a portion of a time series data from an output of a preceding neural network in the ordering of neural networks within the chain is provided with a null input.

23. The computer-implemented method of claim 21, comprising sequentially training, by the processor, each neural network in the chain via backpropagation following the ordering of the chain from the first neural network to the last neural network, wherein:

each neural network is trained using a training data set comprising sets of input values and corresponding sets of output values;

each set of output values is generated as time series data from the corresponding set of input values through use of non-neuromorphic processing;

the sets of input values of the training data are provided to the external inputs of each neural network, and separate portions of the corresponding sets of output values are provided to the outputs of each neural network;

each of the separate portions of the corresponding sets of output values that are provided to the outputs of each neural network corresponds to the portion of the time series data that is to be generated by the neural network; and

during the training of each neural network in the chain other than the first neural network at the head of the chain, the preceding neural network is operated to generate, from each set of input values of the training data, a corresponding portion of time series data that is provided at the additional inputs of the neural network.

24. The computer-implemented method of claim 23, comprising:

analyzing, by the processor, the neural network training data to identify a portion of the output data values across the sets of output values of the neural network training data that shows a relatively high degree of correlation; and

deriving, by the processor, a manner of dividing the time series data into the multiple portions of the time series data that are each output by one of the of the neural networks in the chain based, at least in part, on the identified portion of the output data values that shows the relatively high degree of correlation.

25. The computer-implemented method of claim 21, wherein:

the multiple portions of time series data are implemented using the hyperparameters, and are based on a division of the full range of time into the multiple subranges of time that was derived prior to the training of the first neural network; and

the method comprises, following the training of the first neural network and before the use of the first neural network configuration data to train the next neural network, performing operations comprising;

testing, by the processor, the first neural network using a testing data set comprising sets of input values and corresponding sets of output values;

analyzing, by the processor, results of the testing of the first neural network to determine whether a degree of accuracy of the first neural network in generating the first portion of time series data meets a first threshold degree of accuracy; and

in response to determination that the degree of accuracy of the first neural network does not meet the first threshold degree of accuracy, performing operations comprising;

deriving, by the processor, a different division of the full range of time into the multiple subranges of time;

altering, by the processor, the hyperparameters to implement the multiple portions of time series data based on the different division of the full range of time; and

repeating, by the processor, the training and testing of the first neural network following the alteration of the hyperparameters.

26. The computer-implemented method of claim 25, comprising, in response to a determination that the degree of accuracy of the first neural network does meet the first threshold degree of accuracy, performing operations comprising:

using, by the processor, the first neural network configuration data to train the next neural network;

testing, by the processor, the next neural network using the test data set;

analyzing, by the processor, the results of the testing of the next neural network to determine whether a degree of accuracy of the next neural network in generating the next portion of time series data meets a second threshold degree of accuracy; and

in response to a determination that the degree of accuracy of the next neural network does not meet the second threshold degree of accuracy, performing operations comprising;

deriving, by the processor, the different division of the full range of time into the multiple subranges of time;

altering, by the processor, the hyperparameters to implement the multiple portions of time series data based on the different division of the full range of time; and

repeating, by the processor, the training and testing of at least the first neural network following the alteration of the hyperparameters.

27. The computer-implemented method of claim 21, wherein:

the hyperparameters specify at least a quantity of artificial neurons within each neural network of the chain and a quantity of layers of artificial neurons within each neural network of the chain;

the quantity of layers includes an input layer of artificial neurons connected to the external inputs and the additional inputs; and

the quantity of layers includes an output layer of artificial neurons connected to the outputs.

28. The computer-implemented method of claim 27, wherein the chain comprises a type of chain selected from a group consisting of:

a single-link chain, wherein;

each neural network in the chain, except the first neural network at the head of the chain, receives the portion of the multiple portions of time series data that is generated at the outputs of the immediately preceding neural network in the ordering of neural networks within the chain; and

the quantity of additional inputs enables each neural network, except the first neural network at the head of the chain, to receive all of the outputs of the immediately preceding neural network at its additional inputs; and

a multi-link chain, wherein;

each neural network in the chain, except the first neural network at the head of the chain, receives all of the multiple portions of time series data that are generated at the outputs of all of the preceding neural networks in the ordering of neural networks within the chain; and

the quantity of additional inputs enables the last neural network to receive all of the outputs of all of the other neural networks in the chain.

29. The computer-implemented method of claim 21, wherein:

the trained parameters of the first neural network configuration data comprise weights and biases that represent what was learned by the first neural network during training; and

the method comprises training, by the processor, the set of neural networks sequentially in an order that follows the ordering of neural networks in the chain from the first neural network at the head of the chain to the last neural network at the tail of the chain.

30. The computer-implemented method of claim 21, comprising:

deriving, by the processor, the portion of the multiple portions of time series data that is output by each neural network from the input data values during operation of the chain to generate time series data based on at least one of;

a quantity of artificial neurons within each neuromorphic device of a plurality of neuromorphic devices communicatively coupled to the processor;

a maximum quantity of layers that each neuromorphic device of the plurality of neuromorphic devices is able to support;

a maximum quantity of inputs that each neuromorphic device of the plurality of neuromorphic devices is able to support;

or

a maximum quantity of outputs that each neuromorphic device of the plurality of neuromorphic devices is able to support;

providing the first neural network configuration data to at least a first neuromorphic device of the plurality of neuromorphic devices to instantiate the first neural network;

providing the next neural network configuration data to at least a second neuromorphic device of the plurality of neuromorphic devices to instantiate the second neural network; and

providing a last neural network configuration data to at least a third neuromorphic device of the plurality of neuromorphic devices to instantiate the last neural network.