Cooperative execution of a genetic algorithm with an efficient training algorithm for data-driven model creation

• US 9,785,886 B1
• Filed: 04/17/2017
• Issued: 10/10/2017
• Est. Priority Date: 04/17/2017
• Status: Active Grant

• Current Assignee: SparkCognition Inc.
• Original Assignee: SparkCognition Inc.
• Inventors: Andoni, Sari, Moore, Keith D., Husain, Syed Mohammad Amir
• Primary Examiner(s): Holmes, Michael B
• Application Number:
  US15/489,564
• Time in Patent Office: 176 Days
• Field of Search: 706 13
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/086 using evolutionary algorithms, e.g. genetic algorithms or genetic programming

Abstract

A method includes, based on a fitness function, selecting a subset of models from a plurality of models. The plurality of models is generated based on a genetic algorithm and corresponds to a first epoch of the genetic algorithm. Each of the plurality of models includes data representative of a neural network. The method also includes performing at least one genetic operation of the genetic algorithm with respect to at least one model of the subset to generate a trainable model and sending the trainable model to an optimization trainer. The method includes adding a trained model received from the optimization trainer as input to a second epoch of the genetic algorithm that is subsequent to the first epoch.

Claims

1. A computer system comprising:

a memory configured to store an input data set and a plurality of data structures, each of the plurality of data structures including data representative of a neural network;

a processor configured to execute a recursive search, wherein executing the recursive search comprises, during a first iteration;

determining a fitness value for each of the plurality of data structures based on at least a subset of the input data set;

selecting a subset of data structures from the plurality of data structures based on the fitness values of the subset of data structures;

performing at least one of a crossover operation or a mutation operation with respect to at least one data structure of the subset to generate a trainable data structure; and

providing the trainable data structure to an optimization trainer, the optimization trainer configured to;

train the trainable data structure based on a portion of the input data set to generate a trained data structure; and

provide the trained data structure as input to a second iteration of the recursive search that is subsequent to the first iteration.

2. The computer system of claim 1, wherein the optimization trainer is executed on a different device, graphics processing unit (GPU), processor, core, thread, or any combination thereof, than the recursive search.

3. A method comprising:

based on a fitness function, selecting, by a processor of a computing device, a subset of models from a plurality of models, the plurality of models generated based on a genetic algorithm and corresponding to a first epoch of the genetic algorithm, wherein each of the plurality of models includes data representative of a neural network;

performing at least one genetic operation of the genetic algorithm with respect to at least one model of the subset to generate a trainable model;

sending the trainable model to an optimization trainer; and

adding a trained model received from the optimization trainer as input to a second epoch of the genetic algorithm that is subsequent to the first epoch.

4. The method of claim 3, wherein the fitness function is evaluated based on an input data set associated with the genetic algorithm.

5. The method of claim 3, wherein the optimization trainer is configured to use a portion of an input data set associated with the genetic algorithm to train the trainable model.

6. The method of claim 3, wherein the data representative of the neural network includes node data corresponding to a plurality of nodes of the neural network.

7. The method of claim 6, wherein the node data includes an activation function, an aggregation function, a bias, or any combination thereof.

8. The method of claim 3, wherein the data representative of the neural network includes connection data corresponding to one or more connections of the neural network.

9. The method of claim 8, wherein the connection data includes node pairs, connection weights, or any combination thereof.

10. The method of claim 3, wherein the optimization trainer is configured to update connection weights of the trainable model but not a topology or activation functions of the trainable model.

11. The method of claim 3, wherein the first epoch is an initial epoch of the genetic algorithm.

12. The method of claim 3, wherein the first epoch is a non-initial epoch of the genetic algorithm.

13. The method of claim 3, wherein the second epoch and the first epoch are separated by at least one epoch.

14. The method of claim 3, wherein the at least one genetic operation includes crossover, mutation, or a combination thereof.

15. The method of claim 3, wherein each of the plurality of models includes at least one output node configured to generate an output value corresponding to a field of an input data set associated with the genetic algorithm and wherein a fitness value of a particular model is based at least partially on a comparison of the output value and the field of the input data set.

16. The method of claim 3, wherein each of the plurality of models includes at least one output node configured to generate a classifier result.

17. The method of claim 3, wherein the plurality of models corresponds to an input population of the first epoch, and further comprising:

grouping the models of the plurality of models into species based on genetic distance between the models;

determining a species fitness value of each of the species;

selectively removing one or more species from the genetic algorithm responsive to determining that the one or more species satisfy a stagnation criterion;

determining one or more elite species based on their respective species fitness values;

identifying elite members of each elite species; and

generating an output population to be input into the second epoch, wherein the output population includes each of the elite members and at least one model generated based on intra-species reproduction.

18. The method of claim 17, wherein the output population further includes the trained model received from the optimization trainer.

19. A non-transitory computer-readable storage device storing instructions that, when executed, cause a computer to perform operations comprising:

based on a fitness function, selecting a subset of models from a plurality of models, the plurality of models generated based on a genetic algorithm and corresponding to a first epoch of the genetic algorithm, wherein each of the plurality of models includes data representative of a neural network;

performing at least one genetic operation of the genetic algorithm with respect to at least one model of the subset to generate a trainable model;

sending the trainable model to a trainer; and

adding a trained model received from the trainer as input to a second epoch of the genetic algorithm that is subsequent to the first epoch.

20. The non-transitory computer-readable storage device of claim 19, wherein the trainer comprises a backpropagation trainer and wherein the operations further comprise:

grouping the models of the plurality of models into species based on genetic distance between the models;

determining a species fitness value of each of the species;

selectively removing one or more species from the genetic algorithm responsive to determining that the one or more species satisfy a stagnation criterion;

determining one or more elite species based on their respective species fitness values;

identifying elite members of each elite species; and

generating an output population to be input into the second epoch, wherein the output population includes each of the elite members and at least one model generated based on intra-species reproduction.