System and method to enable training a machine learning network in the presence of weak or absent training exemplars

• US 8,095,480 B2
• Filed: 07/31/2007
• Issued: 01/10/2012
• Est. Priority Date: 07/31/2007
• Status: Active Grant

• Current Assignee: Cornell Research Foundation Incorporated (Cornell University)
• Original Assignee: Cornell Research Foundation Incorporated (Cornell University)
• Inventors: Kristal, Bruce S., Martin, Rolf J.
• Primary Examiner(s): Rivas, Omar Fernandez
• Assistant Examiner(s): COUGHLAN, PETER D
• Application Number:
  US11/831,416
• Publication Number:
  US 20090037351A1
• Time in Patent Office: 1,624 Days
• Field of Search: 706/11, 381/314
• US Class Current: 706/12
• CPC Class Codes: G06Q 10/04 Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" market predictions or forecasting for commercial activities G06Q30/0202
• CPC Class Codes: G09B 7/06 of the multiple-choice answer-type, i.e. where a given question is provided with a series of answers and a choice has to be made from the answers

Abstract

Described is a system and method for training a machine learning network. The method comprises initializing at least one of nodes in a machine learning network and connections between the nodes to a predetermined strength value, wherein the nodes represent factors determining an output of the network, providing a first set of questions to a plurality of users, the first set of questions relating to at least one of the factors, receiving at least one of choices and guesstimates from the users in response to the first set of questions and adjusting the predetermined strength value as a function of the choices/guesstimates. The real and simulated examples presented demonstrate that synthetic training sets derived from expert or non-expert human guesstimates can replace or augment training data sets comprised of actual training exemplars that are too limited in size, scope, or quality to otherwise generate accurate predictions.

Claims

1. A method, comprising:

initializing at least one of i)nodes in a first machine learning network and ii) connections between the nodes, to a predetermined strength value, wherein the nodes represent factors determining an output of the network;

providing a first set of questions to a plurality of users, the first set of questions relating to at least one of the factors;

receiving guesstimates from the users in response to the first set of questions;

adjusting the predetermined strength value as a function of the guesstimates; and

combining guesstimates received from the users with those of other users to develop and evaluate the network, which is a consensus network.

2. The method according to claim 1, wherein the guesstimates comprise a synthetic data set capable of being applied at any time to the network.

3. The method according to claim 2, further comprising combining the synthetic data set with additional training data derived from a non-human source.

4. The method according to claim 1, further comprising:

after the step of adjusting, providing the network output to a second machine learning network;

providing a second question to a plurality of users of the second network; and

receiving at least one of choices and guesstimates from the users of the second network in response to the second question.

5. The method according to claim 4, wherein the second network is the same as the first network.

6. The method according to claim 4, wherein at least one of the users of the first network is the same as a user of the second network.

7. The method according to claim 1, further comprising:

receiving user information from each of the users; and

applying a weight to the guesstimates from a particular user as a function of the user information.

8. The method according to claim 7, wherein the user information is one of a self-reported expertise level and a user role indicative of expertise.

9. The method according to claim 1, wherein the at least one question is directed to one of stock analysis, drug development, terrorist activity, event wagering, medical diagnosis, detection of credit card fraud, classification of DNA sequences, speech and handwriting recognition, object recognition in computer vision, game playing and robot locomotion.

10. The method according to claim 1, further comprising:

providing a reward to a selected user based on at least one of a frequency with which the selected user inputs the guesstimates, a number of questions to which the selected user responds and whether the selected user includes explanatory information in support of a particular choice.

11. The method according to claim 10, wherein the explanatory information includes at least one of a reference and an identification of supporting literature.

12. The method according to claim 10, wherein the reward has at least one of monetary and perceived value.

13. The method according to claim 1, further comprising:

displaying at least one of a valid choice and a valid guesstimate option to a user who has yet to respond to the question.

14. The method according to claim 1, further comprising:

receiving explanations from the users regarding the guesstimates.

15. The method according to claim 14, wherein the explanations include at least one of supporting literature and an opinion of a source.

16. The method according to claim 1, wherein the predetermined strength value is one of a constant and a series of related constants.

17. The method according to claim 1, wherein a question provided to the plurality of users includes choices that are one of true/false, yes/no, multiple choice, a ranking inquiry and a rating inquiry.

18. The method according to claim 17, wherein the users are provided an opportunity to add to a list of possible choices.

19. The method according to claim 1, wherein the guesstimates include at least one of true/false, yes/no, a quality level, a probability, a percentage, a rank, a qualitative modifier, a quantitative modifier, an item and a series of items.

20. The method according to claim 1, further comprising generating a training set as a function of the user responses so that the adjusting of the predetermined strength values is performed automatically in accordance with a learning algorithm.

21. The method according to claim 1, further comprising:

receiving a query;

identifying connections between nodes in the network related to the query;

generating a response to the query as a function of strength values of the identified connections; and

outputting the response.

22. The method according to claim 21, wherein the connections include connections within and between input and output node layers.

23. The method according to claim 22, wherein the connections further include connections within and between a hidden node layer and each of the input and output node layers.

24. The method according to claim 21, wherein the users comprise a first user panel and the query is specified by a second user panel.

25. The method according to claim 21, wherein the outputted response is fed as input into a further machine learning network.

26. The method according to claim 21, further comprising:

feeding the response back into the network;

providing at least one of the first set of questions to, and receiving at least one of choices and guesstimates from, a third user panel; and

adjusting the strength value of the nodes as a function of the third user panel'"'"'s choices/guesstimates and the actual input/output.

27. The method according to claim 21, further comprising:

selecting at least one of the users based on a degree to which the selected user'"'"'s choice matches the response; and

providing a reward to the selected at least one user.

28. A device, comprising:

a communications arrangement receiving guesstimates from a plurality of users in response to questions related to a predetermined topic; and

a processor initializing at least one of i) nodes in a machine learning network and ii) connections between the nodes, to a predetermined strength value, the processor adjusting the predetermined strength value as a function of the choices, wherein the initialization is performed prior to any input of actual input and actual output into the network;

wherein the network is a consensus network and the users are able to combine their own guesstimates with those of other users to develop and evaluate the network.

29. The device according to claim 28, wherein the guesstimates comprise a synthetic data set that is capable of being applied to the network at any time.

30. The device according to claim 28, wherein the communications arrangement receives user information from each of the users, and the processor applies a weight to the guesstimates from a particular user as a function of the user information provided by the particular user.

31. The device according to claim 30, wherein the user information is one of a self-reported expertise level and a user role indicative of expertise.

32. The device according to claim 30, further comprising:

a memory storing the questions, wherein the questions are directed to a topic that is capable of being analyzed by a machine learning algorithm.

33. The device according to claim 32, wherein the topic is one of stock analysis, drug development, terrorist activity, event wagering, medical diagnosis, detection of credit card fraud, classification of DNA sequences, speech and handwriting recognition, object recognition in computer vision, game playing and robot locomotion.

34. The device according to claim 28, wherein the questions include choices that are at least one of true/false, yes/no, multiple choice, a ranking inquiry and a rating inquiry.

35. The device according to claim 28, wherein the guesstimates are at least one of true/false, yes/no, a quality level, a probability, a percentage, a rank, a qualitative modifier and a quantitative modifier.

36. The device according to claim 28, wherein the processor receives and inputs a query into the network to identify connections between nodes related to the query and generates a response to the query as a function of respective strength values of the identified connections.

37. A system, comprising:

a host computing device including a machine learning network, the host computing device initializing at least one of i) nodes in the network and ii) connections between the nodes, to a respective predetermined strength value, the host computing device outputting questions, each question corresponding to at least one of the connections; and

a plurality of client computing devices receiving the questions, the client computing devices transmitting guesstimates from users thereof in response to the questions,

wherein the host computing device adjusts the predetermined strength value as a function of the guesstimates, and

wherein the network is a consensus network and the users are able to combine their own guesstimates with those of other users to develop and evaluate the network.

38. The system according to claim 37, wherein the guesstimates comprise a synthetic data set that is capable of being applied at any time to the network.

39. The system according to claim 37, wherein the host computing device receives user information from at least one of the users and applies a weight to the guesstimates from a particular user as a function of the user information corresponding to the particular user.

40. The system according to claim 39, wherein the user information includes one of a self-reported expertise level and a user role indicative of expertise.

41. The system according to claim 37, wherein, when the host computing device receives a query from one of the client computing devices, the host computing device identifies connections between nodes in the network related to the query, generates a response to the query as a function of respective strength values of the identified connections and transmits the response to the one of the client computing devices.

42. The system according to claim 37, wherein the machine learning network executes at least one of the following types of machine learning algorithms, rules, trees and decision strategies:

Bayesnet, ComplementNaiveBayes, NaiveBayes, NaiveBayesMultinomial, NaiveBayesSimple, NaiveBayesUpdateable, HillClimber, RepeatedHillClimber, SimulatedAnnealing, TabuSearch, Logistic, SimpleLogistic, MultilayerPerceptron, VotedPerceptron, RBFNetwork, SMO, lazy.IB1, lazy.KStar, AdaBoostM1, ConjunctiveRule, DecisionTable, JRip, NNge, OneR, PART, ZeroR, J48, LMT, NBTree, RandomForest, RandomTree, REPTree, meta.AttributeSelectedClassifier, CfsSubsetEval, BestFirst, meta.Bagging, ClassificationViaRegression, M5P, CVParameterSelection, meta.Decorate, meta.FilteredClassifier, ClassOrder, supervised. attribute.AttributeSelection, Discretize, meta. Grading, meta.MultiScheme, MultiBoostAB, DecisionStump, NominalToBinary, OrdinalClassClassifier, meta.LogitBoost, meta.MultiClassClassifier, meta.RacedIncrementalLogitBoost, meta.RandomCommittee, meta.StackingC, meta.Stacking, meta.Vote, HyperPipes, VFI, Ridor and Hebbian learning.

43. The system according to claim 37, wherein additional nodes and connections are formed in the network in response to the guesstimates.

44. The system according to claim 37, wherein the guesstimates include estimates of plausible ranges for key values, and wherein upper and lower boundaries of the ranges define part of a training set.

45. The system according to claim 37, wherein the guesstimates are combined with output from a further machine learning network to generate a hybrid training set comprising both human and artificial machine learning choices.

46. The system according to claim 45, wherein a plurality of training sets are generated based on values close to those of the hybrid training set.

47. The system according to claim 37, wherein additional user guesstimates are made after examining output from the machine learning network.

48. The system according to claim 37, wherein the users are able to adjust a weight of their own guesstimates before the combining.

49. The system according to claim 37, wherein the guesstimates are made based on a perceived relevancy of specific input, and wherein the choices generate corresponding connections between nodes in a hidden layer of the network.

50. A non-transitory computer-readable medium storing a set of instructions for execution by a processor to perform a method comprising:

initializing at least one of i) nodes in a machine learning network and ii) connections between the nodes, to a predetermined strength value, wherein the nodes represent factors determinative of an output of the network;

providing at least one question to a plurality of users, the at least one question relating to at least one of the factors;

receiving guesstimates from the users in response to the at least one question;

adjusting the predetermined strength value as a function of the choices; and

combining guesstimates received from the users with those of other users to develop and evaluate the network, which is a consensus network.

51. The computer-readable medium according to claim 50, wherein the method executed by the processor further comprises:

receiving user information from each of the users; and

applying a weight to the guesstimates from a particular user as a function of the user information.

52. The computer-readable medium according to claim 50, wherein the method executed by the processor further comprises:

providing a reward to at least one selected user based on at least one of a frequency with which the selected user inputs the guesstimates, a number of questions to which the selected user responds and whether the selected user includes explanatory information in support of a particular choice.

53. The computer-readable medium according to claim 50, wherein the method executed by the processor further comprises:

displaying valid choices to a user who has yet to respond to the question.

54. The computer-readable medium according to claim 50, wherein the method executed by the processor further comprises:

receiving explanations from the users regarding the guesstimates.

55. The computer-readable medium according to claim 50, wherein the method executed by the processor further comprises:

receiving a query;

identifying connections between nodes in the network related to the query;

generating a response to the query as a function of strength values of the identified connections; and

outputting the response.

56. The computer-readable medium according to claim 55, wherein the method executed by the processor further comprises:

feeding the outputted response back into the network;

providing the at least one question to, and receiving guesstimates from, a third user panel; and

adjusting the strength value of the nodes as a function of the third user panel'"'"'s guesstimates and the actual input/output.

57. The computer-readable medium according to claim 55, wherein the method executed by the processor further comprises:

selecting at least one of the users based on a degree to which the selected user'"'"'s guesstimates match at least one of real world events and true training exemplars; and

providing a reward to the selected user.