ASSESSING ACCURACY OF TRAINED PREDICTIVE MODELS
2 Assignments
0 Petitions
Accused Products
Abstract
A system includes a computer(s) coupled to a data storage device(s) that stores a training data repository and a predictive model repository. The training data repository includes retained data samples from initial training data and from previously received data sets. The predictive model repository includes at least one updateable trained predictive model that was trained with the initial training data and retrained with the previously received data sets. A new data set is received. A richness score is assigned to each of the data samples in the set and to the retained data samples that indicates how information rich a data sample is for determining accuracy of the trained predictive model. A set of test data is selected based on ranking by richness score the retained data samples and the new data set. The trained predictive model is accuracy tested using the test data and an accuracy score determined.
-
Citations
42 Claims
-
1-21. -21. (canceled)
-
22. A computer-implemented method comprising:
-
receiving a first data set of data samples, each data sample comprising input data and corresponding output data, wherein the first data set is new relative to a retained data set of data samples, where each data sample in the retained data set comprising input data and corresponding output data, and where the retained data set was used in training predictive models in a repository of predictive models; determining a richness score for each of the data samples included in the first data set and to each of the data samples in the retained data set, wherein the richness score for a particular data sample indicates how information rich the particular data sample is, relative to other data samples in the set of retained data samples and the first data set, for determining an accuracy of a trained predictive model, and where the richness score for a particular data sample is based, at least in part, on how redundant the particular data sample is of other, different data samples; ranking the data samples included in the first data set and the set of retained data samples based on the assigned richness scores; and selecting a first set of test data from the data samples included in the first data set and the set of retained data samples based on the ranking. - View Dependent Claims (23, 24, 25, 26, 27, 28, 29)
-
-
30. A computer-implemented system comprising:
-
one or more computers; and one or more data storage devices coupled to the one or more computers, storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising; receiving a first data set of data samples, each data sample comprising input data and corresponding output data, wherein the first data set is new relative to a retained data set of data samples, where each data sample in the retained data set comprising input data and corresponding output data, and where the retained data set was used in training predictive models in a repository of predictive models; determining a richness score for each of the data samples included in the first data set and to each of the data samples in the retained data set, wherein the richness score for a particular data sample indicates how information rich the particular data sample is, relative to other data samples in the set of retained data samples and the first data set, for determining an accuracy of a trained predictive model, and where the richness score for a particular data sample is based, at least in part, on how redundant the particular data sample is of other, different data samples; ranking the data samples included in the first data set and the set of retained data samples based on the assigned richness scores; and selecting a first set of test data from the data samples included in the first data set and the set of retained data samples based on the ranking. - View Dependent Claims (31, 32, 33, 34, 35, 36, 37)
-
-
38. A computer-readable storage device encoded with a computer program product, the computer program product comprising instructions that when executed on one or more computers cause the one or more computers to perform operations comprising:
-
receiving a first data set of data samples, each data sample comprising input data and corresponding output data, wherein the first data set is new relative to a retained data set of data samples, where each data sample in the retained data set comprising input data and corresponding output data, and where the retained data set was used in training predictive models in a repository of predictive models; determining a richness score for each of the data samples included in the first data set and to each of the data samples in the retained data set, wherein the richness score for a particular data sample indicates how information rich the particular data sample is, relative to other data samples in the set of retained data samples and the first data set, for determining an accuracy of a trained predictive model, and where the richness score for a particular data sample is based, at least in part, on how redundant the particular data sample is of other, different data samples; ranking the data samples included in the first data set and the set of retained data samples based on the assigned richness scores; and selecting a first set of test data from the data samples included in the first data set and the set of retained data samples based on the ranking. - View Dependent Claims (39, 40, 41, 42)
-
Specification