System and method for testing prediction model

• US 5,893,069 A
• Filed: 01/31/1997
• Issued: 04/06/1999
• Est. Priority Date: 01/31/1997
• Status: Expired due to Term

• Current Assignee: Quantmetrics R&D Associates LLC
• Original Assignee: Quantmetrics R&D Associates LLC
• Inventors: White, Halbert L. Jr.
• Primary Examiner(s): MacDonald, Allen R.
• Assistant Examiner(s): Irshadullah, M.
• Application Number:
  US08/790,716
• Time in Patent Office: 795 Days
• Field of Search: 705/7, 705/11, 705/1, 395/184.01, 702/179, 702/182
• US Class Current: 705/348
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06Q 10/06393 Score-carding, benchmarking or key performance indicator [KPI] analysis
• CPC Class Codes: G06Q 10/067 Enterprise or organisation modelling
• CPC Class Codes: G06T 9/004 Predictors, e.g. intraframe, interframe coding

Abstract

A computer-implemented prediction model evaluation method includes specifying many prediction models and a benchmark model against which the prediction models will be evaluated. A primary data matrix is arranged by data indices, and the primary matrix is sampled with replacement N times to bootstrap N observation matrices. Then, all the matrices are filled with measurement criteria, with each criteria being representative of a respective data index and a respective model. A p-value estimate is returned that measures the statistical significance of the best prediction model relative to the benchmark, where the p-value represents the probability of wrongly rejecting the null hypothesis that a best prediction model has expected performance no better than that of a benchmark. The p-value accounts for the examination of all of the prediction models, i.e., the p-value depends on the examination of all of the models as a group, and not simply on a single model.

Claims

1. A computer including a data storage device including a computer usable medium having computer usable code means for evaluating the effectiveness of a best of a plurality of prediction models vis-a-vis a benchmark model, the computer usable code means having:

computer readable code means for receiving, from a computer input device, past market data from a database;

computer readable code means for generating the prediction models to be evaluated, at least one prediction model outputting at least one indicator of predicted performance;

computer readable code means for generating an effectiveness measurement of the benchmark model using predetermined measurement criteria, the predetermined measurement criteria being based on the past market data;

computer readable code means for generating an effectiveness measurement of each prediction model using the measurement criteria;

computer readable code means for determining the best one of a plurality of prediction models;

computer readable code means for generating a statistic representative of the statistical significance of the effectiveness of a best one of the prediction models vis-a-vis the benchmark model using the effectiveness measurements, wherein the statistic is determined based on the evaluation of all the prediction models; and

based on the statistic, using the best one of the prediction models to predict future performance.

2. The computer of claim 1, further comprising a primary data matrix including data grouped by data indices, and the computer further comprises computer readable code means for defining a predetermined number of observation data matrices by sampling, with replacement, the primary data matrix, the effectiveness measurements of the models being based on the data.

3. The computer of claim 2, wherein for each data matrix computer readable code means generate effectiveness measurements of the benchmark model and the prediction models, and the computer further comprises:

computer readable code means for generating, for each data matrix, a difference value representative of the difference between the effectiveness measurements of a model and a benchmark for the respective data matrix.

4. The computer of claim 3, wherein for each matrix the computer readable code means generate an average difference value representing the average difference between the effectiveness measurements of a model and a benchmark for the respective data matrix.

5. The computer of claim 4, further comprising:

computer readable code means for determining a maximum primary average difference value among the plurality of models; and

computer readable code means for determining an observation maximum average difference value among the plurality of models as a maximum among the plurality of models of the difference between the observation average difference value and the primary difference value.

6. The computer of claim 5, further comprising:

computer readable code means for sorting the observation maximum average difference values to generate a sorted list; and

computer readable code means for determining a location in the sorted list for the maximum primary average difference value.

7. The computer of claim 6, wherein the location in the sorted list of the maximum primary average difference value is at the nth location in the list, and wherein the statistic representative of the statistical significance of the effectiveness of the best among the prediction models is the difference between unity and the ratio of n to the total number of observation matrices.

8. A computer-implemented method for evaluating the effectiveness of the best among plural prediction models against a benchmark model, comprising the steps of:

collecting past performance data in a database;

specifying the prediction models;

defining a primary matrix arranged using data indices, the primary data matrix including the past performance data;

sampling the primary matrix with replacement N times to define N observation matrices;

filling the matrices with effectiveness measurement criteria, each criterion being representative of a respective data index and a respective model;

returning a statistic representative of the statistical significance of a most effective prediction model vis-a-vis a benchmark, based on the matrices;

determining the best prediction model;

using the statistic to assess the significance of a best prediction model vis-a-vis the benchmark, and

using the best prediction model to predict future performance.

9. The computer-implemented method of claim 8, further comprising the steps of:

generating an effectiveness measurement of the benchmark model using predetermined measurement criteria; and

generating an effectiveness measurement of each prediction model using the measurement criteria, wherein the statistic is based on the effectiveness measurements.

10. The computer-implemented method of claim 9, further comprising the steps of:

generating, for each data matrix, a difference value, the difference value being an average difference value representing the average difference between effectiveness measurements of a model and a benchmark for the respective data matrix.

11. The computer-implemented method of claim 10, further comprising the steps of:

determining a maximum primary average difference value among the plurality of models; and

determining an observation maximum average difference value among the plurality of models as a maximum among the plurality of models of the difference between the observation average difference value and the primary difference value.

12. The computer-implemented method of claim 11, further comprising the steps of:

sorting the observation maximum average difference values to generate a sorted list; and

determining a location in the sorted list for the maximum primary average difference value, wherein the location in the sorted list of the maximum primary average difference value is at the n^th location in the list, and wherein the statistic representative of the statistical significance of the effectiveness of the best among the prediction models is the difference between unity and the ratio of n to the total number of observation matrices.

13. A computer program product comprising:

a computer program storage device readable by a digital processing apparatus; and

a program mean on the program storage device and including instructions executable by the digital processing apparatus for performing method steps for evaluating plural prediction models, the method steps comprising;

receiving past performance data from a database, the past performance data being input by means of a computer input device;

generating the prediction models to be evaluated, the prediction models outputting one or more indicators of predicted future performance based on the past performance data;

generating an effectiveness measurement of a benchmark model using predetermined measurement criteria;

generating an effectiveness measurement of each prediction model using the measurement criteria;

generating a statistic representative of the statistical significance of the effectiveness of a best one of the prediction models vis-a-vis the benchmark model using the effectiveness measurements, wherein the statistic is determined based on the evaluation of all the prediction models;

based on the statistic, determining the best one of a plurality of prediction models; and

using the best one of the prediction models to predict future performance.

14. The computer program product of claim 13, wherein the method steps further comprise:

grouping data in a primary data matrix by appropriate data indices; and

generating a predetermined number of observation matrices by sampling, with replacement from the primary data matrix, the effectiveness measurements of the models being based on the data.

15. The computer program product of claim 14, wherein the method steps further comprise:

for each data matrix, generating effectiveness measurements of the benchmark model and the prediction models; and

generating, for each data matrix, an average difference value representative of the difference between the effectiveness measurements of a model and a benchmark for the respective data matrix.

16. The computer program product of claim 15, wherein the method steps further comprise:

determining a maximum primary average difference value among the plurality of models; and

determining an observation maximum average difference value among the plurality of models as a maximum among the plurality of models of the difference between the observation average difference value and the primary difference value.

17. The computer program product of claim 16, wherein the method steps further comprise:

sorting the observation maximum average difference values to generate a sorted list; and

determining a location in the sorted list for the maximum primary average difference value.

18. The computer program product of claim 17, wherein the location in the sorted list of the maximum primary average difference value is at the n^th location in the list, and wherein the statistic is the difference between unity and the ratio of n to the total number of observation matrices.