System and method for testing prediction models and/or entities

• US 6,088,676 A
• Filed: 03/26/1999
• Issued: 07/11/2000
• Est. Priority Date: 01/31/1997
• Status: Expired due to Term

• Current Assignee: Quantmetrics R&D Associates LLC
• Original Assignee: Quantmetrics R&D Associates LLC
• Inventors: White, Halbert L. Jr.
• Primary Examiner(s): Stamber, Eric W.
• Assistant Examiner(s): Irshadullah, M.
• Application Number:
  US09/277,515
• Time in Patent Office: 473 Days
• Field of Search: 705/1, 705/7, 705/10, 706/58, 707/6, 707/102
• US Class Current: 705/7.39
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06Q 10/06393 Score-carding, benchmarking or key performance indicator [KPI] analysis
• CPC Class Codes: G06Q 10/067 Enterprise or organisation modelling
• CPC Class Codes: G06T 9/004 Predictors, e.g. intraframe, interframe coding

Abstract

A computer-implemented performance evaluation method includes specifying a group of comparable entities and a benchmark against which the comparable entities are evaluated. The entities evaluated may be a process, technology, strategy, treatment, organization, individual, or other identifiable unit. A primary data matrix is arranged by data indices, and the primary matrix is sampled with replacement N times to bootstrap N observation matrices. Alternatively, a Monte Carlo approach can be used. Then, all the matrices are filled with measurement criteria, with each criterion being representative of a respective data index and a respective entity. A p-value estimate is returned that measures the statistical significance of the best of the comparable entities relative to the benchmark, where the p-value represents the probability of wrongly rejecting the null hypothesis that a best of the comparable entities has expected performance no better than that of a benchmark. The p-value accounts for the examination of all of the comparable entities, i.e., the p-value depends on the examination of all of the entities as a group, and not simply on a single entity.

Claims

1. A computer including a data storage device including a computer usable medium having computer usable code means for evaluating the effectiveness of a best of a plurality of comparable entities vis-a-vis a benchmark, the computer usable code means having:

computer readable code means for receiving, from a computer input device, past market data from a database;

computer readable code means for generating the entities to be evaluated, at least one entity outputting at least one indicator of predicted performance;

computer readable code means for generating an effectiveness measurement of the benchmark using at least one predetermined measurement criterion, the predetermined measurement criterion being based on the past market data;

computer readable code means for generating an effectiveness measurement of each entity evaluated using the at least one measurement criterion;

computer readable code means for determining the best one of a plurality of entities;

computer readable code means for generating a statistic representative of the statistical significance of the effectiveness of a best one of the comparable entities vis-a-vis the benchmark using the effectiveness measurements such that the statistic is determined based on the evaluation of all the entities; and

based on the statistic, using the best one of the entities to predict future performance.

2. The computer of claim 1, further comprising a primary data matrix including data grouped by data indices, and the computer further comprises computer readable code means for defining a predetermined number of observation data matrices by sampling, with replacement, the primary data matrix, the effectiveness measurements of the models being based on the data.

3. The computer of claim 2, further comprising a predetermined number of observation data matrices, wherein for each data matrix computer readable code means generate effectiveness measurements of the benchmark and the comparable entities, and the computer further comprises:

computer readable code means for generating, for each data matrix, a difference value representative of the difference between the effectiveness measurements of an entity and a benchmark for the respective data matrix.

4. The computer of claim 3, wherein for each matrix the computer readable code means generate an average difference value representing the average difference between the effectiveness measurements of an entity and a benchmark for the respective data matrix.

5. The computer of claim 4, further comprising:

computer readable code means for determining a maximum primary average difference value among the plurality of comparable entities; and

computer readable code means for determining an observation maximum average difference value among the plurality of comparable entities as a maximum among the plurality of comparable entities of the difference between the observation average difference value and the primary difference value.

6. The computer of claim 5, further comprising:

computer readable code means for sorting the observation maximum average difference values to generate a sorted list; and

computer readable code means for determining a location in the sorted list for the maximum primary average difference value.

7. The computer of claim 6, wherein the location in the sorted list of the maximum primary average difference value is at the nth location in the list, and wherein the statistic representative of the statistical significance of the effectiveness of the best among the comparable entities is the difference between unity and the ratio of n to the total number N of observation matrices.

8. A computer-implemented method for evaluating the effectiveness of the best among plural comparable entities against a benchmark, comprising the acts of:

collecting past performance data in a database;

specifying the comparable entities;

defining a primary matrix arranged using data indices, the primary data matrix including the past performance data;

sampling the primary matrix with replacement N times to define N observation matrices;

filling the matrices with effectiveness measurement criteria, each criterion being representative of a respective data index and a respective entity;

returning a statistic representative of the statistical significance of a most effective entity vis-a-vis a benchmark, based on the matrices;

using the most effective entity to predict future performance.

9. The computer-implemented method of claim 8, further comprising the acts of:

generating an effectiveness measurement of the benchmark using predetermined measurement criteria; and

generating an effectiveness measurement of each comparable entity using the measurement criteria, wherein the statistic is based on the effectiveness measurements.

10. The computer-implemented method of claim 9, further comprising the acts of:

generating, for each data matrix, a difference value, the difference value being an average difference value representing the average difference between effectiveness measurements of an entity and a benchmark for the respective data matrix.

11. The computer-implemented method of claim 10, further comprising the acts of:

determining a maximum primary average difference value among the plurality of comparable entities; and

determining an observation maximum average difference value among the plurality of comparable entities as a maximum among the plurality of comparable entities of the difference between the observation average difference value and the primary difference value.

12. The computer-implemented method of claim 11, further comprising the acts of:

sorting the observation maximum average difference values to generate a sorted list; and

determining a location in the sorted list for the maximum primary average difference value, wherein the location in the sorted list of the maximum primary average difference value is at the nth location in the list, and wherein the statistic representative of the statistical significance of the effectiveness of the best among the comparable entities is the difference between unity and the ratio of n to the total number N of observation matrices.

13. A computer program product comprising:

a computer program storage device readable by a digital processing apparatus; and

a program means on the program storage device and including instructions executable by the digital processing apparatus for performing method acts for evaluating plural comparable entities, the method acts comprising;

receiving past performance data from a database, the past performance data being input by means of a computer input device;

generating the entities to be evaluated, the entities outputting one or more indicators of predicted future performance based on the past performance data;

generating an effectiveness measurement of a benchmark using at least one predetermined measurement criterion;

generating an effectiveness measurement of each comparable entity using the measurement criterion;

generating a statistic representative of the statistical significance of the effectiveness of a best one of the comparable entities vis-a-vis the benchmark using the effectiveness measurements such that the statistic is determined based on the evaluation of all the comparable entities;

based on the statistic, determining the best one of a plurality of entities; and

using the best one of the entities to predict future performance.

14. The computer program product of claim 13, wherein the method acts further comprise:

grouping data in a primary data matrix by appropriate data indices; and

generating a predetermined number of observation matrices by sampling, with replacement from the primary data matrix, the effectiveness measurements of the models being based on the data.

15. The computer program product of claim 14, wherein the method acts further comprise:

generating a predetermined number of observation data matrices;

for each data matrix, generating effectiveness measurements of the benchmark and the comparable entities; and

generating, for each data matrix, an average difference value representative of the difference between the effectiveness measurements of an entity and a benchmark for the respective data matrix.

16. The computer program product of claim 15, wherein the method acts further comprise:

determining a maximum primary average difference value among the plurality of comparable entities; and

determining an observation maximum average difference value among the plurality of comparable entities as a maximum among the plurality of comparable entities of the difference between the observation average difference value and the primary difference value.

17. The computer program product of claim 16, wherein the method acts further comprise:

sorting the observation maximum average difference values to generate a sorted list; and

determining a location in the sorted list for the maximum primary average difference value.

18. The computer program product of claim 17, wherein the location in the sorted list of the maximum primary average difference value is at the n^th location in the list, and wherein the statistic is the difference between unity and the ratio of n to the total number N of observation matrices.