System and method for combining valuations of multiple evaluators

• US 7,240,039 B2
• Filed: 10/29/2003
• Issued: 07/03/2007
• Est. Priority Date: 10/29/2003
• Status: Active Grant

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Kirshenbaum, Evan R.
• Primary Examiner(s): Knight; Anthony
• Assistant Examiner(s): Holmes; Michael B.
• Application Number:
  US10/695,707
• Publication Number:
  US 20050097067A1
• Time in Patent Office: 1,343 Days
• Field of Search: 706 45- 47, 705 2- 10
• US Class Current: 706/46
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data

Abstract

Disclosed herein are systems and methods for combining valuations of multiple evaluators. One disclosed method embodiment comprises (for each evaluator): sorting the choices into an order indicative of preference; determining interval sizes between adjacent choices; obtaining a rank for each interval size; and using interval size ranks to determine an evaluator score for each choice. The evaluator scores can be combined to determine an overall score for each choice.

Claims

1. A processor-based method of combining valuations of multiple choices from each of multiple evaluators, the method comprising:

for each evaluator,

sorting the choices into an order that indicates preference;

determining interval sizes between choice valuations;

obtaining a rank for each interval size; and

using interval size ranks, at least in part, to determine an evaluator score for each choice;

combining evaluator scores for each choice to determine, at least in part, an overall score for each choice; and

presenting to a user a list of choices ordered in accordance with the overall scores.

2. The method of claim 1, wherein said obtaining a rank for each interval size comprises ranking interval sizes from smallest to largest.

3. The method of claim 1, wherein said using interval size ranks to determine an evaluator score for each choice comprises:

associating a predetermined score with a first choice; and

iteratively calculating a score for each subsequent choice by combining the score of a preceding choice with an increment based on the rank of the interval size between the preceding choice and said subsequent choice.

4. The method of claim 1, further comprising:

determining equivalence classes for choices based on choice valuations, wherein said determining equivalence classes precedes said obtaining a rank for each interval size, and wherein said obtaining a rank for each interval size involves obtaining a rank for each interval size between valuations associated with different equivalence classes;

calculating scores for equivalence classes based on said interval size ranks; and

associating the score for each equivalence class with each choice that is a member of that equivalence class.

5. The method of claim 4, wherein said determining equivalence classes is based on a statistical interval distribution.

6. The method of claim 4, wherein said determining equivalence classes is based on a predetermined minimum interval size.

7. The method of claim 4, further comprising:

generating a valuation to associate with each equivalence class by operating on the one or more valuations associated with the one or more choices in that equivalence class, said operating using an operator from a group consisting of;

an arithmetic mean of the one or more valuations;

a geometric mean of the one or more valuations;

a harmonic mean of the one or more valuations;

a root mean square of the one or more valuations;

a median of the one or more valuations;

a value corresponding to a predetermined percentile among the one or more valuations;

a most frequent valuation of the one or more valuations;

a most frequent valuation determined within a predetermined precision;

a maximum of the one or more valuations; and

a minimum of the one or more valuations.

8. The method of claim 1, further comprising:

assigning an adverse valuation to each choice not evaluated by a given evaluator, wherein the adverse valuation indicates that the choice is less preferred than the least preferred choice evaluated by the given evaluator, and wherein the adverse valuation separates the choice from the least preferred choice by an interval greater than each other interval between adjacent choice valuations.

9. The method of claim 1, wherein said ranking includes:

assigning equalized ranks to intervals that are substantially equal.

10. The method of claim 1, wherein the overall score for each choice is a weighted or unweighted sum of evaluator scores for that choice.

11. The method of claim 1, wherein said combining includes:

scaling evaluator scores to a uniform range.

12. The method of claim 1, wherein said combining includes:

scaling evaluator scores to provide each evaluator with an equal sum of evaluator scores.

13. The method of claim 1, further comprising:

selecting a predetermined number of choices having the most favorable overall scores.

14. The method of claim 1, further comprising:

selecting from a group of choices a predetermined number of choices having the most favorable overall scores.

15. The method of claim 1, further comprising:

sorting the choices according to the overall score determined for each choice.

16. The method of claim 1, wherein at least one of the evaluators is selected from a group consisting of:

a provider of values derived from sensor measurements,

a provider of values derived from performance measurements,

a provider of values derived from solicited personal opinions,

a provider of values derived by a Bayesian network,

a provider of values derived by a decision tree,

a provider of values derived by an evolved algorithm,

a provider of values derived by a neural network,

a provider of values derived by a fitness estimator, and

a provider of values derived from complexity measurements.

17. The method of claim 1, wherein each choice represents an alternative from a group consisting of:

classifications of an item;

candidate solutions to a problem;

contest competitors; and

features to be used for classification.

18. A system for combining valuations of multiple choices from each of multiple evaluators, wherein the choices are different classifications and the valuations represent probabilities that a web site visitor belongs that classification, the system comprising:

a processor; and

a memory coupled to the processor and configured to provide the processor with software that configures the processor to iterate through the multiple evaluators, and for each evaluator configures the processor to;

obtain ranks of intervals between a valuation-ordered group of choices;

determine a score for each choice from the interval ranks;

combine choice scores for each evaluator to obtain an overall score for each choice; and

select the choice having the highest overall score as the classification for the web site visitor.

19. The system of claim 18, wherein the software further configures the processor to retrieve choice valuations from at least one external evaluator.

20. The system of claim 18, wherein the software further configures the processor to operate as at least one evaluator.

21. The system of claim 18, wherein the software further configures the processor to employ one or more of the choices in a manner responsive to the overall scores.

22. The system of claim 18, wherein as part of configuring the processor to rank intervals, the software configures the processor to assign a rank of zero to intervals of insignificant size.

23. The system of claim 18, wherein as part of configuring the processor to determine a score for each choice, the software configures the processor to:

determine equivalence classes for choices based on choice valuations before obtaining ranks of intervals, wherein said obtaining comprises finding ranks of the intervals between valuations associated with the equivalence classes;

calculate scores for equivalence classes based on the interval ranks; and

associate the score for each equivalence class with each choice that is a member of that equivalence class.

24. The system of claim 18, wherein the software further configures the processor to assign an adverse valuation to each of an evaluator'"'"'s unevaluated choices, wherein the adverse valuation indicates that the choice is less preferred than the least preferred choice evaluated by that evaluator, and wherein the interval between the least preferred choice and the unevaluated choice no smaller than each interval between that evaluator'"'"'s evaluated choices.

25. The system of claim 18, wherein as part of configuring the processor to rank intervals, the software configures the processor to assign equalized ranks to intervals that are substantially equal.

26. The system of claim 18, wherein as part of configuring the processor to combine choice scores, the software configures the processor to scale choice scores to a uniform range.

27. The system of claim 18, wherein as part of configuring the processor to combine choice scores, the software configures the processor to scale choice scores to provide each evaluator with an equal sum of choice scores.

28. The system of claim 18, wherein the overall scores for each choice are weighted or unweighted sums of the scores for each choice.

29. An evaluation system for ranking a company'"'"'s products comprising:

a means for determining a valuation by each of multiple evaluators for each of multiple products, wherein at least one evaluator determines a valuation representing revenue, and another evaluator determines a valuation representing profit;

a means for ranking intervals between valuation-ordered products for each evaluator;

a means for determining product scores using interval ranks for each evaluator;

a means for combining product scores to determine an overall score for each product; and

a means for selecting a predetermined number of products in accordance with their overall scores.

30. The system of claim 29, wherein said means for determining product scores comprises:

a means for associating a predetermined score with a first product; and

a means for iteratively calculating a score for each subsequent product by combining the score of a preceding product with an increment based on the rank of the interval size between the preceding product and said subsequent product.