Computerized system for enhancing expert-based processes and methods useful in conjunction therewith

• US 8,346,685 B1
• Filed: 04/22/2009
• Issued: 01/01/2013
• Est. Priority Date: 04/22/2009
• Status: Active Grant

• Current Assignee: Microsoft Israel Research And Development (2002) Ltd (Microsoft Corporation)
• Original Assignee: Equivio Ltd. (Microsoft Corporation)
• Inventors: Ravid, Yiftach
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Gonzales, Vincent
• Application Number:
  US12/428,100
• Time in Patent Office: 1,350 Days
• Field of Search: 706/12
• US Class Current: 706/12
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition

Abstract

A computerized system for enhancing expert-based processes, the system comprising a computerized expert based data analyzer receiving input from a plurality of experts by operating a corresponding plurality of expert-based processes on a body of data, the input including a discrepancy set including at least one point of discrepancy regarding which less than all of the plurality of experts agree and an agreement set including at least one point of agreement regarding which all of the plurality of experts agree; and an oracle from which oracle input is received resolving at least the point of discrepancy and not resolving any point of agreement in the agreement set; wherein the computerized analyzer is operative to select and to subsequently actuate for purposes of receiving input regarding the body of data, a subset of better experts from among the plurality of experts based on the oracle input.

Claims

1. A method for enhancing expert-based processes when receiving input from a plurality of experts operating a corresponding plurality of computerized expert-based processes on a body of data, said data including an agreement set including at least one point of agreement regarding which all of said plurality of experts agree, the method comprising:

determining a discrepancy set including at least one point of discrepancy regarding which less than all of the plurality of experts agree, said determining including performing a computerized comparison of input received from said plurality of experts thereby to identify points of discrepancy;

providing at least a portion of said discrepancy set to an oracle and receiving oracle input from the oracle resolving at least said point of discrepancy; and

selecting a subset of better computerized expert-based processes, from among said plurality of computerized expert-based processes, based on said oracle input,

wherein said oracle input resolves less than all of a total number of points of discrepancy in said discrepancy set by randomly sampling a subset of the total number of points of discrepancy,

and wherein each of the expert-based processes has a binary output defining first and second output values and wherein said discrepancy set includes a first set of points of discrepancy in which first and second experts from among said plurality of experts output the first and second output values respectively and a second set of points of discrepancy in which first and second experts from among said plurality of experts output the second and first output values respectively.

2. A method according to claim 1 wherein said input comprises a quality assurance indication regarding a multiplicity of manufactured items.

3. A method according to claim 1 wherein said input comprises a determination of relevancy of a multiplicity of disclosed documents to a legal proceeding.

4. A method according to claim 1 wherein said input comprises a determination of relevancy of a multiplicity of disclosed documents to a search term entered into a search engine.

5. A method according to claim 1 wherein only said discrepancy set, and not said agreement set, is provided to the oracle.

6. A method according to claim 1 wherein said oracle comprises a computerized process which is more costly than the plurality of experts.

7. A method according to claim 1 wherein said selecting comprises using said oracle input and said input from the plurality of experts to estimate recall of an individual one of said plurality of experts.

8. A method according to claim 1 wherein said selecting comprises using said oracle input and said input from the plurality of experts to estimate precision of an individual one of said plurality of experts.

9. A method according to claim 1 wherein each of the expert-based processes has a binary output defining a first desired output value and a second undesired output value and also comprising using said oracle input and said input from the plurality of experts to estimate richness of the body of data including the proportion of the body of data having said first output value.

10. A method according to claim 1 wherein said portion of said discrepancy set comprises a random sample of each of said first and second sets.

11. A method according to claim 1 and also comprising receiving input from a plurality of experts operating a corresponding plurality of computerized expert-based processes on a body of data.

12. A method according to claim 11 wherein said body of data comprises a population of items each associated with a unique computer-readable ID.

13. A method according to claim 12 wherein each of said plurality of computerized expert-based processes comprises:

analyzing an individual item from among said population of items including determining a binary output representing said analyzing; and

generating a computerized representation of said binary output and associating, said representation with said ID.

14. A method according to claim 1 and also comprising actuating said subset of better computerized expert-based processes for purposes of receiving input regarding a body of data.

15. A computer program product, comprising a non-transitory computer usable medium having a computer readable program code embodied therein, said computer readable program code adapted to be executed to implement a method for enhancing expert-based processes, the method comprising:

receiving input from a plurality of experts by operating a corresponding plurality of expert-based processes on a body of data, the input including a discrepancy set including at least one point of discrepancy regarding which less than all of said plurality of experts agree and an agreement set including at least one point of agreement regarding which all of said plurality of experts agree;

receiving oracle input from an oracle which receives at least a portion of said discrepancy set and resolving at least said point of discrepancy and not resolving any point of agreement in said agreement set; and

selecting, and subsequently actuating for purposes of receiving input regarding the body of data, a subset of better experts from among the plurality of experts based on said oracle input,

wherein said oracle input resolves less than all of a total number of points of discrepancy in said discrepancy set by randomly sampling a subset of the total number of points of discrepancy,

and wherein each of the expert-based processes has a binary output defining first and second output values and wherein said discrepancy set includes a first set of points of discrepancy in which first and second experts from among said plurality of experts output the first and second output values respectively and a second set of points of discrepancy in which first and second experts from among said plurality of experts output the second and first output values respectively.

16. A computerized system for enhancing expert-based processes, the system comprising:

a computerized expert based data analyzer residing on a tangible non-transitory computer readable storage medium and operative for receiving input from a plurality of experts by operating a corresponding plurality of expert-based processes on a body of data, the input including a discrepancy set including at least one point of discrepancy regarding which less than all of said plurality of experts agree and an agreement set including at least one point of agreement regarding which all of said plurality of experts agree;

an oracle, receiving at least a portion of said discrepancy set and from which oracle input is received resolving at least said point of discrepancy and not resolving any point of agreement in said agreement set; and

wherein said computerized analyzer is operative to select and to subsequently actuate for purposes of receiving input regarding the body of data, a subset of better experts from among the plurality of experts based on said oracle input,

wherein said oracle input resolves less than all of a total number of points of discrepancy in said discrepancy set by randomly sampling a subset of the total number of points of discrepancy,

and wherein each of the expert-based processes has a binary output defining first and second output values and wherein said discrepancy set includes a first set of points of discrepancy in which first and second experts from among said plurality of experts output the first and second output values respectively and a second set of points of discrepancy in which first and second experts from among said plurality of experts output the second and first output values respectively.

17. A method according to claim 16 wherein said input comprises a quality assurance indication regarding a multiplicity of manufactured items.

18. A method according to claim 16 wherein said input comprises a determination of relevancy of a multiplicity of disclosed documents to a legal proceeding.

19. A method according to claim 16 wherein said input comprises a determination of relevancy of a multiplicity of disclosed documents to a search term entered into a search engine.

20. A method according to claim 16 wherein only said discrepancy set, and not said agreement set, is provided to the oracle.

21. A method according to claim 16 wherein said oracle comprises a computerized process which is more costly than the plurality of experts.

22. A method according to claim 16 wherein said selecting comprises using said oracle input and said input from the plurality of experts to estimate recall of an individual one of said plurality of experts.

23. A method according to claim 16 wherein said selecting comprises using said oracle input and said input from the plurality of experts to estimate precision of an individual one of said plurality of experts.

24. A method according to claim 16 wherein each of the expert-based processes has a binary output defining a first desired output value and a second undesired output value and also comprising using said oracle input and said input from the plurality of experts to estimate richness of the body of data including the proportion of the body of data having said first output value.

25. A method according to claim 16 wherein said portion of said discrepancy set comprises a random sample of each of said first and second sets.

26. A method according to claim 16 and also comprising receiving input from a plurality of experts operating a corresponding plurality of computerized expert-based processes on a body of data.

27. A method according to claim 26 wherein said body of data comprises a population of items each associated with a unique computer-readable ID.

28. A method according to claim 27 wherein each of said plurality of computerized expert-based processes comprises:

analyzing an individual item from among said population of items including determining a binary output representing said analyzing; and

generating a computerized representation of said binary output and associating said representation with said ID.

29. A method according to claim 16 and also comprising actuating said subset of better computerized expert-based processes for purposes of receiving input regarding a body of data.