Preference judgements for relevance
First Claim
1. A system comprising:
- a processor;
one or more memories communicatively coupled to the processor, the one or more memories having stored instructions that, when executed at the processor, causes the processor to implement a component that at least one of trains or evaluates ranking techniques by at least one of employing or obtaining relative preference judgments, the component configured to retrieve a set of documents from a storage device, combine the set of documents with a judgment task received via an interface to form a comparative selection panel and present the comparative selection panel for evaluation by an assessor, the component requests the assessor to compare documents in the set of documents with each other and make a selection as to which document included in the set of documents most satisfies the judgment task, the component produces a ranked comparative assessment of the set of documents based at least in part on the selection elicited from the assessor,the component automatically determining whether to present one or more of a plurality of interfaces to the assessor based on the judgment task, the plurality of interfaces including an absolute judgment panel, a binary selection panel, and a graded preference panel, andwherein the absolute judgment panel presents a single document for the assessor to evaluate, the absolute judgment panel requests the assessor to select at least one ranking criteria, the ranking criteria selected by the assessor being associated with the single document,the binary selection panel presents at least two documents from the set of documents for review by the assessor, the assessor selects one of the at least two documents as being binarily preferable over another of the at least two documents, andthe graded preference panel presents at least two documents from the set of documents for review by the assessor, the assessor identifies one of the at least two documents as being preferable, the one of the at least two documents identified as preferable being retained as one of the at least two documents for a next judgment.
2 Assignments
0 Petitions
Accused Products
Abstract
The claimed subject matter provides a system that trains or evaluates ranking techniques by employing or obtaining relative preference judgments. The system can include mechanisms that retrieve a set of documents from a storage device, combine the set of documents with a query or judgment task received via an interface to form a comparative selection panel, and present the comparative selection panel for evaluation by an assessor. The system further requests the assessor to make a selection as to which document included in the set of documents and presented in the comparative selection panel most satisfies the query or judgment task, and thereafter produces a comparative assessment of the set of documents based on the selections elicited from the assessor and associated with the set of documents.
-
Citations
15 Claims
-
1. A system comprising:
-
a processor; one or more memories communicatively coupled to the processor, the one or more memories having stored instructions that, when executed at the processor, causes the processor to implement a component that at least one of trains or evaluates ranking techniques by at least one of employing or obtaining relative preference judgments, the component configured to retrieve a set of documents from a storage device, combine the set of documents with a judgment task received via an interface to form a comparative selection panel and present the comparative selection panel for evaluation by an assessor, the component requests the assessor to compare documents in the set of documents with each other and make a selection as to which document included in the set of documents most satisfies the judgment task, the component produces a ranked comparative assessment of the set of documents based at least in part on the selection elicited from the assessor, the component automatically determining whether to present one or more of a plurality of interfaces to the assessor based on the judgment task, the plurality of interfaces including an absolute judgment panel, a binary selection panel, and a graded preference panel, and wherein the absolute judgment panel presents a single document for the assessor to evaluate, the absolute judgment panel requests the assessor to select at least one ranking criteria, the ranking criteria selected by the assessor being associated with the single document, the binary selection panel presents at least two documents from the set of documents for review by the assessor, the assessor selects one of the at least two documents as being binarily preferable over another of the at least two documents, and the graded preference panel presents at least two documents from the set of documents for review by the assessor, the assessor identifies one of the at least two documents as being preferable, the one of the at least two documents identified as preferable being retained as one of the at least two documents for a next judgment. - View Dependent Claims (2, 3, 4, 5, 6)
-
-
7. A machine implemented method, comprising:
-
obtaining a plurality of documents from at least one of a storage device or a network; receiving a judgment task utilized by an assessor for a comparative evaluation of the plurality of documents; generating a selection panel that includes at least one document from the plurality of documents for evaluation by the assessor, the generating including determining whether to present one or more of a plurality of interfaces to the assessor based on the judgment task, the plurality of interfaces including an absolute judgment panel, a binary selection panel, and a graded preference panel, wherein the absolute judgment panel presents a single document for the assessor to evaluate, the absolute judgment panel requests the assessor to select at least one ranking criteria associated with the single document, the binary selection panel presents at least two documents from the plurality of documents for review by the assessor, the assessor selects one of the at least two documents as being binarily preferable over another of the at least two documents, and the graded preference panel presents at least two documents from the plurality of documents for review by the assessor, the assessor identifies one of the at least two documents as being preferable, the one of the at least two documents identified as preferable being retained as one of the at least two documents for a next judgment; presenting the selection panel to the assessor and receiving a response from the assessor, the response based on a comparison of documents comprising the plurality of documents with each other; comparatively ranking the response from the assessor with a previously elicited response from the assessor; and producing a ranked list of documents based at least in part on the ranking of the response from the assessor. - View Dependent Claims (8, 9, 10, 11, 12, 13)
-
-
14. A system comprising:
-
one or more processors; memory communicatively coupled to the one or more processors, the memory having stored instructions that, when executed at the processor, causes the processor to implement a component, the component comprising; means for utilizing a trainable probabilistic model; means for producing a comparative selection panel that includes a query and a set of documents, the query and the set of documents included in the comparative selection panel based on a result from the trainable probabilistic model; means for ranking a response from assessors based on a previously elicited response from the assessors, the response based on a comparison of documents comprising the set of documents with each other; and means for generating a ranked list of assessments associated with the set of documents based at least in part on a result obtained from the means for ranking, wherein the component automatically determines whether to present one or more of a plurality of interfaces to an assessor based on the query, the plurality of interfaces including an absolute judgment panel, a binary selection panel, and a graded preference panel, and wherein the absolute judgment panel presents a single document for the assessor to evaluate, the absolute judgment panel requests the assessor to select at least one ranking criteria associated with the single document, the binary selection panel presents at least two documents from the set of documents for review by the assessor, the assessor selects one of the at least two documents as being binarily preferable over another of the at least two documents, and the graded preference panel presents at least two documents from the set of documents for review by the assessor, the assessor identifies one of the at least two documents as being preferable, the one of the at least two documents identified as preferable being retained as one of the at least two documents for a next judgment. - View Dependent Claims (15)
-
Specification