PREFERENCE JUDGEMENTS FOR RELEVANCE

• US 20090271389A1
• Filed: 04/24/2008
• Published: 10/29/2009
• Est. Priority Date: 04/24/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Dumais, Susan T., Chickering, David M., Carterette, Benjamin Austin, Bennett, Paul N.
• Granted Patent: US 8,069,179 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

The claimed subject matter provides a system that trains or evaluates ranking techniques by employing or obtaining relative preference judgments. The system can include mechanisms that retrieve a set of documents from a storage device, combine the set of documents with a query orjudgment task received via an interface to form a comparative selection panel, and present the comparative selection panel for evaluation by an assessor. The system further requests the assessor to make a selection as to which document included in the set of documents and presented in the comparative selection panel most satisfies the query or judgment task, and thereafter produces a comparative assessment of the set of documents based on the selections elicited from the assessor and associated with the set of documents.

Claims

1. A system effectuated on a machine that at least one of trains or evaluates ranking techniques by at least one of employing or obtaining relative preference judgments, comprising:

a component that retrieves a set of documents from a storage device, combines the set of documents with a judgment task received via an interface to form a comparative selection panel and presents the comparative selection panel for evaluation by an assessor, the component requests the assessor to make a selection as to which document included in the set of documents most satisfies the query, the component produces a ranked comparative assessment of the set of documents based at least in part on the selection elicited from the assessor.

2. The system of claim 1, the component dynamically chooses another set of documents based at least in part on a previous selection made by the assessor as to which document included in the set of documents most satisfied the query.

3. The system of claim 1, the component employs a property of transitivity in order to form the comparative selection panel for evaluation.

4. The system of claim 1, the component automatically determines whether the comparative selection panel should comprise at least one of an absolute judgment panel, a binary selection panel, or a graded preference panel.

5. The system of claim 4, the absolute judgment panel presents a single document for the assessor to evaluate, the absolute judgment panel requests the assessor to select from at least one ranking criteria, the ranking criteria selected by the assessor associated with the single document.

6. The system of claim 5, selection by the assessor of the bad assessment causes elimination of the single document associated with the bad assessment.

7. The system of claim 4, the binary selection panel presents at least two documents from the set of documents for review by the assessor, the assessor selects one of the at least two documents as being binarily preferable over another of the at least two documents.

8. The system of claim 4, the graded preference panel presents at least two documents for the set of documents for review by the assessor, the assessor identifies one of the at least two documents as being preferable, the one of the at least two documents identified as being preferable retained as one of the at least two documents for a next judgment.

9. The system of claim 4, the component utilizes a probabilistic model to determine whether the comparative selection panel should comprise at least one of the absolute judgment panel, the binary selection panel, or the enhanced binary selection panel.

10. The system of claim 9, the component employs a property of transitivity to learn the probabilistic model utilized to determine whether the comparative selection panel should comprise at least one of the absolute judgment panel, the binary selection panel, or the enhanced binary selection panel.

11. A machine implemented method that one or more of trains or evaluates ranking techniques by one or more of utilizing or acquiring relative preference judgments, comprising:

obtaining a plurality of documents from at least one of a storage device or a network;

receiving a judgment task utilized by an assessor for a comparative evaluation of the plurality of documents;

generating a selection panel that includes at least one document from the plurality of documents or the judgment task for evaluation by the assessor;

presenting the selection panel to the assessor and receiving a response from the assessor;

comparatively ranking the response from the assessor with a previously elicited response from the assessor; and

producing a ranked list of documents based at least in part on the ranking of the response from the assessor.

12. The method of claim 11, the presenting further comprising reducing a cognitive load imposed on the assessor by ensuring that at least one of the plurality of documents remains static, until all judgments with respect to the at least one of the plurality of documents have been obtained, in relation to other documents included in the plurality of documents during evaluation by the assessor of the plurality of documents in its entirety.

13. The method of claim 11, the generating further comprising identifying the selection panel for evaluation by the assessor based at least in part on the judgment task and the at least one document from the plurality of documents, the identifying the selection panel including determining whether the selection panel should comprise at least one of an absolute judgment panel, a binary selection panel, or an enhanced binary selection panel.

14. The method of claim 13, the absolute judgment panel includes the judgment task and the at least one document and further includes a preference ranking, the preference ranking associated with the at least one document when the assessor selects the preference ranking as the response.

15. The method of claim 14, the preference ranking affiliated with a lowest preference and associated with the at least one document.

16. The method of claim 13, the binary selection panel presents a pair of documents from the plurality of documents for evaluation by the assessor, the assessor identifies one of the pair documents as being preferred over the other one of the pair of documents.

17. The method of claim 11, one or more of the generating, the presenting, the receiving, the ranking, or the producing further utilizing a property of transitivity.

18. The method of claim 11, one or more of the generating, the presenting, the receiving, the ranking, or the producing further employing a dynamically trainable probabilistic model.

19. A system that trains or evaluates ranking methodologies by employing or obtaining relative preference judgments from assessors, comprising:

means for producing a comparative selection panel that includes a query and a set of documents, the query and the set of documents included in the comparative selection panel based on a result from a means for utilizing a trainable probabilistic model;

means for ranking a response from the assessors based on a previously elicited response from the assessors; and

means for generating a ranked list of assessments associated with the set of documents based at least in part on a result obtained from the means for ranking.

20. The system of claim 19, at least one of the means for producing, the means for ranking, or the means for generating further employs a property of transitivity to at least one of producing the comparative selection panel, ranking the response from the assessors, or generating the ranked list of assessments associated with the set of documents.