Method and apparatus for search ranking using human input and automated ranking

• US 7,599,911 B2
• Filed: 08/05/2002
• Issued: 10/06/2009
• Est. Priority Date: 08/05/2002
• Status: Expired due to Fees

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Manber, Udi, Chang, Chi-Chao
• Primary Examiner(s): AL HASHEMI, SANA A
• Application Number:
  US10/213,353
• Publication Number:
  US 20040024752A1
• Time in Patent Office: 2,619 Days
• Field of Search: 707 3- 5, 707/10, 707/104.1, 707100-102, 705/26
• US Class Current: 1/1
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

A search system provides search results to searchers in response to search queries and the search results are ranked. The ranking is determined by an automated ranking process in combination with human editorial input. A search system might comprise a query server for receiving a current query, a corpus of documents to which the current query is applied, ranking data storage for storing information from an editorial session involving a human editor and a reviewed query at least similar to the current query, and a rank adjuster for generating a ranking of documents returned from the corpus responsive to the current query taking into account at least the information from the editorial session.

Claims

1. A search system, wherein search results are provided to user search client in response to search queries, comprising:

a query server for receiving a current query from one of a user search client;

a corpus of documents to which the current query is applied;

a review scheduler for generating a reviewed query and transmitting a review assignment based on the reviewed query to a human editor;

a human editor ranking system for presenting review assignments to a plurality of human editors during an editorial session;

ranking data storage for storing information from the editorial session involving a human editor including at least one ranking adjustment performed by the human editor and including at least one representation of a reviewed query at least similar to the current query, wherein the at least one ranking adjustment is an editorial adjustment corresponding to search results resulting from a search using the reviewed query; and

a rank adjuster for generating a ranking of documents returned from the corpus responsive to the current query from the one of the user search clients taking into account at least the information from the editorial session by the human editor, wherein ranking of documents is based on rankings stored in the ranking data storage specific to at least one of the current query and the reviewed query.

2. The search system of claim 1, wherein the corpus of documents are documents available on the World Wide Web.

3. The search system of claim 1, wherein the corpus of documents are Web pages.

4. The search system of claim 1, wherein the information from an editorial session comprises one or more of indications of documents presented to the human editor, indications of ranking changes applied by the human editor during the editorial session and indications of documents from a suggestion list presented to the human editor.

5. The search system of claim 4, wherein the indications of ranking changes applied by the human editor during the editorial session include one or more of promotions, relative promotions, demotions, relative demotions, insertions and deletions.

6. The search system of claim 1, wherein the rank adjuster also takes into account one or more of user context, advertising context and time of day.

7. The search system of claim 1, wherein ranking of the search results is perceptible by an ordering of the documents, or representations of documents, returned from the corpus.

8. A method of generating search results in response to a search query, wherein search results comprise at least a listing of items from a corpus that are responsive to the search query and an indication, for at least some of the items, a ranking among the items, the method comprising:

receiving a current query from at least one search;

applying the current query to the corpus upon receiving the current query;

obtaining search results responsive to the current query;

obtaining an automated ranking of items in the search results;

generating a reviewed query and transmitting a review assignment based on the reviewed query to a human editor;

obtaining human editorial ranking of items from stored editorial data for an editor context related to a reviewed query generated by a review scheduler, the reviewed query being at least related to the current query, wherein the stored editorial data represents ranking decisions made by human editor in one or more editorial sessions in a humand editor ranking system;

adjusting the automated ranking of items in the search results based on the human-editorial ranking to form an adjusted ranking of items in the search results; and

providing the adjusted ranking of items in response to the current query from the at least one searcher.

9. The method of claim 8, wherein the step of responding occurs soon enough after the receiving step to be perceived as happening in substantially real-time.

10. The method of claim 8, wherein the reviewed query and current query are identical.

11. The method of claim 8, wherein the reviewed query and current query are similar.

12. The method of claim 8, wherein the corpus comprises Web pages.

13. The method of claim 8, wherein the step of adjusting comprises inserting items in the search results other than those returned in response to querying the corpus.

14. The method of claim 8, wherein the step of adjusting comprises deleting items from the search results returned in response to querying the corpus.

15. The method of claim 8, wherein the step of adjusting comprises reordering items in the search results returned in response to querying the corpus.

16. The method of claim 8, wherein the editorial data comprises one or more of indications of documents presented to the human editor, indications of ranking changes applied by the human editor during the editorial session and indications of documents from a suggestion list presented to the human editor.

17. The method of claim 16, wherein the indications of ranking changes applied by the human editor during the editorial session include one or more of promotions, relative promotions, demotions, relative demotions, insertions and deletions.

18. The method of claim 8, wherein the step of adjusting takes into account one or more of user context, advertising context and time of day.