Search engine that applies feedback from users to improve search results

• US 8,185,523 B2
• Filed: 03/17/2006
• Issued: 05/22/2012
• Est. Priority Date: 03/18/2005
• Status: Active Grant

• Current Assignee: Pinterest, Inc.
• Original Assignee: Search Engine Technologies, LLC
• Inventors: Lu, Yunshan, Rios, Gordon P., Tanne, Michael
• Primary Examiner(s): Fleurantin, Jean B
• Assistant Examiner(s): GIRMA, ANTENEH B
• Application Number:
  US11/378,756
• Publication Number:
  US 20070106659A1
• Time in Patent Office: 2,258 Days
• Field of Search: 707/1, 707/2, 707/3, 707/5, 707/7, 707/101, 707/103.R, 709/203
• US Class Current: 707/723
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/3326 using relevance feedback from the user, e.g. relevance feedback on documents, documents sets, document terms or passages
• CPC Class Codes: G06F 16/338 Presentation of query results
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9536 Search customisation based on social or collaborative filtering
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

The present invention is directed to methods of and systems for ranking results returned by a search engine. A method in accordance with the invention comprises determining a formula having variables and parameters, wherein the formula is for computing a relevance score for a document and a search query, and ranking the document based on the relevance score. Preferably, determining the formula comprises tuning the parameters based on user input. Preferably, the parameters are determined using a machine learning technique, such as one that includes a form of statistical classification.

Claims

1. A computer-implemented method of ranking document returned in a results list in response to a search query, comprising

entering a search query;

displaying a results list of documents returned in response to the search query;

receiving and storing input from the user indicating the relevance to the user of a document in the results list of documents;

modifying the weight of previous input from the user indicating the relevance of the document to the user;

associating the user input indicating the relevance to the user of the document in the results list with the search query and with the document;

determining a relevance formula having variables and parameters, the relevance formula for computing a relevance score for the document and the search query, wherein the variables comprise a plurality of features, a feature of the plurality of features comprises one of;

a frequency of occurrence of a term in the document, a frequency of hyperlinks containing a term in the document, and a frequency of occurrence of a term in an abstract of the document, and the parameters comprise a plurality of weighting factors corresponding to each of the features;

ranking the document based on the relevance score, wherein the relevance score is dependent on the user input associated with the document and the search query; and

presenting the ranked results list of documents to the user.

2. The method of claim 1, wherein the user input is blocking the document, and the relevance score is lowered.

3. The method of claim 1, wherein the user input regarding the relevance of the document is a vote, and the relevance of the document is increased if the vote is in favor of the document and the relevance of the document is decreased if the vote disfavors the document.

4. The method of claim 1, wherein the weight of a user input is reduced if the user input is repetitive of previous user input.

5. The method of claim 1, wherein the weight of a user input is reduced in relation to a deviation of the user input from previously stored input by other users.

6. The method of claim 1, wherein the previously stored input from a user is retrieved prior to the initiation of the search query.

7. The method of claim 1, wherein determining the relevance formula comprises utilizing the user input to update the relevance formula and modify at least one of the weighting factors.

8. The method of claim 1, wherein at least one of the plurality of features is selected from the group consisting of a tag, a frequency of occurrence of a term in the summary information in the document, a structure of the document, the length of the document, the type of the document, the date of creation of the document, a link to the document, a position of the document in a search results list, a number of times the document has been accessed from a search results list, term scores, and link structures.

9. The method of claim 1, wherein at least one of the plurality of features is selected from the group consisting of a user representation, a time of a user input, a user blocking the document, a user identifier for the document, a user saving the document, a user bookmarking the document, a user tagging the document, a user rating of the document, and a user indicating one of a positive and a negative, relevance of the document to the user and the search query.

10. The method of claim 7, wherein the relevance formula corresponds to a user model and a group model, wherein the user model is for determining a relevance score of the document and a search query for a user and the group model is for determining a relevance score of the document and a search query for a group of users.

11. The method of claim 10, further comprising comparing the user model to the group model to determine a bias toward the document, and decreasing the weight of the user model if the user model bias diverges from the group model bias.

12. The method of claim 7, wherein the parameters are determined using machine learning technique.

13. The method of claim 12, wherein the machine learning technique comprises a form of statistical classification.

14. The method of claim 13, wherein the statistical classification is any one of a logistic regression analysis, a support vector machine, neural networks, boosted trees, random forests, naive Bayes, and graphical models.

15. The method of claim 12, wherein the machine learning technique uses a conjugate gradient descent.

16. The method of claim 1, wherein the relevance score is for a user registered on a search engine site and is used to determine a relevance score for a document and a search query for a user not registered on the search engine site.

17. The method of claim 1, further comprising updating the relevance formula wherein updating the relevance formula comprises updating the parameters, and the plurality of features comprise at least two from the group consisting of a tag, a term within the document, a frequency of occurrence of a term in the link text of the document, a frequency of occurrence of a term in the abstract of the document, a frequency of occurrence of a term in the summary of the document, a frequency of occurrence of a term in anchor text in the document, a frequency of occurrence of a term in the summary information in the document, a structure of the document, the length of the document, the type of the document, the date of creation of the document, a link to the document, a position of the document in a search results list, a number of times the document has been accessed from a search results list, term scores, link structures, a user representation, a time of a user input, a user blocking the document, a user identifier for the document, a user saving the document, a user bookmarking the document, a user tagging the document, and a user rating of the document.

18. The method of claim 1, wherein the parameters are updated using the user feedback for each of the multiple results.

19. The method of claim 18, wherein the user feedback is one of tagging a document, rating a relevance of a document in response to a search query, and blocking a document.

20. The method of claim 1, wherein the relevance model is for determining a set of user specific parameters for determining a relevance score for a search query and a document for a specific user.

21. The method of claim 20, wherein the relevance model is also for determining parameters of a group model for determining a relevance score for a search query and a document for a group of users.

22. The method of claim 1, wherein the plurality of features comprises the frequency of occurrence of a term within the abstract of a document, and a user rating of the relevance of the document to the user.

23. A non-transitory computer readable medium programmed with executable instructions for implementing the method of claim 1.

24. A system for ranking documents returned in a results list of documents returned in response to a search query, the system comprising a computing system, including:

an input system configured to receive a search query;

a display system configured to display the results list of documents returned in response to the search query;

the input system is further configured to receive input from the user indicating the relevance to the user of a document in the results list of documents;

a storage medium configured to store the input from the user indicating the relevance to the user of the document;

a processor coupled to a memory programmed with processor executable instructions to;

reduce the weight of previous input from the user indicating the relevance of the document to the user;

associate the user input indicating the relevance to the user of the document with the search query and with the document;

determine a relevance formula having variables and parameters, the relevance formula for computing a relevance score for the document and the search query, wherein the variables comprise a plurality of features, the parameters comprise a plurality of weighting factors corresponding to each of the features, and a feature of the plurality of features comprises one of a frequency of occurrence of a term in one of the document, a link text of the document, an abstract of the document, a summary of the document, and an anchor text in the document; and

rank the document based on the relevance score, wherein the relevance score is dependent of the user input associated with the document and the search query;

wherein the display system is further configured to the present the ranked results list of documents to the user.