Search engine that applies feedback from users to improve search results

• US 20070106659A1
• Filed: 03/17/2006
• Published: 05/10/2007
• Est. Priority Date: 03/18/2005
• Status: Active Grant

• Current Assignee: Pinterest, Inc.
• Original Assignee: Search Engine Technologies, LLC
• Inventors: Tanne, Michael, Rios, Gordon, Lu, Yunshan
• Granted Patent: US 8,185,523 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/3326 using relevance feedback from the user, e.g. relevance feedback on documents, documents sets, document terms or passages
• CPC Class Codes: G06F 16/338 Presentation of query results
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9536 Search customisation based on social or collaborative filtering
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

The present invention is directed to methods of and systems for ranking results returned by a search engine. A method in accordance with the invention comprises determining a formula having variables and parameters, wherein the formula is for computing a relevance score for a document and a search query; and ranking the document based on the relevance score. Preferably, determining the formula comprises tuning the parameters based on user input. Preferably, the parameters are determined using a machine learning technique, such as one that includes a form of statistical classification.

Claims

1. A method of ranking documents in response to a search query comprising:

a. determining a formula having variables and parameters, the formula for computing a relevance score for a document and a search query; and

b. ranking the document based on the relevance score.

2. The method of claim 1, further comprising returning a results list in response to the search query, wherein the results list contains a link to the document ranked based on the relevance score.

3. The method of claim 1, wherein determining the formula comprises tuning the parameters based on user input.

4. The method of claim 3, wherein the formula is derived from any one or more features selected from the group consisting of a tag, a term within the document, a location of a term within the document, a structure of the document, a link to the document, a position of the document in a search results list, and a number of times the document has been accessed from a search results list, term scores, section information, link structures, anchor text, and summaries.

5. The method of claim 3, wherein the formula is derived from any one or more features selected from the group consisting of a user representation, a time of a user input, blocking, a user identifier, and a user rating of the document.

6. The method of claim 3, wherein the formula corresponds to a user model and a group model, wherein the user model is for determining a relevance score of the document and a search query for a user and the group model is for determining a relevance score of the document and a search query for a group of users.

7. The method of claim 6, further comprising comparing the user model to the group model to determine a bias toward the document.

8. The method of claim 3, wherein the parameters are determined using a machine learning technique.

9. The method of claim 8, wherein the machine learning technique comprises a form of statistical classification.

10. The method of claim 9, wherein the statistical classification is any one of a logistic regression analysis, a support vector machine, neural networks, boosted trees, random forests, naive Bayes, and graphical models.

11. The method of claim 8, wherein the machine learning technique uses a conjugate gradient descent.

12. The method of claim 3, wherein one or more variables of the formula are generated from the user input.

13. The method of claim 12, wherein the user input is one or more of tagging the document, rating a relevance of the document, and blocking the document.

14. The method of claim 1, wherein the relevance score is for a user registered on a search engine site and is used to determine a relevance score for a document and a search query for a user not registered on the search engine site.

15. A method of organizing a search results list:

a. updating a relevance formula comprising variables and parameters, wherein the relevance formula is for determining a relevance score for each of multiple results returned in response to a search query; and

b. returning a search results list containing the multiple results in response to the search query, wherein the multiple results are ranked using a relevance score for each of the multiple results.

16. The method of claim 15, wherein updating the relevance formula comprises updating the parameters.

17. The method of claim 16, wherein the parameters are updated using user input.

18. The method of claim 17, wherein the user input is one of tagging a document, rating a relevance of a document in response to a search query, and blocking a document.

19. The method of claim 15, wherein the relevance formula is derived using a machine learning technique.

20. The method of claim 19, wherein the machine learning technique comprises a form of statistical classification.

21. The method of claim 16, wherein the statistical classification is any one of logistic regression analysis, support vector machine, neural networks, boosted trees, random forests, naive Bayes, and graphical models.

22. The method of claim 15, wherein the machine learning technique uses a conjugate gradient descent.

23. A method of ranking results returned by a first search engine comprising:

a. receiving a first results list containing documents ranked by a first search engine in response to a search query;

b. receiving a second results list containing documents ranked by a second search engine in response to the search query;

c. determining a formula having variables and parameters, the formula for determining a relevance score for a document in response to the search query; and

d. returning a third results list containing documents in the first results list and the second results list ranked using a relevance score for each document in response to the search query.

24. The method of claim 23, wherein the formula is determined using user input.

25. The method of claim 23, wherein the formula is determined using ranks of the documents in the first results list and ranks of the documents in the second results list.

26. The method of claim 23, wherein the parameters are determined using a machine learning technique.

27. The method of claim 26, wherein the machine learning technique comprises a form of statistical classification.

28. The method of claim 27, wherein the statistical classification is any one of a logistic regression analysis, a support vector machine, neural networks, boosted trees, random forests, naive Bayes, and graphical models.

36. The system of claim 28, wherein the relevance model is for determining a set of user specific parameters for determining a relevance score for a search query and a document for a specific user.

37. The system of claim 36, wherein the relevance model is also for determining parameters of a group model for determining a relevance score for a search query and a document for a group of users.

38. The system of claim 28, further comprising a second database for storing one or more features for updating the relevance model.

39. The system of claim 38, wherein the features are selected from the group consisting of a tag, a term within a document, a location of a term within a document, a structure of a document, a link to a document, a position of a document in a search results list, a number of times a document has been accessed from a search results list, term scores, section information, link structures, anchor text, and summaries.

40. The system of claim 38, wherein the features are selected from the group consisting of a user representation, a time of a user input, blocking, a user identifier, and a user rating of the document.

41. The system of claim 38, wherein the features comprise any one or more of a hyperlink to a document, a position of a document in a search results list, a user rating of a document, and a number of times a document has been accessed from a search results list.

29. A system for returning a ranked list of results in response to a search query comprising:

a. a first database for storing user inputs for ranking results from the multiple results in response to a first search query; and

b. a relevance model coupled to the first database, the relevance model for using the user inputs to calculate a relevance score for each result from the multiple results for a second search query.

30. The system of claim 29, further comprising a search engine coupled to the first database, the search engine for receiving a search query, ranking the multiple results based on a relevance score for each result, and returning a search results list containing the ranked multiple results.

31. The system of claim 29, wherein the relevance model is configured to be updated using the user input.

32. The system of claim 29, wherein the relevance model is configured to use a machine learning technique.

33. The system of claim 32, wherein the machine learning technique comprises a form of statistical classification.

34. The system of claim 33, wherein the statistical classification is any one of a logistic regression analysis, a support vector machine, neural networks, boosted trees, random forests, naive Bayes, and graphical models.

35. The system of claim 33, wherein the machine learning technique uses a conjugate gradient descent.

42. A system for organizing multiple results in a search results list comprising:

a. a relevance model for using user input to determine a relevance score for a document and a search query;

b. a search engine coupled to the relevance model, the search engine for receiving a search query and returning a search results list containing results that are ranked according to a relevance score for each document and the search query; and

c. a database coupled to the search engine and the relevance model, the database for storing a set of features used by the relevance model to determine a relevance score for a document and a search query.

43. The system of claim 42, wherein the database comprises a user database coupled to the search engine and a user input database coupled to the relevance model, wherein the user database is for storing user input on the search results list and the user input database is for storing the set of features.

44. The system of claim 43, wherein the set of features includes any one or more of a tag, a term within a document, a location of a term within a document, a structure of a document, a link to a document, a position of a document in a search results list, a number of times a document has been accessed from a search results list, term scores, section information, link structures, anchor text, and summaries.

45. The system of claim 43, wherein the set features includes any one or more of a user representation, a time of a user input, blocking, a user identifier, and a user rating of a document.

46. The system of claim 42, further comprising a Web server coupled to the database.

47. The system of claim 42, further comprising a document index coupled to both the relevance model and the search engine.

48. The system of claim 42, wherein the relevance model is configured to use a machine learning technique.

49. The system of claim 48, wherein the machine learning technique comprises a form of statistical classification.

50. The system of claim 49, wherein the statistical classification is any one of a logistic regression analysis, a support vector machine, neural networks, boosted trees, random forests, naive Bayes, and graphical models.

51. The system of claim 48, wherein the machine learning technique uses a conjugate gradient descent.