Search engine that applies feedback from users to improve search results

• US 9,367,606 B1
• Filed: 05/17/2012
• Issued: 06/14/2016
• Est. Priority Date: 03/18/2005
• Status: Active Grant

• Current Assignee: Pinterest, Inc.
• Original Assignee: Search Engine Technologies, LLC
• Inventors: Lu, Yunshan, Rios, Gordon P., Tanne, Michael
• Primary Examiner(s): Le, Uyen
• Application Number:
  US13/474,628
• Time in Patent Office: 1,489 Days
• Field of Search: 707/723, 707/748, 707/999.003
• US Class Current: 1/1
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/3326 using relevance feedback from the user, e.g. relevance feedback on documents, documents sets, document terms or passages
• CPC Class Codes: G06F 16/338 Presentation of query results
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9536 Search customisation based on social or collaborative filtering
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

The present invention is directed to methods of and systems for ranking results returned by a search engine. A method in accordance with the invention comprises determining a formula having variables and parameters, wherein the formula is for computing a relevance score for a document and a search query; and ranking the document based on the relevance score. Preferably, determining the formula comprises tuning the parameters based on user input. Preferably, the parameters are determined using a machine learning technique, such as one that includes a form of statistical classification.

Claims

1. A method of presenting documents to a user, comprising:

a. assigning relevance scores to a plurality of the documents, by a computer programmed to compute relevance scores based on a relevance model using one or more user input events, wherein at least one of the one or more user input events is an active selection of a user input feature about at least one document and is received outside a context of a presentation of search results of a search, the one or more user input events comprising input about the at least one document from at least one user; and

b. presenting the documents to the user according to the relevance scores.

2. The method of claim 1, wherein presenting the documents is in response to a search query submitted by the user.

3. The method of claim 2, wherein the search query is submitted through a link that has one or more search terms embedded therein.

4. The method of claim 1, further comprising, after presenting the documents to the user, receiving user input about at least one of the documents presented to the user, the user input comprising a user input event.

5. The method of claim 4, further comprises storing the user input event in a database.

6. The method of claim 1, wherein the one or more user input events comprise one of:

voting for a document, rating a document, tagging a document, bookmarking a document, saving a document, liking a document, setting a document as a favorite, expressing interest in a document, voting against a document, and blocking a document.

7. The method of claim 1, wherein presenting the documents comprises displaying user input information based on the input about at least one document by the at least one user.

8. The method of claim 7, wherein displaying user input information comprises displaying a count of user input events.

9. The method of claim 8, wherein the count of user input events comprises one of the number of votes “

for” and

“

likes”

associated with at least one document.

10. The method of claim 9, wherein displaying user input information comprises displaying information about the user associated with the user input event.

11. The method of claim 10, wherein displaying user input information comprises one of:

displaying an image associated with the user and displaying a text description associated with the user.

12. The method of claim 7, wherein displaying user input information comprises displaying a term associated with the document.

13. The method of claim 1, wherein the one or more user input events comprise one of:

the relevance of the document to a query, the bias of the user regarding the document, and the relevance of the document to a subject area.

14. The method of claim 13, wherein the one or more user input events comprise one of:

expressing interest in the document, indicating that the document is relevant, and indicating that the document contains abundant information.

15. The method of claim 14, wherein the one or more user input events comprise rating the document using a rating scale comprising a plurality of values.

16. The method of claim 15, further comprising storing only a latest rating of the document from a user.

17. The method of claim 13, wherein the one or more user input events comprise saving the document for personal reference by the user.

18. The method of claim 17, wherein presenting the documents comprises displaying documents for which the user has given user input comprising a user input event, on a personal Web page of the user.

19. The method of claim 1, wherein the one or more user input events comprise one of:

indicating that the document lacks relevance, voting against the document, and blocking the document from being presented in the future.

20. The method of claim 1, wherein the one or more user input events are provided by a first type of user, the first type of user being associated with one of:

a user login, a user record, and cookie.

21. The method of claim 20, wherein the one or more user input events from the first type of user are used to influence the relevance score of documents to present to a second type of user that has not provided at least one user input event.

22. The method of claim 21, wherein a plurality of the first type of user is treated by the relevance model as a group.

23. The method of claim 22, wherein the user input events from the members of the group are used to generate parameters for the relevance model to assign scores to documents for presenting to the second type of user.

24. The method of claim 22, wherein the user input events from the group are used to generate parameters for the relevance model to detect input from a user attempting to modify the relevance score of a document to have a bias toward the document that is inconsistent with user input events from the group.

25. The method of claim 1, wherein the relevance score is calculated based on a number of the one or more user input events.

26. The method of claim 1, wherein the relevance model comprises a plurality of features including at least one document-specific feature and at least one query-specific feature.

27. The method of claim 26, wherein the plurality of features comprises one of:

frequency of a term within the document, frequency of a term in a hyperlink to the document, frequency of a term in a summary of the document, location of a term within the document, the structure of the document, the length of the document, the type of the document, the date of creation of the document, number of hyperlinks to the document, quality of hyperlinks to the document, terms with which the document has been tagged, or ratings for the document.

28. The method of claim 26, wherein the plurality of features comprises one of:

a user representation, a position of the document in a search results list, a number of times the document has been accessed from a search results list, a time of a user input, a user blocking the document, a user identifier for the document, a user saving the document, a user bookmarking the document, a user tagging the document, and a user rating of the document or otherwise expressing a positive or negative view of the document.

29. The method of claim 26, wherein the relevance model comprises one of a user model and a group model, wherein the user model is for determining a relevance score of a document and a search query for a user and the group model is for determining a relevance score of a document and a search query for a group of users.

30. The method of claim 26, further comprising modifying the relevance model, wherein modifying the relevance model comprises modifying a parameter for at least one feature in the relevance model using at least one user input event.

31. The method of claim 30, wherein modifying the relevance model comprises applying a machine learning technique.

32. The method of claim 31, wherein a machine learning technique comprises a statistical classification.

33. The method of claim 32, wherein a statistical classification comprises one of a logistic regression analysis, a support vector machine, neural networks, boosted trees, random forests, naive Bayes, and graphical models.

34. The method of claim 31, a machine learning technique includes conjugate gradient descent.

35. The method of claim 1, wherein a document is a sponsored link.

36. The method of claim 1, wherein a document comprises one of:

a video, a music file, a picture, and a textual document.

37. The method of claim 1, wherein at least one of the one or more user input events is read from one of a database and a file.

44. The method of claim 1, wherein at least one of the user input events is received within the context of a search.

38. A system for incorporating input to improve the relevance of documents presented to a user, the system comprising:

a computing device comprising a processor coupled to a computer-readable storage medium having processor-executable instructions that configure the computing device to;

a. receive input about at least one document by at least one user, wherein the input is received outside a context of a presentation of a search results of a search;

b. assign a relevance score to each of the documents according to a relevance model, using the input; and

c. present the documents and the input to the user according to the relevance scores of the documents, wherein the documents and the input about the at least one document are different.

39. The system of claim 38, further comprising a database coupled to the computing device, wherein the input is stored in the database and receiving the input comprises receiving the input from the database.

40. The system of claim 38, wherein the processor-executable instructions further configure the computing device to receive input from the user about at least one document in the documents presented to the user.

41. The system of claim 38, wherein the user input is used to modify the relevance model to improve the relevance of the documents presented to the user.

42. The system of claim 38, wherein the processor-executable instructions further configure the computing device to present the documents to the user in response to a search query.

43. The system of claim 38, wherein the processor-executable instructions further configure the computing device to receive the search query from the user.