Search engine that applies feedback from users to improve search results

• US 10,157,233 B2
• Filed: 05/23/2016
• Issued: 12/18/2018
• Est. Priority Date: 03/18/2005
• Status: Active Grant

• Current Assignee: Pinterest, Inc.
• Original Assignee: Pinterest, Inc.
• Inventors: Lu, Yunshan, Rios, Gordon P., Tanne, Michael
• Primary Examiner(s): Le, Uyen T
• Application Number:
  US15/162,444
• Publication Number:
  US 20160299981A1
• Time in Patent Office: 939 Days
• Field of Search: None
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/3326 using relevance feedback from the user, e.g. relevance feedback on documents, documents sets, document terms or passages
• CPC Class Codes: G06F 16/338 Presentation of query results
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9536 Search customisation based on social or collaborative filtering
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

The present invention is directed to methods of and systems for ranking results returned by a search engine. A method in accordance with the invention comprises determining a formula having variables and parameters, wherein the formula is for computing a relevance score for a document and a search query; and ranking the document based on the relevance score. Preferably, determining the formula comprises tuning the parameters based on user input. Preferably, the parameters are determined using a machine learning technique, such as one that includes a form of statistical classification.

Claims

1. A method of ranking documents in response to a search query comprising:

retrieving previously stored user input regarding a relevance of a document to the search query;

associating the previously stored user input with the search query and the document;

determining a relevance formula having variables and parameters, the relevance formula for computing a relevance score for the document and the search query, wherein the variables comprise a plurality of features, and the parameters comprise a plurality of coefficients corresponding to each of the plurality of features; and

ranking the document based on the relevance score, wherein the relevance score is dependent at least in part on the previously stored user input.

2. The method of claim 1, wherein at least one of the plurality of features includes one or more of a user representation, a time of a user input, a user blocking a document, a user identifier for the document, a user saving the document, a user bookmarking the document, a user tagging the document, a user rating of the document, a user indicating a positive relevance of the document or the search query, or a user indicating a negative relevance of the document or the search query.

3. The method of claim 1, wherein the relevance formula is for a user registered on a search engine site and is used to determine the relevance score for a user not registered on the search engine site.

4. The method of claim 1, wherein determining the relevance formula comprises utilizing the previously stored user input to update the relevance formula and modify at least one of the coefficients.

5. The method of claim 4, wherein at least one of the plurality of features includes one or more of a tag, a frequency of occurrence of a term within the document, a frequency of occurrence of a term in a link text of the document, a frequency of occurrence of a term in an abstract of the document, a frequency of occurrence of a term in a summary of the document, a frequency of occurrence of a term in an anchor text in the document, a frequency of occurrence of a term in a summary information in the document, a structure of the document, a length of the document, a type of the document, a date of creation of the document, a link to the document, a position of the document in a search results list, a number of times the document has been accessed from a search results list, a term score, or a link structure.

6. The method of claim 4, wherein:

the relevance formula corresponds to a user model and a group model;

the user model is for determining a relevance score of the document and the search query for a user; and

the group model is for determining a relevance score of the document and the search query for a group of users.

7. The method of claim 6, further comprising comparing the user model to the group model to determine a bias toward the document, and decreasing a weight of the user model if the user model bias diverges from the group model bias.

8. The method of claim 1, wherein the parameters are determined using a machine learning technique.

9. The method of claim 8, wherein the machine learning technique comprises a form of statistical classification.

10. The method of claim 9, wherein the form of statistical classification includes at least one of a logistic regression analysis, a support vector machine, a neural network, a boosted tree, a random forest, a naive Bayes, or a graphical model.

11. The method of claim 8, wherein the machine learning technique uses a conjugate gradient descent.

12. A method of organizing a search results list indicating a plurality of documents, each of the plurality of documents including textual information, the search results list returned in response to a search query, the method comprising:

presenting the search results list in response to the search query;

receiving and storing feedback for a document indicated in the search results list, the feedback being a relevance to the user of the document

associating the feedback with the document and the search query;

updating a relevance formula comprising variables and parameters, wherein the relevance formula is for determining a relevance score for each of the plurality of documents utilizing the feedback for the document, wherein the variables comprise a plurality of features, the parameters comprise a plurality of coefficients corresponding to each of the plurality of features, updating the relevance formula comprises determining a value for at least one of the plurality of coefficients, and the plurality of features comprise one of a rating of a document or a blocking of a document; and

recording the search results list, wherein the plurality of documents are ranked based at least in part on a relevance score determined for each of the plurality of documents based at least in part on the relevance formula.

13. The method of claim 12, wherein updating the relevance formula comprises updating the parameters, and the plurality of features comprise at least two of a tag, a term within the document, a frequency of occurrence of a term in a link text of the document, a frequency of occurrence of a term in an abstract of the document, a frequency of occurrence of a term in a summary of the document, a frequency of occurrence of a term in an anchor text in the document, a frequency of occurrence of a term in a summary information in the document, a structure of the document, a length of the document, a type of the document, a date of creation of the document, a link to the document, a position of the document in a search results list, a number of times the document has been accessed from a search results list, a term score, a link structure, a user representation, a time of a user input, a user blocking of the document, a user identifier for the document, a user saving the document, a user bookmarking the document, a user tagging the document, or a user rating of the document.

14. The method of claim 12, wherein the parameters are updated using the feedback.

15. The method of claim 14, wherein the feedback includes at least one of tagging the document, rating a relevance of the document in response to the search query, or a blocking of the document.

16. The method of claim 12, wherein the relevance formula is derived using a machine learning technique.

17. The method of claim 16, wherein the machine learning technique comprises a form of statistical classification.

18. The method of claim 17, wherein the statistical classification is any one of logistic regression analysis, support vector machine, neural networks, boosted trees, random forests, naive Bayes, and graphical models.

19. The method of claim 16, wherein the machine learning technique uses a conjugate gradient descent.