Systems for and methods of finding relevant documents by analyzing tags

• US 20070185858A1
• Filed: 08/02/2006
• Published: 08/09/2007
• Est. Priority Date: 08/03/2005
• Status: Active Grant

• Current Assignee: Pinterest, Inc.
• Original Assignee: Pinterest, Inc.
• Inventors: Tanne, Michael, Lu, Yunshan
• Granted Patent: US 9,715,542 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A method of determining relevancies of objects to a search query includes associating multiple tags with multiple objects, recording bookmarks to the multiple objects, or both, and determining a relevance score for each of the multiple objects and a search query. One embodiment of the method combines full-text relevance algorithms with tag relevance algorithms. Other embodiments include statistical relevance algorithms such as statistical classification or rank regression algorithms. When a user executes a search query, a results list containing the objects is returned, with the objects organized based on the relevance scores. The objects are organized by, for example, listing those with the highest relevance scores first or by marking them with an indication of their relevance. Preferably, relevance scores for a tag-object pair are based on a number of times a term in the tag has been associated with the object, a number of tags associated with the object, a number of times that the tag has been associated with the multiple objects, a number of tag-object pairs that contain a term in the tag, a number of tag-object pairs that contain a reference to the object, or any combination of these.

Claims

1. A method of determining relevancies of multiple objects to a search query, comprising:

associating multiple tags with the multiple objects, recording bookmarks to the multiple objects, or both; and

determining a relevance for each of the multiple objects and the search query.

2. The method of claim 1, wherein each tag from the multiple tags contains one or more terms, the method further comprising associating each term from the one or more terms with an object, thereby defining one or more corresponding term-object pairs.

3. The method of claim 2, further comprising determining for each term-object pair a term score indicating a degree of relevance between the term and the object.

4. The method for claim 3, wherein determining a relevance score for a tag-object pair comprises combining the term scores for the term-object pairs for each term in the tag.

5. The method of claim 4, wherein combining the term scores comprises summing the term scores.

6. The method of claim 4, wherein combining the term scores comprises weighing each term score with a weight and summing the weighted term scores.

7. The method of claim 2, wherein a relevance score for a tag-object pair is determined from a number of times a term in the tag has been associated with the object, a number of tags associated with the object, a number of times that the tag has been associated with the multiple objects, or any combination of these.

8. The method of claim 2, wherein a relevance score for a tag-object pair is determined from a number of tag-object pairs that contain a term in the tag, a number of tag-object pairs that contain a reference to the object, or both.

9. The method of claim 1, further comprising:

associating a tag with an object from the multiple objects by a first user;

performing a search query containing one or more terms in the tag by a second user;

organizing the multiple objects in the results list based on the relevance scores, thereby defining an organized results list; and

returning the organized results list to the second user.

10. The method of claim 9, wherein a relevance score for an object and the search query is determined from relevance scores for each of term of the search query present in the object or associated with the object.

11. The method of claim 9, wherein a relevance score for a tag-object pair from the multiple tag-object pairs is determined from a number of tags that the first user has associated with any of the multiple objects, a number of objects that the first and second users have associated with tags, or both.

12. The method of claim 9, wherein a relevance score for a tag-object pair is determined from a confidence rating of a selected one of the first user, the second user, or both.

13. The method of claim 12, wherein the confidence rating is determined from a rating for tags that the selected user has associated with objects, a similarity metric between search activities for the first and second users, a connection metric between the first and second users, or any combination of these.

14. The method of claim 9, wherein organizing the multiple objects comprises ordering the multiple objects based on the relevance scores.

15. The method of claim 9, wherein organizing the multiple objects comprises marking at least one of the multiple objects with a graphic element.

16. The method of claim 1, wherein associating a tag with an object comprises entering the tag in an area presented to a user, rating the tag, blocking a link to the object, selecting the tag, or selecting the object.

17. The method of claim 1, wherein associating a tag with an object comprises examining a bookmark or performing a search for the object using the tag.

18. The method of claim 1, wherein the multiple objects comprise hyperlinks to Web pages or groups of hyperlinks to Web pages.

19. The method of claim 1, wherein the multiple objects comprise hyperlinks or groups of hyperlinks to text, images, photographs, tags, groups of tags, subject areas, concepts, user profiles, answers, audio files, video files, software, or any combination of these.

20. The method of claim 1, wherein a tag crawler associates at least one of the multiple tags with at least one of the multiple objects.

21. A method of populating a system used to return objects organized in a results list, comprising:

storing in a tag database multiple tags associated with multiple objects; and

storing in an index database relevance scores between the multiple tags and the multiple objects, wherein the relevance scores are used to organize the multiple documents in an organized results list.

22. The method of claim 21, wherein storing in a tag database multiple tags comprises storing terms that form the multiple tags, and further wherein storing relevance scores comprises scoring relevance scores indicating a relevance between the terms and the objects.

23. The method of claim 22, further comprising storing in the index database multiple indices, each index entry corresponding to a term from the multiple terms, a corresponding object from multiple objects, and a corresponding relevance score between the term and the object.

24. The method of claim 23, wherein each corresponding relevance score between a term and an object is related to a confidence of a user who associated the term with the object.

25. The method of claim 23, wherein a relevance score between a term and an object is determined from a statistical classification or rank regression algorithm.

26. The method of claim 25, wherein the statistical classification or rank regression algorithm is any one of logistic regression, support vector machines, classification or regression tree, and boosted tree ensembles.

27. The method of claim 21, further comprising:

presenting a results list to a user in response to a search query containing a term;

associating the term to an object contained in the results list by the user; and

determining a relevance score between the term and the object.

28. The method of claim 27, wherein a relevance score between an object from the multiple objects and its associated tag is determined from one or more of a number of times that the tag has been associated with the object, a total number of tags associated with the object, a number of times that the tag has been associated with any of the multiple objects, a number of tags that have been associated with all of the multiple objects, a time that the tag was associated with the object, a number of times that the object has been bookmarked, the value and number of ratings applied to the object, or any combination of these.

29. A method of organizing multiple objects for display in a results list, comprising:

correlating terms in a search query with tags associated with multiple objects; and

returning a results list containing the multiple objects organized based on the correlations.

30. The method of claim 29, further comprising:

executing the search query, thereby generating a first list of objects; and

organizing the multiple objects based on the correlations.

31. The method of claim 30, further comprising associating tags with objects.

32. The method of claim 30, wherein each correlation corresponds to a relevance metric.

33. The method of claim 29, further comprising applying a statistical classification or rank regression algorithm to the multiple objects to determine relevance metrics between the multiple objects and terms in the search query.

34. The method of claim 33, wherein the statistical classification or rank regression algorithm is any one of logistic regression, support vector machines, classification or regression tree, or boosted tree ensembles.

35. A system for returning a search results list in response to a search query, the system comprising:

a tag database for storing tags associated with objects; and

a tag analyzer coupled to the tag database, wherein the tag analyzer is programmed to determine a relevance score of multiple objects to a search query.

36. The system of claim 35, further comprising an object index for storing relevance scores between tags associated with objects and objects, between bookmarks to objects, or both.

37. The system of claim 35, wherein a relevance score is determined by summing weighted relevance scores for terms that form a tag and objects.

38. The system of claim 35, wherein a relevance score between a search query containing terms and an object is determined from a number of tags that contain terms in the search query, a number of times that a tag included in the search query is included in the tag database, a number of tags that have been associated with the object, a number of terms in the tag and in the search query that match, a number of times that the object has been bookmarked, a number of times that the object has been rated, or any combination of these.

39. The system of claim 35, wherein the a relevance score for a tag and an object is based on a position of the tag within the object, the frequency of the tag within the object, a density of the tag within the object, or any combination of these.

40. The system of claim 35, wherein the weighting of a relevance score between a tag and an object is based on a level of confidence assigned to a user who associated the tag with the object.

41. The system of claim 35, wherein relevance scores are determined using a statistical classification or rank regression algorithm, a clustering analysis algorithm, or a morphological analysis algorithm.

42. The system of claim 41, wherein the statistical classification or rank regression algorithm comprises any one of logistic regression, support vector machines, classification or regression tree, and boosted tree ensembles.

43. The system of claim 36, further comprising a search engine coupled to the object index, wherein the search engine is programmed to receive search queries containing terms that correspond to tags and to return an organized results list based on relevance scores for tag-object pairs.

44. The system of claim 43, further comprising a user database coupled to the search engine, the user database containing information related to search queries.

45. The system of claim 44, wherein the information related to search queries comprises links followed by a user, tags associated with objects, objects that were blocked by a user, bookmarks, or any combination of these.

46. The system of claim 35, wherein the objects comprise hyperlinks to Web pages or groups of hyperlinks to Web pages.

47. The system of claim 35, wherein the objects comprise hyperlinks or groups of hyperlinks to text, images, photographs, tags, groups of tags, subject areas, concepts, user profiles, answers, audio files, video files, software, or any combination of these.

48. The system of claim 35, further comprising a means for associating a tag with an object.