Systems for and methods of finding relevant documents by analyzing tags

• US 9,715,542 B2
• Filed: 08/02/2006
• Issued: 07/25/2017
• Est. Priority Date: 08/03/2005
• Status: Active Grant

• Current Assignee: Pinterest, Inc.
• Original Assignee: Search Engine Technologies, LLC
• Inventors: Lu, Yunshan, Tanne, Michael
• Primary Examiner(s): LU, KUEN S
• Application Number:
  US11/498,448
• Publication Number:
  US 20070185858A1
• Time in Patent Office: 4,010 Days
• Field of Search: 707 3, 707 4, 707739
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A method of determining relevancies of objects to a search query includes associating multiple tags with multiple objects, recording bookmarks to the multiple objects, or both, and determining a relevance score for each of the multiple objects and a search query. One embodiment of the method combines full-text relevance algorithms with tag relevance algorithms. Other embodiments include statistical relevance algorithms such as statistical classification or rank regression algorithms. When a user executes a search query, a results list containing the objects is returned, with the objects organized based on the relevance scores. The objects are organized by, for example, listing those with the highest relevance scores first or by marking them with an indication of their relevance. Preferably, relevance scores for a tag-object pair are based on a number of times a term in the tag has been associated with the object, a number of tags associated with the object, a number of times that the tag has been associated with the multiple objects, a number of tag-object pairs that contain a term in the tag, a number of tag-object pairs that contain a reference to the object, or any combination of these.

Claims

1. A computerized method of determining relevancies of multiple objects to a search query, comprising:

associating one or more of the multiple objects with user-entered tags as a result of user input, thereby defining one or more corresponding tag-object pairs, wherein each tag comprises one or more tag terms, each tag term comprising text, and wherein each association of an object with a tag comprises a tag object pair;

associating an object with each tag term from the one or more tag terms, thereby defining one or more corresponding tag term object pairs, wherein at least one of the one or more tags comprises a string of multiple terms entered by a user;

determining for each tag term object pair a tag term score indicating a degree of relevance between the tag term and the object, wherein a tag term score for an object X and a term A is a function of a combination comprising a total number of different terms in a tag database storing the tags, a total number of tags present in the tag database, a frequency with which the term A is present in the tag database, a number of different terms with which the object X has been tagged, a total number of tags associated with the object X, and a number of different objects that have been tagged with the term A;

determining for one or more objects a term relevance score comprising combining the tag term scores from the tag term object pairs for each tag associated with each object; and

determining a relevance score for each of the multiple objects for the search query, wherein the relevance score is influenced by tags associated with objects as a result of user input.

2. The method of claim 1, wherein combining the tag term scores comprises summing the tag term scores.

3. The method of claim 1, wherein combining the tag term scores comprises weighing each tag term score with a weight and summing the weighted tag term scores.

4. The method of claim 1, wherein a relevance score for a tag object pair is determined from a number of times a tag term in the tag has been associated with the object, a number of tags associated with the object, a number of times that the tag has been associated with the multiple objects, or any combination of these.

5. The method of claim 1, wherein a relevance score for a tag object pair is determined from a number of tag object pairs that contain a tag term in the tag, a number of tag object pairs that contain a reference to the object, or both.

6. The method of claim 1, further comprising presenting a results list in response to the search query, wherein:

the associating one or more of the multiple objects with tags is performed by a first user, wherein the tag contains one or more tag terms;

the search query contains the one or more tag terms in the tag and is performed by a second user;

the results list is presented to the second user; and

the relevance scores are influenced by a level of confidence placed in the first user or in groups to which the first user belongs.

7. The method of claim 6, wherein a relevance score for an object and the search query is determined from relevance scores for each term of the search query present in the object or associated with the object.

8. The method of claim 6, wherein a relevance score for a tag object pair from the multiple tag object pairs is determined from a number of tags that the first user has associated with any of the multiple objects, a number of objects that the first and second users have associated with tags, or both.

9. The method of claim 6, wherein the level of confidence is a confidence rating of the first user.

10. The method of claim 9, wherein the confidence rating is determined from a rating for tags that the first user has associated with objects, a similarity metric between search activities for the first and second users, a connection metric between the first and second users, or any combination of these.

11. The method of claim 6, wherein the level of confidence is based on how closely the first user'"'"'s tagging behavior matches that of other users.

12. The method of claim 6, further comprising marking at least one of the multiple objects in the results list with a graphic element.

13. The method of Claim 6, wherein the step of presenting the results list to the user includes displaying to the user the tags associated with the one or more of the multiple objects in the results list.

14. The method of claim 1, wherein associating one or more of the multiple objects with tags comprises entering the tag in an area presented to a user, rating the tag, blocking a link to the object, selecting the tag, or selecting the object.

15. The method of claim 1, wherein associating one or more of the multiple objects with tags comprises analyzing input given by a user conducting a search.

16. The method of claim 1, wherein the multiple objects comprise hyperlinks to Web pages or groups of hyperlinks to Web pages.

17. The method of claim 1, wherein the multiple objects comprise hyperlinks or groups of hyperlinks to text, images, photographs, tags, groups of tags, subject areas, concepts, user profiles, answers, audio files, video files, software, or any combination of these.

18. The method of claim 1, wherein a tag crawler associates at least one of the multiple tags with at least one of the multiple objects.

19. A computerized method of organizing multiple objects for display in a results list, comprising:

correlating query terms in a database search query with user-entered tags associated with multiple objects as a result of user input, thereby defining one or more corresponding tag-object pairs, wherein each tag comprises one or more tag terms, each tag term comprising text, and wherein each association of an object with a tag comprises a tag object pair;

associating each tag term from the one or more tag terms with an object, thereby defining one or more corresponding tag term object pairs;

determining for each term object pair a tag term score indicating a degree of relevance between the term and the object, wherein at least one of the tags comprises a string of multiple terms entered by a user, wherein a tag term score for an object X and a term A is a function of a combination comprising a total number of different terms in a tag database storing the tags, a total number of tags present in the tag database, a frequency with which the term A is present in the tag database, a number of different terms with which the object X has been tagged, a total number of tags associated with the object X, and a number of different objects that have been tagged with the term A;

determining for each tag object pair a relevance metric comprising combining the tag term scores from the term object pairs for each tag term in the tag; and

returning the results list containing the multiple objects organized based on the correlations, wherein each correlation corresponds to a relevance metric.

20. The method of claim 19, further comprising:

executing the search query, thereby generating a first list of objects; and

organizing the multiple objects based on the correlations.

21. The method of claim 19, further comprising applying a statistical classification or rank regression algorithm to the multiple objects to determine relevance metrics between the multiple objects and terms in the search query.

22. The method of claim 21, wherein the statistical classification or rank regression algorithm is any one of logistic regression, support vector machines, classification or regression tree, or boosted tree ensembles.

23. A system for returning a search results list in response to a search query, the system comprising:

a tag database for storing user-entered tags associated with objects, wherein one or more objects is associated with tags as a result of user input and each tag comprises one or more tag terms, each tag term comprising text, and wherein each association of an object with a tag comprises a tag object pair, and each term of a tag comprises a tag term pair;

a storage device containing computer-executable instructions comprising a tag analyzer coupled to the tag database, wherein the tag analyzer is programmed to use the associations of tags with objects to compute a relevance score for each of the objects for the search query, wherein the relevance score for a tag object pair is computed by determining a tag term score indicating a degree of relevance between the term and the object, for user-entered strings of one and more terms in a tag, and combining the tag term scores from the term object pairs for each tag term in the tag, wherein a tag term score for an object X and a term A is a function of a combination comprising a total number of different terms in the tag database storing the tags, a total number of tags present in the tag database, a frequency with which the term A is present in the tag database, a number of different terms with which the object X has been tagged, a total number of tags associated with the object X, and a number of different objects that have been tagged with the term A; and

a hardware-implemented search engine for receiving the search query and for presenting the results list in response to the search query, the results list comprising references to one or more of the objects, wherein the order of the multiple objects in the results list is influenced by the relevance scores.

24. The system of claim 23, further comprising an object index for storing relevance scores between tags associated with objects and objects, between bookmarks to objects, or both.

25. The system of claim 24, further comprising a search engine coupled to the object index, wherein the search engine is programmed to receive search queries containing terms that correspond to tag terms and to return an organized results list based on relevance scores for tag object pairs.

26. The system of claim 25, further comprising a user database coupled to the search engine, the user database containing information related to search queries.

27. The system of claim 26, wherein the information related to search queries comprises links followed by a user, tags associated with objects, objects that were blocked by a user, bookmarks, or any combination of these.

28. The system of claim 23, wherein the relevance score is determined by summing weighted relevance scores for tag terms that form a tag and objects.

29. The system of claim 23, wherein the relevance score between a search query containing terms and an object is determined from a number of tags that contain tag terms in the search query, a number of times that a tag term included in the search query is included in the tag database, a number of tags that have been associated with the object, a number of tag terms in the tag and in the search query that match, a number of times that the object has been bookmarked, a number of times that the object has been rated, or any combination of these.

30. The system of claim 23, wherein the relevance score for a tag and an object is based on a position of one or more tag terms within the object, the frequency of the tag terms within the object, a density of the tag terms within the object, or any combination of these.

31. The system of claim 23, wherein weighting of the relevance score between a tag and an object is based on a level of confidence assigned to a user who associated the tag with the object.

32. The system of claim 23, wherein relevance scores are determined using a statistical classification or rank regression algorithm, a clustering analysis algorithm, or a morphological analysis algorithm.

33. The system of claim 32, wherein the statistical classification or rank regression algorithm comprises any one of logistic regression, support vector machines, classification or regression tree, and boosted tree ensembles.

34. The system of claim 23, wherein the objects comprise hyperlinks to Web pages or groups of hyperlinks to Web pages.

35. The system of claim 23, wherein the objects comprise hyperlinks or groups of hyperlinks to text, images, photographs, tags, groups of tags, subject areas, concepts, user profiles, answers, audio files, video files, software, or any combination of these.

36. The system of claim 23, further comprising a means for associating a tag with an object.

37. A system comprising a computer-readable storage medium storing computer-executable instructions that, when executed by a computer, cause the computer to perform a method of determining relevancies of multiple objects to a search query, the method comprising:

associating one or more of the multiple objects with user-entered tags as a result of user input, thereby defining one or more corresponding tag-object pairs, wherein each tag comprises one or more tag terms, each tag term comprising text, and wherein each association of an object with a tag comprises a tag object pair;

associating an object with each tag term from the one or more tag terms, thereby defining one or more corresponding tag term object pairs, wherein at least one of the one or more tags comprises a string of multiple terms entered by a user;

determining for each tag term object pair a tag term score indicating a degree of relevance between the tag term and the object, wherein a tag term score for an object X and a term A is a function of a combination comprising a total number of different terms in a tag database storing the tags, a total number of tags present in the tag database, a frequency with which the term A is present in the tag database, a number of different terms with which the object X has been tagged, a total number of tags associated with the object X, and a number of different objects that have been tagged with the term A;

determining for one or more objects a term relevance score comprising combining the tag term scores from the tag term object pairs for each tag associated with each object; and

determining a relevance score for each of the multiple objects for the search query, wherein the relevance score is influenced by tags associated with objects as a result of user input.

38. A system comprising a computer-readable storage medium storing computer-executable instructions that, when executed by a computer, cause the computer to perform a method of organizing multiple objects for display in a results list, the method comprising:

correlating query terms in a database search query with user-entered tags associated with multiple objects as a result of user input, thereby defining one or more corresponding tag-object pairs, wherein each tag comprises one or more tag terms, each tag term comprising text, and wherein each association of an object with a tag comprises a tag object pair;

associating each tag term from the one or more tag terms with an object, thereby defining one or more corresponding tag term object pairs;

determining for each term object pair a tag term score indicating a degree of relevance between the term and the object, wherein at least one of the tags comprises a string of multiple terms entered by a user, wherein a tag term score for an object X and a term A is a function of a combination comprising a total number of different terms in a tag database storing the tags, a total number of tags present in the tag database, a frequency with which the term A is present in the tag database, a number of different terms with which the object X has been tagged, a total number of tags associated with the object X, and a number of different objects that have been tagged with the term A;

determining for each tag object pair a relevance metric comprising combining the tag term scores from the term object pairs for each tag term in the tag; and

returning the results list containing the multiple objects organized based on the correlations, wherein each correlation corresponds to a relevance metric.