Systems and methods for removing duplicate search engine results

• US 7,185,088 B1
• Filed: 03/31/2003
• Issued: 02/27/2007
• Est. Priority Date: 03/31/2003
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Joy, Navin Martin, Salas, Sally
• Primary Examiner(s): Vu; Viet D.
• Assistant Examiner(s): Hu; Jinsong
• Application Number:
  US10/403,621
• Time in Patent Office: 1,429 Days
• Field of Search: 709/224, 707/5, 707/6, 707/200
• US Class Current: 709/224
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99936 Pattern matching access

Abstract

The present invention is directed toward efficiently locating desired information and, more specifically, to providing a system and method for removing unnecessary multiple references to a common resource such as redundant listed Uniform Resource Locators (URLs) that reference the same display URLs (and thus the same Web page) as another listed URL. Consequently, in circumstances where only a smaller, finite number of listed results are immediately used (such as displaying only the twenty most relevant results on the first page presented to a search engine end-user), the finite number of listed results may corresponds to a greater number of unique display URLs than would otherwise occur absent this form of filtering.

Claims

1. A method for removing duplicate URLs from a plurality of listed URLs, said method comprising:

determining a display URL for each of a plurality of listed URLs; and

determining if any subset comprising at least two listed URLs from among the plurality of listed URLs have the same display URL and, if so, eliminating at least one listed URL from among the subset of listed URLs having the same display URL, wherein specific listed URLs from among the subset of listed URLs having the same display URL are selected for elimination based on a selection rule, wherein the selection rule for the elimination of specific listed URLs from among the subset of listed URLs having the same display URL is for lesser paying listed URLs to be eliminated.

2. The method of claim 1 wherein the plurality of listed URLs are the results of a search engine query.

3. The method of claim 1 wherein the plurality of listed URLs are derived from a compilation of search engine query results.

4. The method of claim 1 wherein the method operates recursively to eliminate all but one listed URL for each unique display URL determined by the method for the plurality of listed URLs.

5. The method of claim 1 wherein the method operates recursively to eliminate all but one listed URL for each unique display URL determined by the method for the plurality of listed URLs, except that any listed URL which is immune from elimination is not eliminated.

6. The method of claim 1 wherein the selection rule for the elimination of specific listed URLs from among the subset of listed URLs having the same display URL is for a lesser relevant listed URL to be eliminated.

7. The method of claim 1 wherein, if any lesser paying listed URL has a higher relevance than the listed URL that is not being eliminated, then the content from the lesser paying listed URL with the highest relevance is swapped with the content for the listed URL that is not being eliminated.

8. The method of claim 1 wherein the selection rule comprises a sub-rule that, if none of the listed URLs among the subset of listed URLs having the same display URL are paying URLs, the lesser relevant URLs from among the subset of listed URLs are eliminated.

9. The method of claim 1 wherein the selection rule comprises a sub-rule that, if at least one listed URL from among the subset of listed URLs having the same display URL is a paying URL and the other listed URLs are not paying URLs, the listed URLs that are not paying URLs are eliminated.

10. The method of claim 9 wherein, if any listed URL that is not a paying URL has a higher relevance than any listed URL that is a paying URL, then the content from a listed URL that is not a paying URL having the highest relevance is swapped with the content for a listed URL that is a paying URL having a lower relevance.

11. The method of claim 9 wherein the method operates recursively, swapping content between listed URLs, until all listed URLs that are paying URLs have a higher relevance than all listed URLs that are not paying URLs.

12. The method of claim 9 further comprising the element of, from among the paying URLs, if none of the paying URLs are immune from elimination, then the lesser relevant paying URLs are eliminated.

13. The method of claim 12 wherein, if any paying URL that is not immune from elimination has a higher relevance than any paying URL that is immune from elimination, then the content from a paying URL that is not immune from elimination having the highest relevance is swapped with the content for a paying URL immune from elimination having a lower relevance.

14. The method of claim 9 further comprising the element of, from among the paying URLs, if at least one paying URL is immune from elimination, then all paying URLs that are not immune from elimination are eliminated.

15. The method of claim 14 wherein, if any lesser paying listed URL have a higher relevance than the listed URL that is not being eliminated, then the content from the lesser paying listed URL with the highest relevance is swapped with the content for the listed URL that is not being eliminated.

16. The method of claim 1 wherein the selection rule comprises a sub-rule that, if the subset of listed URLs is comprised of at least two listed URLs having the same display URL, at least one of which is immune from elimination, then the listed URLs that are not immune from elimination are eliminated.

17. The method of claim 1 wherein the selection rule comprises a sub-rule that, if the subset of listed URLs is comprised of at least two listed URLs having the same display URL, at least one of which is immune from elimination, then the listed URLs that are immune from elimination are not eliminated.

18. A duplicate URL removal system comprising:

means for determining a display URL for each of a plurality of listed URLs;

means determining if any subset comprising at least two listed URLs from among the plurality of listed URLs have the same display URL; and

means for eliminating at least one listed URL from among the subset of listed URLs having the same display URL, wherein specific listed URLs from among the subset of listed URLs having the same display URL are selected for elimination based on a selection rule, wherein the selection rule for the elimination of specific listed URLs from among the subset of listed URLs having the same display URL is for lesser paying listed URLs to be eliminated.

19. The system of claim 18 further comprising means for eliminating all but one listed URL for each unique display URL.

20. The system of claim 18 further comprising means for eliminating all but one listed URL for each unique display URL except for listed URLs which are immune from elimination.

21. The method of claim 18 wherein the selection rule for the elimination of specific listed URLs from among the subset of listed URLs having the same display URL is for a lesser relevant listed URL to be eliminated.

22. The method of claim 18 wherein, if any lesser paying listed URL has a higher relevance than the listed URL that is not being eliminated, then the content from the lesser paying listed URL with the highest relevance is swapped with the content for the listed URL that is not being eliminated.

23. A computer-readable medium bearing computer-readable instructions for:

determining a display URL for each of a plurality of listed URLs; and

determining if any subset comprising at least two listed URLs from among the plurality of listed URLs have the same display URL and, if so, eliminates at least one listed URL from among the subset of listed URLs having the same display URL, wherein specific listed URLs from among the subset of listed URLs having the same display URL are selected for elimination based on a selection rule, wherein the selection rule for the elimination of specific listed URLs from among the subset of listed URLs having the same display URL is for lesser paying listed URLs to be eliminated.

24. The computer-readable medium of claim 23 wherein the plurality of listed URLs are the results of a search engine query.

25. The computer-readable medium of claim 23 wherein the plurality of listed URLs are derived from a compilation of search engine query results.

26. The computer-readable medium of claim 23 wherein the computer-readable medium operates recursively to eliminate all but one listed URL for each unique display URL determined by the computer-readable medium for the plurality of listed URLs.

27. The computer-readable medium of claim 23 wherein the computer-readable medium operates recursively to eliminate all but one listed URL for each unique display URL determined by the computer-readable medium for the plurality of listed URLs, except that any listed URL which is immune from elimination is not eliminated.

28. The computer-readable medium of claim 23 wherein the selection rule for the elimination of specific listed URLs from among the subset of listed URLs having the same display URL is for a lesser relevant listed URL to be eliminated.

29. The computer-readable medium of claim 23 wherein, if any lesser paying listed URL has a higher relevance than the listed URL that is not being eliminated, then the content from the lesser paying listed URL with the highest relevance is swapped with the content for the listed URL that is not being eliminated.

30. The computer-readable medium of claim 23 wherein the selection rule is comprised of at least two sub-rules.

31. The computer-readable medium of claim 23 wherein the selection rule comprises a sub-rule that, if none of the listed URLs among the subset of listed URLs having the same display URL are paying URLs, the lesser relevant URLs from among the subset of listed URLs are eliminated.

32. The computer-readable medium of claim 23 wherein the selection rule comprises a sub-rule that, if at least one listed URL from among the subset of listed URLs having the same display URL is a paying URL and the other listed URLs are not paying URLs, the listed URLs that are not paying URLs are eliminated.

33. The computer-readable medium of claim 32 wherein, if any listed URL that is not a paying URL has a higher relevance than any listed URL that is a paying URL, then the content from a listed URL that is not a paying URL having the highest relevance is swapped with the content for a listed URL that is a paying URL having a lower relevance.

34. The computer-readable medium of claim 33 wherein the instructions operate recursively, swapping content between listed URLs, until all listed URLs that are paying URLs have a higher relevance than all listed URLs that are not paying URLs.

35. The computer-readable medium of claim 32 further comprising the element of, from among the paying URLs, if none of the paying URLs are immune from elimination, then the lesser relevant paying URLs are eliminated.

36. The computer-readable medium of claim 35 wherein, if any paying URL that is not immune from elimination has a higher relevance than any paying URL that is immune from elimination, then the content from a paying URL that is not immune from elimination having the highest relevance is swapped with the content for a paying URL immune from elimination having a lower relevance.

37. The computer-readable medium of claim 36 wherein, if any lesser paying listed URL have a higher relevance than the listed URL that is not being eliminated, then the content from the lesser paying listed URL with the highest relevance is swapped with the content for the listed URL that is not being eliminated.

38. The computer-readable medium of claim 32 further comprising the element of, from among the paying URLs, if at least one paying URL is immune from elimination, then all paying URLs that are not immune from elimination are eliminated.

39. The computer-readable medium of claim 23 wherein the selection rule comprises a sub-rule that, if the subset of listed URLs is comprised of at least two listed URLs having the same display URL, at least one of which is immune from elimination, then the listed URLs that are not immune from elimination are eliminated.

40. The computer-readable medium of claim 23 wherein the selection rule comprises a sub-rule that, if the subset of listed URLs is comprised of at least two listed URLs having the same display URL, at least one of which is immune from elimination, then the listed URLs that are immune from elimination are not eliminated.