SYSTEM AND METHOD FOR FOCUSED RE-CRAWLING OF WEB SITES

• US 20080168041A1
• Filed: 03/25/2008
• Published: 07/10/2008
• Est. Priority Date: 12/21/2005
• Status: Active Grant

• Current Assignee: ServiceNow Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Joshi, Sachindra, Agrawal, Neeraj, Balakrishnan, Sreeram Viswanath
• Granted Patent: US 7,882,099 B2
• Time in Patent Office: Days
• US Class Current: 707/3
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

A method (100) of crawling the Web (620) is disclosed. The method (100) crawls (120) Web pages on the Web starting from a given (110) set of seed Universal Resource Locators (URLs). Crawled Web pages are partitioned (140) into sets of relevant and irrelevant pages. A set of exclusion and/or inclusion patterns are discovered (150) from the sets of relevant and irrelevant pages, and subsequent crawling of the Web is restricted through the set of exclusion and/or inclusion patterns.

Claims

1. A method of crawling the Web, said method comprising:

crawling Web pages on the Web starting from a given set of seed Universal Resource Locators (URLs);

partitioning crawled Web pages into sets of relevant and irrelevant pages;

discovering from said sets of relevant and irrelevant pages a set of exclusion and inclusion patterns; and

restricting subsequent crawling of the Web through said set of exclusion and inclusion patterns.

2. A method according to claim 1, wherein said discovering comprises re-partitioning irrelevant pages containing a link to at least one relevant page as relevant.

3. A method according to claim 1, wherein said discovering comprises forming a first set containing words appearing in URLs from relevant pages.

4. A method according to claim 1, wherein said discovering comprises forming a second set containing words appearing in URLs from irrelevant pages;

and

5. A method according to claim 1, wherein said discovering comprises determining a set of exclusion patterns such that all words from said second set satisfy at least one of said exclusion patterns and no words from said first set satisfy any of said exclusion patterns.

6. A method according to claim 1, wherein said discovering comprises re-partitioning irrelevant pages containing a link to at least one relevant page as relevant.

7. A method according to claim 1, wherein said discovering comprises forming a first set containing words appearing in URLs from relevant pages.

8. A method according to claim 1, wherein said discovering comprises forming a second set containing words appearing in URLs from irrelevant pages.

9. A method according to claim 1, wherein said discovering comprises determining a set of inclusion patterns such that all words from said first set satisfy at least one of said exclusion patterns and no words from said second set satisfy any of said exclusion patterns.

10. A method of crawling a network, said method comprising:

crawling network pages on the network starting from a given set of seed Universal Resource Locators (URLs);

partitioning crawled network pages into sets of relevant and irrelevant pages;

discovering from said sets of relevant and irrelevant pages a set of exclusion and inclusion patterns; and

restricting subsequent crawling of the network through said set of exclusion and inclusion patterns.

11. A method according to claim 10, wherein said discovering comprises re-partitioning irrelevant pages containing a link to at least one relevant page as relevant.

12. A method according to claim 10, wherein said discovering comprises forming a first set containing words appearing in URLs from relevant pages.

13. A method according to claim 10, wherein said discovering comprises forming a second set containing words appearing in URLs from irrelevant pages.

14. A method according to claim 10, wherein said discovering comprises determining a set of exclusion patterns such that all words from said second set satisfy at least one of said exclusion patterns and no words from said first set satisfy any of said exclusion patterns.

15. A method according to claim 10, wherein said discovering comprises re-partitioning irrelevant pages containing a link to at least one relevant page as relevant.

16. A method according to claim 10, wherein said discovering comprises forming a first set containing words appearing in URLs from relevant pages.

17. A method according to claim 10, wherein said discovering comprises forming a second set containing words appearing in URLs from irrelevant pages.

18. A method according to claim 10, wherein said discovering comprises determining a set of inclusion patterns such that all words from said first set satisfy at least one of said exclusion patterns and no words from said second set satisfy any of said exclusion patterns.

19. A method according to claim 10, further comprising finding a minimum Steiner tree before performing said re-partitioning.

20. A computer program product stored on an electronic storage device useable with a computerized device tangibly embodying a program of instructions executable by said computerized device to perform a method comprising:

crawling Web pages on the Web starting from a given set of seed Universal Resource Locators (URLs);

partitioning crawled Web pages into sets of relevant and irrelevant pages;

discovering from said sets of relevant and irrelevant pages a set of exclusion and inclusion patterns; and

restricting subsequent crawling of the Web through said set of exclusion and inclusion patterns.