System and method for focused re-crawling of web sites

• US 7,882,099 B2
• Filed: 03/25/2008
• Issued: 02/01/2011
• Est. Priority Date: 12/21/2005
• Status: Expired due to Fees

• Current Assignee: ServiceNow Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Agrawal, Neeraj, Balakrishnan, Sreeram Viswanath, Joshi, Sachindra
• Primary Examiner(s): FILIPCZYK, MARCIN R
• Application Number:
  US12/054,482
• Publication Number:
  US 20080168041A1
• Time in Patent Office: 1,043 Days
• Field of Search: 707/708, 707/709, 707/726, 707/737, 707/999.001, 707/999.003, 707/999.005, 707/999.006, 707/999.007, 707/999.101, 707/999.102, 709/217
• US Class Current: 707/709
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

A method (100) of crawling the Web (620) is disclosed. The method (100) crawls (120) Web pages on the Web starting from a given (110) set of seed Universal Resource Locators (URLs). Crawled Web pages are partitioned (140) into sets of relevant and irrelevant pages. A set of exclusion and/or inclusion patterns are discovered (150) from the sets of relevant and irrelevant pages, and subsequent crawling of the Web is restricted through the set of exclusion and/or inclusion patterns.

Claims

1. A method of crawling the Web, said method comprising:

crawling Web pages on the Web starting from a given set of seed Universal Resource Locators (URLs);

partitioning crawled Web pages into sets of relevant and irrelevant Web pages;

discovering from said sets of relevant and irrelevant Web pages;

a set of exclusion patterns based on determining that all Web pages present in said irrelevant set of Web pages satisfy at least one exclusion pattern of said set of exclusion patterns, and no Web pages present in said relevant set of Web pages satisfy any pattern of said set of exclusion patterns; and

a set of inclusion patterns based on determining that all Web pages present in said relevant set of Web pages satisfy at least one inclusion pattern of said set of inclusion patterns, and no Web pages present in said irrelevant set of Web pages satisfy any pattern of said set of inclusion patterns;

re-partitioning irrelevant Web pages containing a link to at least one relevant page as relevant;

forming a first set containing words appearing in URLs from relevant Web pages and a second set containing words appearing in URLs from irrelevant Web pages; and

restricting subsequent crawling of the Web pages through said set of exclusion and inclusion patterns.

2. A method according to claim 1, wherein said discovering comprises determining a set of exclusion patterns such that all words from said second set satisfy at least one of said exclusion patterns and no words from said first set satisfy any of said exclusion patterns.

3. A method according to claim 1, wherein said discovering comprises determining a set of inclusion patterns such that all words from said first set satisfy at least one of said exclusion patterns and no words from said second set satisfy any of said exclusion patterns.

4. A method of crawling a network, said method comprising:

crawling network pages on the network starting from a given set of seed Universal Resource Locators (URLs);

partitioning crawled network pages into sets of relevant and irrelevant network pages;

discovering from said sets of relevant and irrelevant network pages;

a set of exclusion patterns based on determining that all network pages present in said irrelevant set of network pages satisfy at least one exclusion pattern of said set of exclusion patterns, and no network pages present in said relevant set of network pages satisfy any pattern of said set of exclusion patterns; and

a set of inclusion patterns based on determining that all network pages present in said relevant set of network pages satisfy at least one inclusion pattern of said set of inclusion patterns, and no network pages present in said irrelevant set of network pages satisfy any pattern of said set of inclusion patterns;

re-partitioning irrelevant network pages containing a link to at least one relevant network page as relevant;

forming a first set containing words appearing in URLs from relevant network pages and a second set containing words appearing in URLs from irrelevant network pages; and

restricting subsequent crawling of the network pages through said set of exclusion and inclusion patterns.

5. A method according to claim 4, wherein said discovering comprises determining a set of exclusion patterns such that all words from said second set satisfy at least one of said exclusion patterns and no words from said first set satisfy any of said exclusion patterns.

6. A method according to claim 4, wherein said discovering comprises determining a set of inclusion patterns such that all words from said first set satisfy at least one of said exclusion patterns and no words from said second set satisfy any of said exclusion patterns.

7. A method according to claim 4, further comprising finding a minimum Steiner tree.

8. A computer program product stored on an electronic storage device useable with a computerized device tangibly embodying a program of instructions executable by said computerized device to perform a method comprising:

crawling Web pages on the Web starting from a given set of seed Universal Resource Locators (URLs);

partitioning crawled Web pages into sets of relevant and irrelevant pages;

discovering from said sets of relevant and irrelevant Web pages;

a set of exclusion patterns based on determining that all Web pages present in said irrelevant set of Web pages satisfy at least one exclusion pattern of said set of exclusion patterns, and no Web pages present in said relevant set of Web pages satisfy any pattern of said set of exclusion patterns; and

a set of inclusion patterns based on determining that all Web pages present in said relevant set of Web pages satisfy at least one inclusion pattern of said set of inclusion patterns, and no Web pages present in said irrelevant set of Web pages satisfy any pattern of said set of inclusion patterns;

re-partitioning irrelevant Web pages containing a link to at least one relevant Web page as relevant;

forming a first set containing words appearing in URLs from relevant Web pages and a second set containing words appearing in URLs from irrelevant Web pages; and

restricting subsequent crawling of the Web pages through said set of exclusion and inclusion patterns.