System and method for prioritizing websites during a webcrawling process

• US 7,966,337 B2
• Filed: 06/23/2008
• Issued: 06/21/2011
• Est. Priority Date: 03/29/2006
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Nguyen, Linda Anh Linh, Meredith, Daniel Norin, Blackman, David L., Dill, Stephen, Ching, Michael, Marcus, Adam, Gonzalez, Ivan Eduardo
• Primary Examiner(s): Vital; Pierre M
• Assistant Examiner(s): Rostami; Mohammad S
• Application Number:
  US12/143,885
• Publication Number:
  US 20080256046A1
• Time in Patent Office: 1,093 Days
• Field of Search: 707/999.002, 707/999.003, 707/999.004, 707/999.005, 707/999.006, 707/999.007, 707/709, 707/714, 707/715, 707/721, 707/723, 707/769, 707/707, 707/731, 707/732, 707/752
• US Class Current: 707/752
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99936 Pattern matching access

Abstract

A system and method for prioritizing a fetch order of web pages. The method comprises extracting by a web crawler a set of candidate web pages to be crawled. Each web page in the set of candidate web pages is associated with a website in a computer network. A determination is made to determine if a first website score for the website is in a website score database. The first website score is associated with web pages in the set of candidate web pages if the first website score exists in the website score database. The set of candidate web pages is prioritized with respect to an associated website score for each web page in the candidate set of web pages. Content is retrieved from the set of candidate web. Hyperlinks are extracted from the content. The hyperlinks are stored in a memory unit.

Claims

1. A prioritization method, comprising:

extracting, by a web crawler in a computing system, a set of candidate web pages to be crawled, wherein said computing system comprises a memory unit, and wherein said memory unit comprises said web crawler, said set of candidate web pages, an online analysis software application, an offline analysis software application, a web page score database, and a website score database;

simultaneously executing, by a computer processor of said computing system, said online analysis software application and said offline analysis software application in order to simultaneously perform an online analysis and an offline analysis;

associating, by said online analysis software application, each web page in said set of candidate web pages with a website in a computer network;

verifying, by said online analysis software application, that said set of candidate web pages comprise data for analysis;

determining online, by said online analysis software application that a first website score for said website is not available in said website score database;

requesting, by said web crawler, a first sample set of web pages from said website, wherein said first sample set of web pages does not include a total set of web pages from said website;

first analyzing offline, by said offline analysis software application, each sample web page of said first sample set of web pages with a plurality of offline heuristics;

generating, by said offline analysis software application, a first group of web page scores for each said sample web page of said first sample set of web pages based on results of said first analyzing offline;

storing, each said first group of web page scores in said web page score database;

determining, by said offline analysis software application, that a number of Web pages in said first sample set of web pages has reached a predetermined threshold;

generating, by said offline analysis software application in response to said determining, that said number of Web pages in said first sample set of web pages has reached said predetermined threshold, a first final web page score for each said web page of said first sample set of web pages, wherein each said first final web page score is generated by combining each web page score within each said first group of web page scores;

storing, each said first final web page score in said web page score database;

generating, by said offline analysis software application in response to said determining online, a first website score for said website, wherein said first website score is generated by combining said first final web page scores for said first sample set of web pages;

storing, said first website score in said website score database;

associating, by said online analysis software application, said first website score for said website with associated web pages in said set of candidate web pages;

prioritizing, said set of candidate web pages with respect to a first associated website score for each web page in said candidate set of web pages;

retrieving, by said web crawler, first content from said set of candidate web pages using said prioritizing;

extracting, by said online analysis software application, first hyperlinks from said first content;

storing said first hyperlinks in said memory unit.

2. The method of claim 1, wherein said offline analysis software application comprises a plurality of data mining software tools.

3. The method of claim 1, wherein said analyzing offline comprises performing a search for specified expressions within said content from each said sample web page of said first sample set of web pages, wherein each specified expression from said specified expressions is selected from the group consisting of a phrase and a word, and wherein said specified expressions are divided into a first group of expressions that are of positive interest to a user of said web crawler and a second group of expressions that are of negative interest to said user.

4. The method of claim 3, wherein said analyzing offline further comprises determining if each said sample web page of said first sample set of web pages comprises a hyperlink to an alternative web page comprising an expression from said first group or said second group.

5. The method of claim 1, wherein said requesting comprises a dynamic determination of a sample size of said first sample set of web pages.

6. The method of claim 1, further comprising:

excluding, by said online analysis software application, a group of web pages from said first sample set of web pages based upon a plurality of heuristics.

18. The method of claim 1, further comprising:

determining by said online analysis software application that a specified time period has elapsed since said generating said first website score.

7. A computing system comprising a computer processor coupled to a computer-readable memory unit, said memory unit comprising a web crawler, a set of candidate web pages, an online analysis software application, an offline analysis software application, a website score database, a web page score database, and instructions that when executed by the computer processor implement a prioritization method, said method comprising:

extracting, by said web crawler, said set of candidate web pages to be crawled;

simultaneously executing, by said computer processor of said computing system, said online analysis software application and said offline analysis software application in order to simultaneously perform an online analysis and an offline analysis;

associating, by said online analysis software application, each web page in said set of candidate web pages with a website in a computer network;

verifying, by said online analysis software application, that said set of candidate web pages comprise data for analysis;

determining online, by said online analysis software application that a first website score for said website is not available in said website score database;

requesting, by said web crawler, a first sample set of web pages from said website, wherein said first sample set of web pages does not include a total set of web pages from said website;

first analyzing offline, by said offline analysis software application, each sample web page of said first sample set of web pages with a plurality of offline heuristics;

generating, by said offline analysis software application, a first group of web page scores for each said sample web page of said first sample set of web pages based on results of said first analyzing offline;

storing, each said first group of web page scores in said web page score database;

determining, by said offline analysis software application, that a number of Web pages in said first sample set of web pages has reached a predetermined threshold;

generating, by said offline analysis software application in response to said determining, that said number of Web pages in said first sample set of web pages has reached said predetermined threshold, a first final web page score for each said web page of said first sample set of web pages, wherein each said first final web page score is generated by combining each web page score within each said first group of web page scores;

storing, each said first final web page score in said web page score database;

generating, by said offline analysis software application in response to said determining online, a first website score for said website, wherein said first website score is generated by combining said first final web page scores for said first sample set of web pages;

storing, said first website score in said website score database;

associating, by said online analysis software application, said first website score for said website with associated web pages in said set of candidate web pages;

prioritizing, said set of candidate web pages with respect to a first associated website score for each web page in said candidate set of web pages;

retrieving, by said web crawler, first content from said set of candidate web pages using said prioritizing;

extracting, by said online analysis software application, first hyperlinks from said first content;

storing said first hyperlinks in said memory unit.

8. The computing system of claim 7, wherein said offline analysis software application comprises a plurality of data mining software tools.

9. The computing system of claim 7, wherein said analyzing offline comprises performing a search for specified expressions within said content from each said sample web page of said first sample set of web pages, wherein each specified expression from said specified expressions is selected from the group consisting of a phrase and a word, and wherein said specified expressions are divided into a first group of expressions that are of positive interest to a user of said web crawler and a second group of expressions that are of negative interest to said user.

10. The computing system of claim 9, wherein said analyzing offline further comprises determining if each said sample web page of said first sample set of web pages comprises a hyperlink to an alternative web page comprising an expression from said first group or said second group.

11. The computing system of claim 7, wherein said requesting comprises a dynamic determination of a sample size of said first sample set of web pages.

12. The computing system of claim 7, wherein said method further comprises:

excluding, by said online analysis software application, a group of web pages from said first sample set of web pages based upon a plurality of heuristics.

13. A computer program product, comprising a computer readable storage medium including an online analysis software application, an offline analysis software application, a website score database, a web crawler, a set of candidate web pages, a web page score database, and computer readable program code embodied therein, said computer readable program code comprising an algorithm that when executed by a computer processor of a computing system is adapted to implement a prioritization method within said computing system, said method comprising:

extracting, by said web crawler, said set of candidate web pages to be crawled;

simultaneously executing, by said computer processor of said computing system, said online analysis software application and said offline analysis software application in order to simultaneously perform an online analysis and an offline analysis;

associating, by said online analysis software application, each web page in said set of candidate web pages with a website in a computer network;

verifying, by said online analysis software application, that said set of candidate web pages comprise data for analysis;

determining online, by said online analysis software application that a first website score for said website is not available in said website score database;

requesting, by said web crawler, a first sample set of web pages from said website, wherein said first sample set of web pages does not include a total set of web pages from said website;

first analyzing offline, by said offline analysis software application, each sample web page of said first sample set of web pages with a plurality of offline heuristics;

generating, by said offline analysis software application, a first group of web page scores for each said sample web page of said first sample set of web pages based on results of said first analyzing offline;

storing, each said first group of web page scores in said web page score database;

determining, by said offline analysis software application, that a number of Web pages in said first sample set of web pages has reached a predetermined threshold;

generating, by said offline analysis software application in response to said determining, that said number of Web pages in said first sample set of web pages has reached said predetermined threshold, a first final web page score for each said web page of said first sample set of web pages, wherein each said first final web page score is generated by combining each web page score within each said first group of web page scores;

storing, each said first final web page score in said web page score database;

generating, by said offline analysis software application in response to said determining online, a first website score for said website, wherein said first website score is generated by combining said first final web page scores for said first sample set of web pages;

storing, said first website score in said website score database;

associating, by said online analysis software application, said first website score for said website with associated web pages in said set of candidate web pages;

prioritizing, said set of candidate web pages with respect to a first associated website score for each web page in said candidate set of web pages;

retrieving, by said web crawler, first content from said set of candidate web pages using said prioritizing;

extracting, by said online analysis software application, first hyperlinks from said first content;

storing said first hyperlinks in said memory unit.

14. The computer program product of claim 13, wherein said offline analysis software application comprises a plurality of data mining software tools.

15. The computer program product of claim 13, wherein said analyzing offline comprises performing a search for specified expressions within said content from each said sample web page of said first sample set of web pages, wherein each specified expression from said specified expressions is selected from the group consisting of a phrase and a word, and wherein said specified expressions are divided into a first group of expressions that are of positive interest to a user of said web crawler and a second group of expressions that are of negative interest to said user.

16. The computer program product of claim 13, wherein said requesting comprises a dynamic determination of a sample size of said first sample set of web pages.

17. The computer program product of claim 13, wherein said method further comprises:

excluding, by said online analysis software application, a group of web pages from said first sample set of web pages based upon a plurality of heuristics.