SYSTEM AND METHOD FOR PRIORITIZING WEBSITES DURING A WEBCRAWLING PROCESS

• US 20080256046A1
• Filed: 06/23/2008
• Published: 10/16/2008
• Est. Priority Date: 03/29/2006
• Status: Active Grant

• Current Assignee: Adam Marcus, Daniel Norin Meredith, David L. Blackman, Ivan Eduardo Gonzalez, Linda Anh Linh Nguyen, Michael Ching, Stephen Dill
• Original Assignee: Adam Marcus, Daniel Norin Meredith, David L. Blackman, Ivan Eduardo Gonzalez, Linda Anh Linh Nguyen, Michael Ching, Stephen Dill
• Inventors: Nguyen, Linda Anh Linh, Meredith, Daniel Norin, Blackman, David L., Dill, Stephen, Ching, Michael, Marcus, Adam, Gonzalez, Ivan Eduardo
• Granted Patent: US 7,966,337 B2
• Time in Patent Office: Days
• US Class Current: 707/3
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99936 Pattern matching access

Abstract

A system and method for prioritizing a fetch order of web pages. The method comprises extracting by a web crawler a set of candidate web pages to be crawled. Each web page in the set of candidate web pages is associated with a website in a computer network. A determination is made to determine if a first website score for the website is in a website score database. The first website score is associated with web pages in the set of candidate web pages if the first website score exists in the website score database. The set of candidate web pages is prioritized with respect to an associated website score for each web page in the candidate set of web pages. Content is retrieved from the set of candidate web. Hyperlinks are extracted from the content. The hyperlinks are stored in a memory unit.

Claims

1. A prioritization method, comprising:

extracting, by a web crawler in a computing system, a set of candidate web pages to be crawled, wherein said computing system comprises a memory unit, and wherein said memory unit comprises said web crawler, said set of candidate web pages, an online analysis software application, an offline analysis software application, and a website score database;

associating, by said online analysis software application, each web page in said set of candidate web pages with a website in a computer network;

determining online, by said online analysis software application, if a first website score for said website, is in said website score database;

associating, by said online analysis software application, said first website score for said website with associated web pages in said set of candidate web pages, if said first website score exists in said website score database;

prioritizing, said set of candidate web pages with respect to an associated website score for each web page in said candidate set of web pages;

retrieving, by said web crawler, content from said set of candidate web pages using said prioritizing;

extracting, by said online analysis software application, hyperlinks from said content;

storing said hyperlinks in said memory unit.

2. The method of claim 1, wherein said determining online determines that said first website score is not available in said website score database, wherein said memory unit further comprises a web page score database, and wherein said method further comprises:

requesting, by said web crawler, a sample set of web pages from said website, wherein said sample set of web pages does not include a total set of web pages from said website;

analyzing offline, by said offline analysis software application, each sample web page of said sample web pages with a plurality of offline heuristics;

generating, by said offline analysis software application, a group of web page scores for each said sample web page based on results of said analyzing offline;

storing, each said group of web page scores in said web page score database;

generating, by said offline analysis software application, a final web page score for each said web page of said sample set of web pages, wherein each said final web page score is generated by combining each web page score within each said group of web page scores;

storing, each said final web page score in said web page score database;

generating, by said offline analysis software application, a second website score for said website, wherein said second website score is generated by combining said final web page scores for said sample set of web pages, and wherein said first website score is generated using a same technique as used to generate said second website score; and

storing, said second website score in said website score database.

3. The method of claim 2, wherein said offline analysis software application comprises a plurality of data mining software tools.

4. The method of claim 2, wherein said analyzing offline comprises performing a search for specified expressions within said content from each said sample web page, wherein each specified expression from said specified expressions is selected from the group consisting of a phrase and a word, and wherein said specified expressions are divided into a first group of expressions that are of positive interest to a user of said web crawler and a second group of expressions that are of negative interest to said user.

5. The method of claim 4, wherein said analyzing offline further comprises determining if each said sample web page comprises a hyperlink to an alternative web page comprising an expression from said first group or said second group.

6. The method of claim 2, wherein said requesting comprises a dynamic determination of a sample size of said sample set of web pages.

7. The method of claim 2, further comprising:

excluding, by said online analysis software application, a group of web pages from said sample set of web pages based upon a plurality of heuristics.

8. A computing system comprising a processor coupled to a computer-readable memory unit, said memory unit comprising a web crawler, a set of candidate web pages, an online analysis software application, an offline analysis software application, a website score database, and instructions that when executed by the processor implement a prioritization method, said method comprising:

extracting, by said web crawler, said set of candidate web pages to be crawled;

associating, by said online analysis software application, each web page in said set of candidate web pages with a website in a computer network;

determining online, by said online analysis software application, if a first website score for said website, is in said website score database;

associating, by said online analysis software application, said first website score for said website with associated web pages in said set of candidate web pages, if said first website score exists in said website score database;

prioritizing, said set of candidate web pages with respect to an associated website score for each web page in said candidate set of web pages;

retrieving, by said web crawler, content from said set of candidate web pages using said prioritizing;

extracting, by said online analysis software application, hyperlinks from said content;

storing said hyperlinks in said memory unit.

9. The computing system of claim 8, further comprising a web page score database, wherein said determining online determines that said first website score is not available in said website score database, and wherein said method further comprises;

requesting, by said web crawler, a sample set of web pages from said website, wherein said sample set of web pages does not include a total set of web pages from said website;

analyzing offline, by said offline analysis software application, each sample web page of said sample web pages with a plurality of offline heuristics;

generating, by said offline analysis software application, a group of web page scores for each said sample web page based on results of said analyzing offline;

storing, each said group of web page scores in said web page score database;

generating, by said offline analysis software application, a final web page score for each said web page of said sample set of web pages, wherein each said final web page score is generated by combining each web page score within each said group of web page scores;

storing, each said final web page score in said web page score database;

generating, by said offline analysis software application, a second website score for said website, wherein said second website score is generated by combining said final web page scores for said sample set of web pages, and wherein said first website score is generated using a same technique as used to generate said second website score; and

storing, said second website score in said website score database.

10. The computing system of claim 9, wherein said offline analysis software application comprises a plurality of data mining software tools.

11. The computing system of claim 9, wherein said analyzing offline comprises performing a search for specified expressions within said content from each said sample web page, wherein each specified expression from said specified expressions is selected from the group consisting of a phrase and a word, and wherein said specified expressions are divided into a first group of expressions that are of positive interest to a user of said web crawler and a second group of expressions that are of negative interest to said user.

12. The computing system of claim 11, wherein said analyzing offline further comprises determining if each said sample web page comprises a hyperlink to an alternative web page comprising an expression from said first group or said second group.

13. The computing system of claim 9, wherein said requesting comprises a dynamic determination of a sample size of said sample set of web pages.

14. The computing system of claim 9, wherein said method further comprises:

excluding, by said online analysis software application, a group of web pages from said sample set of web pages based upon a plurality of heuristics.

15. A computer program product, comprising a computer usable medium including an online analysis software application, an offline analysis software application, a website score database, a web crawler, a set of candidate web pages, and computer readable program code embodied therein, said computer readable program code comprising an algorithm adapted to implement a prioritization method within a computing system, said method comprising:

extracting, by said web crawler, said set of candidate web pages to be crawled;

associating, by said online analysis software application, each web page in said set of candidate web pages with a website in a computer network;

determining online, by said online analysis software application, if a first website score for said website, is in said website score database;

associating, by said online analysis software application, said first website score for said website with associated web pages in said set of candidate web pages, if said first website score exists in said website score database;

prioritizing, said set of candidate web pages with respect to an associated website score for each web page in said candidate set of web pages;

retrieving, by said web crawler, content from said set of candidate web pages using said prioritizing;

extracting, by said online analysis software application, hyperlinks from said content;

storing said hyperlinks in said memory unit.

16. The computer program product of claim 15, further comprising a web page score database, wherein said determining online determines that said first website score is not available in said website score database, and wherein said method further comprises;

requesting, by said web crawler, a sample set of web pages from said website, wherein said sample set of web pages does not include a total set of web pages from said website;

analyzing offline, by said offline analysis software application, each sample web page of said sample web pages with a plurality of offline heuristics;

generating, by said offline analysis software application, a group of web page scores for each said sample web page based on results of said analyzing offline;

storing, each said group of web page scores in said web page score database;

generating, by said offline analysis software application, a final web page score for each said web page of said sample set of web pages, wherein each said final web page score is generated by combining each web page score within each said group of web page scores;

storing, each said final web page score in said web page score database;

generating, by said offline analysis software application, a second website score for said website, wherein said second website score is generated by combining said final web page scores for said sample set of web pages, and wherein said first website score is generated using a same technique as used to generate said second website score; and

storing, said second website score in said website score database.

17. The computer program product of claim 16, wherein said offline analysis software application comprises a plurality of data mining software tools.

18. The computer program product of claim 16, wherein said analyzing offline comprises performing a search for specified expressions within said content from each said sample web page, wherein each specified expression from said specified expressions is selected from the group consisting of a phrase and a word, and wherein said specified expressions are divided into a first group of expressions that are of positive interest to a user of said web crawler and a second group of expressions that are of negative interest to said user.

19. The computer program product of claim 16, wherein said requesting comprises a dynamic determination of a sample size of said sample set of web pages.

20. The computer program product of claim 16, wherein said method further comprises:

excluding, by said online analysis software application, a group of web pages from said sample set of web pages based upon a plurality of heuristics.