Uniform resource locator scoring for targeted web crawling

• US 20080104113A1
• Filed: 10/26/2006
• Published: 05/01/2008
• Est. Priority Date: 10/26/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Natarajan, Ramakrishnan, Yao, Tong, Kim, Joon Young, Wong, Sandy, Thogersen, Michael D., Huynh, Yet L.
• Granted Patent: US 7,672,943 B2
• Time in Patent Office: Days
• US Class Current: 707/104.1
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

A web crawler system as described herein utilizes a targeted approach to increase the likelihood of downloading web pages of a desired type or category. The system employs a plurality of URL scoring metrics that generate individual scores for outlinked URLs contained in a downloaded web page. For each outlinked URL, the individual scores are combined using an appropriate algorithm or formula to generate an overall score that represents a downloading priority for the outlinked URL. The web crawler application can then download subsequent web pages in an order that is influenced by the downloading priorities.

Claims

1. A targeted web crawling method comprising:

analyzing a downloaded web page that links to an outlinked web page having a URL and belonging to a sub-group of web pages;

generating a domain density score in response to the sub-group, where the domain density score indicates relevance of the URL to a desired web page type;

generating an anchor text score in response to anchor text of the URL, where the anchor text score indicates probability that the outlinked web page is of the desired web page type;

generating a URL string score in response to characters of the URL, where the URL string score indicates probability that the outlinked web page is of the desired web page type;

generating a category need score in response to characteristics of the downloaded web page, where the category need score is influenced by a current distribution of a plurality of categories associated with the desired web page type; and

calculating a downloading priority for the URL in response to the domain density score, the anchor text score, the URL string score, and the category need score.

2. A method according to claim 1, further comprising downloading, in an order determined by the downloading priority, a second web page that corresponds to the URL.

3. A method according to claim 1, further comprising generating a link proximity score for the URL, where the link proximity score indicates linking distance from the URL to a linking URL that corresponds to the desired web page type.

4. A method according to claim 3, wherein the downloading priority is influenced by the link proximity score.

5. A method according to claim 3, further comprising:

downloading, in an order determined by the downloading priority, a second web page that corresponds to the URL, the second web page containing a second outlinked web page having a second URL; and

processing the downloading priority when generating a second link proximity score for the second URL.

6. A method according to claim 1, wherein generating the domain density score comprises:

obtaining a ratio of a number of indexed pages to a number of processed pages, where the number of indexed pages represents a number of web pages from the sub-group having the desired web page type, and where the number of processed pages represents a total number of web pages from the sub-group processed by the web crawling method; and

calculating the domain density score from the ratio.

7. A method according to claim 1, wherein generating the anchor text score comprises:

extracting words from the anchor text of the URL; and

for each word extracted from the anchor text of the URL, calculating a word score that indicates probability of relevance to the desired web page type.

8. A method according to claim 1, wherein generating the anchor text score comprises:

extracting words from the anchor text of the URL;

identifying at least one combination of words extracted from the anchor text of the URL; and

for each combination of words, calculating a combined word score that indicates probability of relevance to the desired web page type.

9. A method according to claim 1, wherein generating the URL string score comprises:

extracting strings from the URL; and

for each string extracted from the URL, calculating a string score that indicates probability of relevance to the desired web page type.

10. A method according to claim 1, wherein generating the URL string score comprises:

extracting strings from the URL;

identifying at least one combination of strings extracted from the URL; and

for each combination of strings, calculating a combined string score that indicates probability of relevance to the desired web page type.

11. A method according to claim 1, wherein generating the category need score comprises:

identifying a category for the downloaded web page in response to the analyzing step, wherein the category is one of the plurality of categories;

if representation of the category in the current distribution is relatively high, generating a relatively low category need score; and

if representation of the category in the current distribution is relatively low, generating a relatively high category need score.

12. A computer-readable medium having computer-executable instructions for performing steps comprising:

obtaining information about a first web page that includes a link to a second web page having a URL;

generating a plurality of scores for the URL, each of the plurality of scores indicating a different measure related to whether the URL corresponds to a desired web page type;

calculating a downloading priority for the URL in response to the plurality of scores; and

providing the URL along with the downloading priority to a web crawler application.

13. A computer-readable medium according to claim 12, further comprising computer-executable instructions for performing a domain density metric that results in a domain density score for the URL.

14. A computer-readable medium according to claim 12, further comprising computer-executable instructions for performing an anchor text metric that results in an anchor text score for the URL.

15. A computer-readable medium according to claim 12, further comprising computer-executable instructions for performing a URL string score metric that results in a URL string score for the URL.

16. A computer-readable medium according to claim 12, further comprising computer-executable instructions for performing a category need metric that results in a category need score for the URL.

17. A computer-readable medium according to claim 12, further comprising computer-executable instructions for performing a link proximity metric that results in a link proximity score for the URL.

18. A web crawler system comprising:

a web crawler core module configured to download web pages in accordance with a downloading priority scheme;

a web page classifier coupled to the web crawler core module, the web page classifier being configured to analyze a first web page downloaded by the web crawler core module, the first web page having an outgoing link to a second web page corresponding to a URL; and

a URL scoring module coupled to the web page classifier and to the web crawler core module, the URL scoring module being configured to assign a downloading priority to the URL based upon a plurality of metrics;

wherein

each of the plurality of metrics indicates a different measure of probability that the second web page is of a designated type.

19. A web crawler system according to claim 18, wherein the URL scoring module is configured to assign the downloading priority to the URL based upon a category need metric that indicates a predicted category for the second web page.

20. A web crawler system according to claim 18, wherein the plurality of metrics comprises:

a domain density metric that results in a domain density score for the URL;

an anchor text metric that results in an anchor text score for the URL;

a URL string score metric that results in a URL string score for the URL; and

a link proximity metric that results in a link proximity score for the URL.