Managing URLs

• US 7,509,315 B1
• Filed: 06/24/2005
• Issued: 03/24/2009
• Est. Priority Date: 06/24/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Tammana, Praveen K., Alpert, Jesse L., Kurzion, Yair
• Primary Examiner(s): LEWIS, CHERYL RENEA
• Application Number:
  US11/166,786
• Time in Patent Office: 1,369 Days
• Field of Search: 707 2- 5, 707/100, 707/101, 715/205, 715/234
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation

Abstract

Crawling pages is disclosed. Pages are crawled up to a target number of pages. Additional pages, that have an importance that is equal to or greater than an importance threshold, are crawled beyond the target number of pages. In some embodiments, pages having an importance less than an importance threshold are deleted.

Claims

1. A method of crawling pages including:

crawling pages up to a target number of pages, at least a subset of which are not constrained to have an importance;

crawling additional pages beyond the target number of pages, wherein the additional pages are constrained to have an importance that is equal to or greater than an importance threshold; and

providing as output for each of at least a subset of the crawled pages and additional pages a crawl data associated with the respective page;

wherein an importance is a query independent metric associated with the page.

2. The method of claim 1 wherein the target limit is a license limit.

3. The method of claim 1 wherein the target limit is based on a license limit.

4. The method of claim 1 wherein the importance threshold is a fixed value.

5. The method of claim 1 wherein the importance threshold is dynamic.

6. The method of claim 1 wherein the importance threshold is based at least in part on a histogram.

7. The method of claim 1 wherein the importance is a PageRank.

8. The method of claim 1 further comprising deleting one or more pages.

9. The method of claim 8 wherein the pages deleted have an importance that is less than an unimportance threshold.

10. The method of claim 9 wherein the unimportance threshold is the importance threshold.

11. The method of claim 8 wherein deleting one or more pages is performed asynchronously with crawling pages.

12. The method of claim 1 wherein the target number of pages represents a soft limit.

13. A system for crawling pages comprising:

a processor, coupled to a memory, configured to;

crawl pages up to a target number of pages, at least a subset of which are not constrained to have an importance;

crawl additional pages beyond the target number of pages, wherein the additional pages are constrained to have an importance that is greater than an importance threshold; and

provide as output for each of at least a subset of the crawled pages and additional pages a crawl data associated with the respective page;

wherein an importance is a query independent metric associated with the page;

and

a memory coupled to the processor, wherein the memory provides the processor with instructions.

14. The system of claim 13 wherein the target limit is a license limit.

15. The system of claim 13 wherein the importance threshold is dynamic.

16. The system of claim 13 wherein the importance threshold is based at least in part on a histogram.

17. The system of claim 13 wherein the processor further comprises instructions for deleting one or more pages.

18. The system of claim 17 wherein the pages deleted have an importance that is less than an unimportance threshold.

19. A computer program product for crawling pages, the computer program product being embodied in a computer readable medium and comprising computer instructions for:

crawling pages up to a target number of pages, at least a subset of which are not constrained to have an importance;

crawling additional pages beyond the target number of pages, wherein the additional pages are constrained to have an importance that is greater than an importance threshold; and

providing as output for each of at least a subset of the crawled pages and additional pages a crawl data associated with the respective page;

wherein an importance is a query independent metric associated with the page.

20. The computer program product of claim 19, wherein the importance threshold is dynamic.

21. The computer program product of claim 19 wherein the importance threshold is a fixed value.

22. The computer program product of claim 19 wherein the importance threshold is based at least in part on a histogram.

23. The computer program product of claim 19 wherein the importance in a PageRank.

24. The computer program product of claim 19 further comprising computer instructions for deleting one or more pages.

25. The computer program product of claim 24 wherein the pages deleted have an importance that is less than an unimportance threshold.

26. The computer program product of claim 25 wherein the unimportance threshold is the importance threshold.

27. The computer program product of claim 24 wherein deleting one or more pages is performed asynchronously with crawling pages.