Managing URLs

• US 7,991,762 B1
• Filed: 03/24/2009
• Issued: 08/02/2011
• Est. Priority Date: 06/24/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Tammana, Praveen K., Alpert, Jesse L., Kurzion, Yair
• Primary Examiner(s): LEWIS, CHERYL RENEA
• Application Number:
  US12/410,440
• Time in Patent Office: 861 Days
• Field of Search: 707/709, 707/713, 707/692, 707/696
• US Class Current: 707/709
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation

Abstract

Crawling pages is disclosed. Pages are crawled up to a target number of pages. Additional pages, that have an importance that is equal to or greater than an importance threshold, are crawled beyond the target number of pages. In some embodiments, pages having an importance less than an importance threshold are deleted.

Claims

1. A system comprising:

a processor, coupled to a memory, configured to;

check an index of crawled items for a number of index entries in the index, wherein the index entries include references to the crawled items; and

when the number of index entries is equal to or greater than a target number, select one or more of the index entries for deletion from the index based on an importance of the crawled items referenced by the index entries.

2. The system of claim 1 wherein the selection of index entries selected for deletion from the index is based on the crawled items referenced by the selected index entries having an importance that is less than an unimportance threshold.

3. The system of claim 2 wherein the processor is further configured to:

calculate the unimportance threshold based at least in part on a histogram of a number of crawled items referenced by the index entries having various different importances.

4. The system of claim 2 wherein the unimportance threshold is set to a value lower than an importance threshold that is used to determine whether an item is to be crawled for inclusion in an index entry in the index.

5. The system of claim 1 wherein the target number is based on a license limit that governs a number of index entries in the index according to terms of the license.

6. The system of claim 1 wherein the processor is further configured to:

delete the selected index entries from the index; and

crawl additional items that have an importance that is equal to or greater than an importance threshold.

7. The system of claim 1, further comprising a ranking engine configured to assign an importance to items.

8. The system of claim 1, wherein an importance of an item is a query-independent metric associated with an importance of the item.

9. The system of claim 1, wherein the importance of an item is independent of a time the item was last accessed.

10. The system of claim 1, wherein the importance of an item is independent of a time the item was last crawled.

11. The system of claim 1, wherein the crawled items include documents.

12. The system of claim 1, wherein the crawled items include webpages.

13. The system of claim 1, wherein the crawled items include text files.

14. The system of claim 1, wherein the crawled items include multimedia files.

15. The system of claim 1, wherein the crawled items include items accessible within an intranet.

16. A recordable storage medium having recorded and stored thereon instructions that, when executed, perform the actions of:

crawling items until a specified number of items is crawled; and

when the specified number of items is crawled, crawling additional items only if the additional items comprise a designated criterion.

17. The recordable storage medium of claim 16 wherein the designated criterion comprises an importance that is equal to or greater than an importance threshold that is used to determine whether an item is to be crawled for inclusion in an index entry in the index, wherein an importance of an item is a query-independent metric associated with an importance of the item.

18. The recordable storage medium of claim 16 wherein the specified number is based on a license limit that governs a number of index entries in the index according to terms of the license.

19. The recordable storage medium of claim 16 wherein crawling additional items includes crawling the additional items until another specified number of items is crawled.

20. A method of crawling items, the method comprising:

crawling, using a processor of one or more devices, items until a specified number of items is crawled; and

when the specified number of items is crawled, crawling, using a processor of the one or more devices, additional items only if the additional items comprise a designated criterion.

21. The method of claim 20 wherein the designated criterion comprises an importance that is equal to or greater than an importance threshold that is used to determine whether an item is to be crawled for inclusion in an index entry in the index, wherein an importance of an item is a query-independent metric associated with an importance of the item.

22. The method of claim 20 wherein the specified number is based on a license limit that governs a number of index entries in the index according to terms of the license.

23. The method of claim 20 wherein crawling additional items comprises crawling, using a processor of the one or more devices, the additional items until another specified number of items is crawled.

24. A system comprising:

a processor, coupled to a memory, configured to;

crawl items until a specified number of items is crawled; and

when the specified number of items is crawled, crawl additional items only if the additional items comprise a designated criterion.

25. The system of claim 24 wherein the designated criterion comprises an importance that is equal to or greater than an importance threshold that is used to determine whether an item is to be crawled for inclusion in an index entry in the index, wherein an importance of an item is a query-independent metric associated with an importance of the item.

26. The system of claim 24 wherein the specified number is based on a license limit that governs a number of index entries in the index according to terms of the license.

27. The system of claim 24 wherein crawling additional items includes crawling the additional items until another specified number of items is crawled.

28. A recordable storage medium having recorded and stored thereon instructions that, when executed, perform the actions of:

checking an index of crawled items for a number of index entries in the index, wherein the index entries include references to the crawled items; and

when the number of index entries is equal to or greater than a target number, selecting one or more of the index entries for deletion from the index based on an importance of the crawled items referenced by the index entries.

29. The recordable storage medium of claim 28 wherein the selection of index entries selected for deletion from the index is based on the crawled items referenced by the selected index entries having an importance that is less than an unimportance threshold.

30. The recordable storage medium of claim 29 further comprising instructions that, when executed, perform the action of calculating the unimportance threshold based at least in part on a histogram of a number of crawled items referenced by the index entries having various different importances.

31. The recordable storage medium of claim 29 wherein the unimportance threshold is set to a value lower than an importance threshold that is used to determine whether an item is to be crawled for inclusion in an index entry in the index.

32. The recordable storage medium of claim 28 wherein the target number is based on a license limit that governs a number of index entries in the index according to terms of the license.

33. The recordable storage medium of claim 28 further comprising instructions that, when executed, perform the actions of:

deleting the selected index entries from the index; and

crawling additional items that have an importance that is equal to or greater than an importance threshold.

34. The recordable storage medium of claim 28, further comprising instructions that, when executed, perform the action of assigning an importance to items.

35. The recordable storage medium of claim 28, wherein the importance of an item is a query-independent metric associated with an importance of the item.

36. The recordable storage medium of claim 28, wherein the importance of an item is independent of a time the item was last accessed.

37. The recordable storage medium of claim 28, wherein the importance of an item is independent of a time the item was last crawled.

38. The recordable storage medium of claim 28, wherein the crawled items include documents.

39. The recordable storage medium of claim 28, wherein the crawled items include webpages.

40. The recordable storage medium of claim 28, wherein the crawled items include text files.

41. The recordable storage medium of claim 28, wherein the crawled items include multimedia files.

42. The recordable storage medium of claim 28, wherein the crawled items include items accessible within an intranet.

43. A method, comprising:

checking, using a processor of one or more devices, an index of crawled items for a number of index entries in the index, wherein the index entries include references to the crawled items; and

when the number of index entries is equal to or greater than a target number, selecting, using a processor of the one or more devices, one or more of the index entries for deletion from the index based on an importance of the crawled items referenced by the index entries.

44. The method of claim 43 wherein the selection of index entries selected for deletion from the index is based on the crawled items referenced by the selected index entries having an importance that is less than an unimportance threshold.

45. The method of claim 44 further comprising calculating the unimportance threshold based at least in part on a histogram of a number of crawled items referenced by the index entries having various different importances.

46. The method of claim 44 wherein the unimportance threshold is set to a value lower than an importance threshold that is used to determine whether an item is to be crawled for inclusion in an index entry in the index.

47. The method of claim 43 wherein the target number is based on a license limit that governs a number of index entries in the index according to terms of the license.

48. The method of claim 43 further comprising:

deleting, using a processor of the one or more devices, the selected index entries from the index; and

crawling, using a processor of the one or more devices, additional items that have an importance that is equal to or greater than an importance threshold.

49. The method of claim 43, further comprising assigning an importance to items.

50. The method of claim 43, wherein the importance of an item is a query-independent metric associated with an importance of the item.

51. The method of claim 43, wherein the importance of an item is independent of a time the item was last accessed.

52. The method of claim 43, wherein the importance of an item is independent of a time the item was last crawled.

53. The method of claim 43, wherein the crawled items include documents.

54. The method of claim 43, wherein the crawled items include webpages.

55. The method of claim 43, wherein the crawled items include text files.

56. The method of claim 43, wherein the crawled items include multimedia files.

57. The method of claim 43, wherein the crawled items include items accessible within an intranet.

58. A method comprising:

crawling items up to a target number of items, wherein at least a subset of the crawled items are not constrained to have an importance;

crawling additional items in the collection beyond the target number of items, wherein the additional items are constrained to have an importance that is equal to or greater than an importance threshold;

providing, in an index, for each of at least a subset of the crawled items and the additional crawled items crawl data associated with the respective item;

checking the index for a number of index entries in the index, wherein the index entries include references to the crawled items; and

selecting, when the number of index entries is equal to or greater than a target number, one or more of the index entries for deletion from the index based on an importance of the crawled items referenced by the index entries.