Assigning document identification tags

• US 9,411,889 B2
• Filed: 03/13/2012
• Issued: 08/09/2016
• Est. Priority Date: 07/03/2003
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Zhu, Huican, Acharya, Anurag
• Primary Examiner(s): Tran, Quoc A
• Application Number:
  US13/419,349
• Publication Number:
  US 20120173552A1
• Time in Patent Office: 1,610 Days
• Field of Search: 715/862, 715/783, 715/273, 715/708, 715201-203, 715/740, 715/745, 434/118, 434/362, 709213-219
• US Class Current: 1/1
• CPC Class Codes: G06F 16/156 Query results presentation
• CPC Class Codes: G06F 16/214 Database migration support
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0246 Traffic
• CPC Class Codes: H04L 41/12 Discovery or management of network topologies
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 63/08 for authentication of entities cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32
• CPC Class Codes: H04L 63/102 Entity profiles
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/14 Session management for real-time applications in data packet communications networks H04L65/1066
• CPC Class Codes: H04L 67/561 Adding application-functional data or data for application control, e.g. adding metadata
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

Document identification tags are assigned to documents to be added to a collection of documents. Based on query-independent information about a new document, a document identification tag is assigned to the new document. The document identification tag so assigned is used in the indexing of the new document. When a list of document identification tags are produced by an index in response to a query, the list is approximately ordered with respect to a measure of query-independent relevance. In some embodiments, the measure of query-independent relevance is related to the connectivity matrix of the World Wide Web. In other embodiments, the measure is related to the recency of crawling. In still other embodiments, the measure is a mixture of these two. The provided systems and methods allow for real-time indexing of documents as they are crawled from a collection of documents.

Claims

1. A computer-implemented method of assigning a document identifier to a new document, the new document to be added to a collection of documents, the method being performed on a computer system having one or more processors and memory storing one or more programs for execution by the one or more processors, the method comprising:

partitioning a set of document identifiers into a plurality of segments, each segment associated with a respective subset of the set of document identifiers, wherein the document identifiers comprise a predetermined set of monotonically ordered document identification tags;

subdividing each of the segments into a plurality of tiers, wherein each tier is associated with a respective subset of the set of document identifiers, and wherein the plurality of tiers are monotonically ordered with respect to a query-independent document importance metric;

receiving query-independent information about the new document, the information including a value of the query-independent document importance metric and a unique document identifier for the new document;

selecting, based at least in part on the unique document identifier, one of the segments;

selecting, based at least on the query-independent information, one of the tiers associated with the selected segment;

assigning to the new document a document identifier from the respective subset of document identifiers associated with the selected tier, the assigned document identifier not previously assigned to any of the documents in the collection of documents, and

repeating the receiving, selecting a segment, selecting a tier, and assigning, with respect to one or more additional new documents.

2. The method of claim 1, wherein each tier in the plurality of tiers is associated with a respective predetermined range of metric values;

and

wherein selecting a tier comprises selecting the tier for which the query-independent document importance metric of the new document falls within the respective predetermined range of metric values associated with the selected tier.

3. The method of claim 2, wherein the respective predetermined ranges of metric values associated with the plurality of tiers are non-overlapping.

4. The method of claim 2, wherein the respective subset of the document identifiers associated with each tier monotonically increases with the position of the tier in the ordering;

and

wherein the respective predetermined range of metric values associated with each tier monotonically increases with the position of the tier in the ordering.

5. The method of claim 4, wherein assigning a document identifier to the new document comprises assigning to the new document a minimum available document identifier identification tag from the respective subset of document identifiers associated with the selected tier.

6. The method of claim 4, wherein assigning a document identifier to the new document comprises assigning to the new document a maximum available document identifier from the respective subset of document identifiers associated with the selected tier.

7. The method of claim 1, further comprising:

when a flush condition is satisfied, performing a flush operation, including building a sorted map, the sorted map relating globally unique identifiers to document identifiers assigned to documents since a prior flush operation.

8. The method of claim 7, further comprising:

when a merge condition is satisfied, performing a merge operation, the merge operation including merging a layered plurality of sorted maps produced by previous flushing operations, the merge operation further including producing a merged map relating globally unique identifiers to document identifiers assigned to documents.

9. The method of claim 1, further comprising:

when a flush condition is satisfied, performing a flush operation, the flush operation including building a first sorted map and a second sorted map;

wherein the first sorted map is keyed and sorted by globally unique identifiers, and includes for each globally unique identifier a corresponding document identifier; and

wherein the second sorted map is keyed and sorted by document identifiers assigned to documents since a prior flush operation, and includes for each such document identifier a corresponding globally unique identifier.

10. The method of claim 9, wherein the globally unique identifiers are URL fingerprints.

11. The method of claim 10, wherein each URL fingerprint comprises a value produced by applying a one way mapping function to an address associated with a document in the collection of documents.

12. A computer-implemented method of assigning a plurality of document identification tags to a plurality of new documents, the plurality of new documents to be added to a collection of documents, the method comprising:

on a computer system having one or more processors and memory storing one or more programs for execution by the one or more processors;

partitioning a set of valid globally unique document identifiers into a plurality of segments, each segment associated with a respective subset of the set of valid globally unique document identifiers;

subdividing each of the segments into a plurality of tiers, wherein the plurality of tiers are monotonically ordered with respect to a query-independent document importance metric, each segment having an associated, predetermined set of monotonically ordered document identification tags, and each tier of a respective segment having an associated subset of the set of document identification tags for the respective segment;

receiving query-independent information about a new document, the information including a value of the query-independent document importance metric and a globally unique document identifier for the new document;

selecting, based at least in part on the globally unique document identifier, one of the segments;

selecting, based at least on the query-independent information, one of the tiers associated with the selected segment;

assigning to the new document a document identification tag from the subset of document identification tags associated with the selected tier, wherein the document identification tag assigned to the new document is unique with respect to document identification tags assigned to other documents in the collection of documents; and

repeating the receiving, selecting a segment, selecting a tier, and assigning, with respect to one or more additional new documents;

wherein the assigned document identification tags are assigned to documents in the collection of documents having globally unique document identifiers associated with the respective segment.

13. The method of claim 12, wherein the set of document identification tags for a first segment includes a plurality of document identification tags in the set of document identification tags for a second segment.

14. A system for assigning a document identification tag to a new document, the new document to be added to a collection of documents, the system comprising:

one or more processors; and

memory storing one or more programs to be executed by the one or more processors;

the one or more programs comprising instructions for;

partitioning a set of monotonically ordered document identification tags into a plurality of segments, each segment associated with a respective subset of the set of monotonically ordered document identification tags;

subdividing each of the segments into a plurality of tiers, wherein each tier is associated with a respective subset of the set of document identification tags, and wherein the plurality of tiers are monotonically ordered with respect to a query-independent document importance metric;

receiving query-independent information about the new document, the information including a value of the query-independent document importance metric and a globally unique document identifier for the new document;

selecting, based at least in part on the globally unique document identifier, one of the segments;

selecting, based at least on the query-independent information, one of the tiers associated with the selected segment;

assigning to the new document a document identification tag from the subset of document identification tags associated with the selected tier, the assigned document identification tag not previously assigned to any of the documents in the collection of documents;

repeating the receiving, selecting a segment, selecting a tier, and assigning, with respect to one or more additional new documents; and

wherein the assigned document identification tags are assigned to documents in the collection of documents having globally unique document identifiers associated with the respective segment.

15. The system of claim 14, wherein each tier in the plurality of tiers is associated with a respective predetermined range of metric values;

and

wherein instructions for selecting a tier comprise instructions for selecting the tier for which the query-independent information of the new document falls within the respective predetermined range of metric values associated with the selected tier.

16. The system of claim 15, wherein the respective predetermined ranges of metric values associated with the plurality of tiers are non-overlapping.

17. The system of claim 15, wherein the respective subset of the set of document identification tags associated with each tier monotonically increase with the position of the tier in the ordering;

and

wherein the respective predetermined range of metric values associated with each tier monotonically increases with the position of the tier in the ordering.

18. The system of claim 17, wherein the instructions for assigning a document identification tag to the new document comprise instructions for assigning to the new document a minimum available document identification tag from the respective subset of document identification tags associated with the selected tier.

19. The system of claim 17, wherein the instructions for assigning a document identification tag to the new document comprise instructions for assigning to the new document a maximum available document identification tag from the respective subset of document identification tags associated with the selected tier.

20. The system of claim 14, further comprising instructions for:

when a flush condition is satisfied, performing a flush operation, including building a sorted map, the sorted map relating globally unique identifiers to document identification tags assigned to documents since a prior flush operation.

21. The system of claim 20, further comprising instructions for:

when a merge condition is satisfied, performing a merge operation, the merge operation including merging a layered plurality of sorted maps produced by previous flushing operations, the merge operation further including producing a merged map relating globally unique identifiers to document identification tags assigned to documents.

22. The system of claim 14, further comprising instructions for:

when a flush condition is satisfied, performing a flush operation, the flush operation including building a first sorted map and a second sorted map;

wherein the first sorted map is keyed and sorted by globally unique identifiers, and includes for each globally unique identifier a corresponding document identification tag; and

wherein the second sorted map is keyed and sorted by document identification tags assigned to documents since a prior flush operation, and includes for each such document identification tag a corresponding globally unique identifier.

23. The system of claim 14, wherein the globally unique identifiers are URL fingerprints.

24. The system of claim 23, wherein each URL fingerprint comprises a value produced by applying a one way mapping function to an address associated with a document in the collection of documents.

25. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

partitioning a set of monotonically ordered document identification tags into a plurality of segments, each segment associated with a respective subset of the set of monotonically ordered document identification tags;

subdividing each of the segments into a plurality of tiers, wherein each tier is associated with a respective subset of the set of document identification tags, and wherein the plurality of tiers are monotonically ordered with respect to a query-independent document importance metric;

receiving query-independent information about a new document, the information including a value of the query-independent document importance metric and a globally unique document identifier for the new document;

selecting, based at least in part on the globally unique document identifier, one of the segments;

selecting, based at least on the query-independent information, one of the tiers associated with the selected segment;

assigning to the new document a document identification tag from the respective subset of document identification tags associated with the selected tier, the assigned document identification tag not previously assigned to any of the documents in a collection of documents;

repeating the receiving, selecting a segment, selecting a tier, and assigning, with respect to one or more additional new documents; and

wherein the assigned document identification tags are assigned to documents in the collection of documents having globally unique document identifiers associated with the respective segment.

26. The non-transitory computer readable storage medium of claim 25, wherein each tier in the plurality of tiers is associated with a respective predetermined range of metric values;

and

wherein instructions for selecting a tier comprise instructions for selecting the tier for which the query-independent information of the new document falls within the respective predetermined range of metric values associated with the selected tier.

27. The non-transitory computer readable storage medium of claim 26, wherein the respective predetermined ranges of metric values associated with the plurality of tiers are nonoverlapping.

28. The non-transitory computer readable storage medium of claim 26, wherein the respective subset of the document identification tags associated with each tier monotonically increase with the position of the tier in the ordering;

and

wherein the respective predetermined range of metric values associated with each tier monotonically increases with the position of the tier in the ordering.

29. The non-transitory computer readable storage medium of claim 28, wherein the instructions for assigning a document identification tag to the new document comprise instructions for assigning to the new document a minimum available document identification tag from the respective subset of document identification tags associated with the selected tier.

30. The non-transitory computer readable storage medium of claim 28, wherein the instructions for assigning a document identification tag to the new document comprise instructions for assigning to the new document a maximum available document identification tag from the respective subset of document identification tags associated with the selected tier.

31. The non-transitory computer readable storage medium of claim 25, further comprising instructions for:

when a flush condition is satisfied, performing a flush operation, including building a sorted map, the sorted map relating globally unique identifiers to document identification tags assigned to documents since a prior flush operation.

32. The non-transitory computer readable storage medium of claim 31, further comprising instructions for:

when a merge condition is satisfied, performing a merge operation, the merge operation including merging a layered plurality of sorted maps produced by previous flushing operations, the merge operation further including producing a merged map relating globally unique identifiers to document identification tags assigned to documents.

33. The non-transitory computer readable storage medium of claim 25, instructions for:

when a flush condition is satisfied, performing a flush operation, the flush operation including building a first sorted map and a second sorted map;

wherein the first sorted map is keyed and sorted by globally unique identifiers, and includes for each globally unique identifier a corresponding document identification tag; and

wherein the second sorted map is keyed and sorted by document identification tags assigned to documents since a prior flush operation, and includes for each such document identification tag a corresponding globally unique identifier.

34. The non-transitory computer readable storage medium of claim 25, wherein the globally unique identifiers are URL fingerprints.

35. The non-transitory computer readable storage medium of claim 34, wherein each URL fingerprint comprises a value produced by applying a one way mapping function to an address associated with a document in the collection of documents.