Index updating using segment swapping

• US 7,702,614 B1
• Filed: 03/30/2007
• Issued: 04/20/2010
• Est. Priority Date: 03/30/2007
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Shah, Nirav, McCulloch, Trevor
• Primary Examiner(s): Wong; Don
• Assistant Examiner(s): Nguyen; Merilyn P
• Application Number:
  US11/694,789
• Time in Patent Office: 1,117 Days
• Field of Search: 707 1- 5, 707/7, 707/100, 707/200, 707/203
• US Class Current: 707/741
• CPC Class Codes: G06F 16/319 Inverted lists
• CPC Class Codes: G06F 16/3341 using boolean model
• CPC Class Codes: Y10S 707/99931 Database or file accessing

Abstract

An information retrieval system uses phrases to index, retrieve, organize and describe documents. Phrases are extracted from the document collection. Documents are the indexed according to their included phrases, using phrase posting lists. The phrase posting lists are stored in an cluster of index servers. The phrase posting lists can be tiered into groups, and sharded into partitions. Phrases in a query are identified based on possible phrasifications. A query schedule based on the phrases is created from the phrases, and then optimized to reduce query processing and communication costs. The execution of the query schedule is managed to further reduce or eliminate query processing operations at various ones of the index servers.

Claims

1. A computer implemented method of maintaining a phrase index for a plurality

of documents in a document collection, the method comprising:

providing a set of phrase posting lists, each phrase posting list associated with a phrase;

establishing a plurality of segments, each segment associated with a subset of the plurality of the documents;

periodically updating each segment by;

for documents associated with the segment, identifying phrases contained in the document, and updating the phrase posting list for each such phrase to include the document;

sharding the phrase posting lists for the identified phrases into a plurality of segment shards, each segment shard containing a disjoint subset of the list of documents in the segment that contain the phrase associated with the phrase posting list;

associating each segment shard with an index shard, such that at least one index shard is associated with a plurality of segment shards, each index shard being served by an index server;

determining a recently updated segment having updated segment shards;

for at least one index shard being served;

determining the index shard'"'"'s associated updated segment shards, and merging the updated segment shards with the index shard to form an updated index shard; and

replacing the index shard with the updated index shard.

2. The method of claim 1, wherein at least one document is included in multiple segments.

3. The method of claim 2, further comprising assigning different document identifiers to instances of the document included in different segments.

4. The method of claim 1, further comprising establishing a segment phrase index of phrase posting lists for the documents associated with a segment.

5. The method of claim 1, further comprising determining locations of segment shard files corresponding to the most recent version of each segment.

6. A computer program product stored on a computer readable medium and comprising instructions that when executed cause a computer system to:

provide a set of phrase posting lists, each phrase posting list associated with a phrase;

establish a plurality of segments, each segment associated with a subset of a plurality of the documents;

periodically update each segment by;

for documents associated with the segment, identifying phrases contained in the document, and updating the phrase posting list for each such phrase to include the document;

sharding the phrase posting lists for the identified phrases into a plurality of segment shards, each segment shard containing a disjoint subset of the list of documents in the segment that contain the phrase associated with the phrase posting list;

associating each segment shard with an index shard, such that at least one index shard is associated with a plurality of segment shards, each index shard being served by an index server;

determine a recently updated segment having updated segment shards;

for at least one index shard being served;

determine the index shard'"'"'s associated updated segment shards, and merging the updated segment shards with the index shard to form an updated index shard; and

replace the index shard with the updated index shard.

7. The computer program product of claim 6, wherein at least one document is included in multiple segments.

8. The computer program product of claim 7, further comprising instructions that when executed cause the computer system to assign different document identifiers to instances of the document included in different segments.

9. The computer program product of claim 6 further comprising instructions that when executed cause the computer system to establish a segment phrase index of phrase posting lists for the documents associated with a segment.

10. The computer program product of claim 6 further comprising instructions that when executed cause the computer system to determine locations of segment shard files corresponding to the most recent version of each segment.