Index server architecture using tiered and sharded phrase posting lists

• US 9,652,483 B1
• Filed: 11/23/2015
• Issued: 05/16/2017
• Est. Priority Date: 03/30/2007
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Cao, Pei, Mazumdar, Soham, Eiron, Nadav, Zunger, Yonatan, Patterson, Anna L., Power, Russell
• Primary Examiner(s): Vy, Hung T
• Application Number:
  US14/949,194
• Time in Patent Office: 540 Days
• Field of Search: 707736, 707741, 707792, 707802, 707830
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: Y10S 707/99931 Database or file accessing

Abstract

An information retrieval system uses phrases to index, retrieve, organize and describe documents. Phrases are extracted from the document collection. Documents are the indexed according to their included phrases, using phrase posting lists. The phrase posting lists are stored in an cluster of index servers. The phrase posting lists can be tiered into groups, and sharded into partitions. Phrases in a query are identified based on possible phrasifications. A query schedule based on the phrases is created from the phrases, and then optimized to reduce query processing and communication costs. The execution of the query schedule is managed to further reduce or eliminate query processing operations at various ones of the index servers.

Claims

1. A phrase-based indexing system comprising:

a first tier of index servers including N index servers, each of which stores a portion of a phrase posting list for each of a plurality of phrases, each phrase posting list being associated with a phrase and a list of documents having at least one occurrence of the phrase; and

M additional tiers of index servers, wherein;

M is one to a predetermined number,

each M^th tier includes T index servers, where T is an integer multiple of N when M equals one and where T is an integer multiple of T for an (M−

1)^th tier when M is greater than or equal to two, and

each M^th tier index server stores a portion of a phrase posting list of each of a plurality of phrases, each phrase posting list being associated with a phrase and a list of documents having at least one occurrence of the phrase.

2. The system of claim 1, wherein an index server operates as a server program.

3. The system of claim 1, wherein documents are assigned to an index server based on a document identifier for the document.

4. The system of claim 1, wherein phrases are assigned to a tier based on a cost of the respective phrase posting list.

5. The system of claim 1, wherein for a given phrase posting list, the index servers that store the given phase posting list operate concurrently to serve their documents to other index servers for query intersection.

6. The system of claim 1, wherein a given index server in the first tier communicates with at most the integer multiple of the N index servers in the M equals one tier during query processing.

7. The system of claim 1, wherein M is greater than one and a given index server in the M equals two tier communicates with at most the integer multiple of T for the M equals one tier during query processing.

8. The system of claim 1, wherein an index server in tier M uses the integer multiple of tier (M+1) to determine, using a shard assignment function, which index servers in the M+1 tier to communicate with during query processing.