Realtime indexing and search in large, rapidly changing document collections

• US 7,634,466 B2
• Filed: 08/02/2006
• Issued: 12/15/2009
• Est. Priority Date: 06/28/2005
• Status: Expired due to Fees

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Rose, Daniel E., Walters, Chad, Mao, Jianchang
• Primary Examiner(s): Fleurantin; Jean B.
• Assistant Examiner(s): LE, THU NGUYET T
• Application Number:
  US11/498,706
• Publication Number:
  US 20060294086A1
• Time in Patent Office: 1,231 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

The present invention is directed to systems and methods for searching content items indexed in real-time. The method according to one embodiment comprises generating an index of word location pairs that identifies the location of one or more words in one or more content items available on a network. One or more additional content items are received over the network. The received content items are stored in a stream search queue, the stream search queue operative to allow for a stream search of the one or more additional content items.

Claims

1. A computerized method for indexing content items, the method comprising:

generating, using a processor, an inverted index of word location pairs that identifies the location of one or more words in one or more content items available on a network;

storing the inverted index in an index data store;

dynamically receiving one or more additional content items over the network;

prior to elapsing of a predetermined time threshold, storing the one or more additional content items in a stream search queue, the stream search queue operative to allow for a stream search of the one or more additional content items;

once the time threshold elapses, indexing, using the processor, the one or more additional content items in the stream search queue and then writing the indexed content from the stream search queue into the inverted index;

receiving a query from a user, the query comprising one or more query executing a stream search of the stream search queue to identify a given one of the query terms and to generate a stream search result set;

executing an index search of the inverted index of word location pairs to identify a given one of the query terms and generate an index result set; and

generating a merge result set on the basis of the stream result set and the index result set.

2. The method of claim 1 wherein generating the inverted index of word location pairs comprises indexing the location of one or more words that appear in one or more web pages.

3. The method of claim 1 comprising:

transmitting the merge result set to the user for display on a client device.

4. The method of claim 1 wherein if the time threshold has not expired, adding the one or more content items in the stream search queue.

5. The method of claim 1 comprising indexing the one or more content items from the stream search queue into the inverted index of word location pairs upon receipt of a subsequent one or more content items.

6. The method of claim 1 wherein generating the inverted index comprises indexing one or more items of personalized information.

7. The method of claim 6 comprising retrieving the personalized information from one or more user profiles.

8. The method of claim 6 comprising:

if the time threshold has expired, indexing the one or more items of personalized information into the inverted index of word location pairs.

9. The method of claim 8 wherein if the time threshold has not expired, adding the personalized information in the stream search queue.

10. The method of claim 1 comprising:

receiving one or more items of personalized information; and

serially indexing a given item of the one or more items of personalized information into the inverted index when the given item reaches the beginning of the stream search queue.

11. The method of claim 3 wherein executing the stream search and executing the index search comprises applying a scoring technique and wherein generating the merge result set comprises ordering according to the scoring technique.

12. An apparatus for indexing content items, the apparatus comprising:

a processor, in response to executable instructions from a computer readable medium, operative to;

generate an inverted index of word location pairs that identifies the location of one or more words in one or more content items available on a network;

store the inverted index in an index data store;

dynamically receive one or more additional content items over the network;

prior to elapsing of a predetermined time threshold, store the one or more additional content items in a stream search queue, the stream search queue operative to allow for a stream search of the one or more additional content items;

once the time threshold elapses, index the one or more additional content items in the stream search queue and then writing the indexed content from the stream search queue into the inverted index;

receiving a query from a user, the query comprising one or more query terms;

executing a stream search of the stream search queue to identify a given one of the query terms and to generate a stream search result set;

executing an index search of the inverted index of word location pairs to identify a given one of the query terms and generate an index result set; and

generating a merge result set on the basis of the stream result set and the index result set.