Bulk Search Index Updates

• US 20090193406A1
• Filed: 01/29/2008
• Published: 07/30/2009
• Est. Priority Date: 01/29/2008
• Status: Abandoned Application

• Current Assignee: MetaLINCS Corporation (Seagate Technology Holdings Plc)
• Original Assignee: MetaLINCS Corporation (Seagate Technology Holdings Plc)
• Inventors: Williams, James Charles
• Application Number:
  US12/022,073
• Publication Number:
  US 20090193406A1
• Time in Patent Office: Days
• US Class Current: 717/168
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Embodiments of the present invention perform bulk updates of a search index for an information repository. In embodiments, a batched set of update requests is run and a set of documents to be updated based on the set of requests is identified. In embodiments, a bulk update method to use is selected based on an estimate of the cost of performing the bulk update. In embodiments, a bulk update method based on updating only the indexes of the documents to be updated may be used instead of a bulk update method that involves re-indexing the full set of documents in the repository.

Claims

1. A method for updating a document search index having a plurality of index segments, the method comprising:

executing at least one update request comprising at least one transformation, the at least one update request identifying a plurality of documents to be updated within the document index;

identifying a set of matching index segments, within the document search index, associated with the plurality of documents to be updated;

updating a first set of stored fields associated with the plurality of documents to be updated by applying the at least one transformation to modify the first set of stored fields;

generating a modified postings list for the document search index corresponding to the updated first set of stored fields; and

updating a list of postings of the document search index based on the modified postings list.

2. The method of claim 1 wherein:

a set of postings and a second set of stored fields being associated with a document are represented by a set of multiple indexes within the document search index; and

the second set of stored fields is a subset of the first set of stored fields.

3. The method of claim 2 wherein at least one of the multiple indexes, within the set of multiple indexes, comprises an immutable stored field.

4. The method of claim 1 wherein the step of updating a first set of stored fields comprises:

generating an inverted index of transformations associated with the plurality of documents to be updated; and

updating a second set of stored fields of a document, within the plurality of documents, by;

copying an unmodified stored field, within the second set of stored fields, into a new index segment; and

updating a modified stored field, within the second set of stored fields, by applying at least one transformation to the modified stored field and writing the modified stored field into the new index segment.

5. The method of claim 1 wherein the step of generating a modified postings list comprises creating an inverted index of modified postings associated with the plurality of documents to be updated.

6. The method of claim 1 wherein the step of updating the list of postings comprises:

copying a posting into a new index segment in response to the posting not being on the modified postings list; and

writing a modified posting into the new index segment in response to the modified posting being on the modified postings list.

7. The method of claim 1 wherein a bit vector is maintained that identifies the plurality of documents to be updated within the document search index.

8. A computer readable medium having instructions for performing the method of claim 1.

9. A method for identifying a bulk update for a document search index having a plurality of index segments, the method comprising:

identifying a plurality of documents to be updated, within the document search index, based on at least one update request comprising at least one transformation;

identifying a first set of matching index segments, within the plurality of index segments, associated with the plurality of documents to be updated;

determining a first processing cost for updating the document search index associated with the plurality of documents to be updated;

determining a second processing cost for updating the document search index associated with the first set of matching index segments; and

selecting the bulk update for updating the document search index, the selected bulk update being at least partially based on a relative comparison of the first processing cost to the second processing cost.

10. The method of claim 9 wherein the first processing cost relates to an amount of computer resources required to update the document search index with complete index data from the documents to be updated.

11. The method of claim 9 wherein the second processing cost relates to an amount of computer resources required to update the document search index with index data from the first set of matching index segments.

12. The method of claim 11 wherein the index data from the first set of matching index segments is updated responsive to the second processing cost being less than the first processing cost.

13. The method of claim 12 further comprising the steps of:

updating a first set of stored fields associated with the first set of matching index segments by applying the at least one transformation;

generating a modified postings list for the document search index corresponding to the updated first set of stored fields; and

updating a list of postings of the document search index based on the modified postings list.

14. The method of claim 13 wherein:

a set of postings and a second set of stored fields being associated with a document are represented by a set of multiple indexes within the document search index; and

the second set of stored fields is a subset of the first set of stored fields.

15. The method of claim 14 wherein at least one of the multiple indexes, within the set of multiple indexes, comprises an immutable stored field.

16. The method of claim 13 wherein the step of updating a first set of stored fields comprises:

generating an inverted index of transformations associated with the plurality of documents to be updated; and

updating a second set of stored fields of a document, within the plurality of documents, by performing a method of steps comprising;

copying an unmodified stored field, within the second set of stored fields, into a new index segment; and

updating a modified stored field, within the second set of stored fields, by applying at least one transformation to the modified stored field and writing the modified stored field into the new index segment.

17. The method of claim 13 wherein the step of updating the list of postings comprises:

copying a posting into a new index segment in response to the posting not being on the modified postings list; and

writing a modified posting into the new index segment in response to the modified posting being on the modified postings list.

18. A computer readable medium having instructions for performing the method of claim 9.

19. A system for applying a set of update requests to a document search index having a plurality of index segments, the system comprising:

a matching document identifier, coupled to receive the set of update requests, that identifies a plurality of documents to be updated within the document search index;

an update method selector, coupled to receive the plurality of documents to be updated, that selects a bulk update for the document search index by performing a method comprising the steps of;

identifying a first set of matching index segments, within the plurality of index segments, associated with the plurality of documents to be updated;

determining a first processing cost for updating the document search index associated with the plurality of documents to be updated;

determining a second processing cost for updating the document search index associated with the first set of matching index segments; and

selecting the bulk update for updating the document search index, the selected bulk update being at least partially based on a relative comparison of the first processing cost to the second processing cost; and

an index updater that applies the selected bulk update to the document search index.

20. The system of claim 19 wherein a bit vector is maintained that identifies the plurality of documents to be updated.

21. The system of claim 19 wherein index updater performs the steps of:

updating a first set of stored fields associated with the first set of matching index segments by applying the at least one transformation;

generating a modified postings list for the document search index corresponding to the updated first set of stored fields; and

updating a list of postings of the document search index based on the modified postings list.

22. The system of claim 21 wherein:

a set of postings and a second set of stored fields being associated with a document are represented by a set of multiple indexes within the document search index; and

the second set of stored fields is a subset of the first set of stored fields.

23. The system of claim 21 wherein the step of updating a first set of stored fields comprises:

generating an inverted index of transformations associated with the plurality of documents to be updated; and

updating a second set of stored fields of a document, within the plurality of documents, by performing a method comprising the steps of;

copying an unmodified stored field, within the second set of stored fields, into a new index segment; and

updating a modified stored field, within the second set of stored fields, by applying at least one transformation to the modified stored field and writing the modified stored field into the new index segment.

24. An index updater that applies a set of update requests to a document search index having a plurality of index segments, the system comprising:

a stored fields updater, coupled to receive a plurality of documents to be updated and a set of matching index segments, the stored fields updater updates a set of stored fields associated with the plurality of documents to be updated; and

a postings updater, coupled to receive the updated set of stored fields, the postings updater updates a list of postings of the document search index by performing a method comprising the steps of;

generating a modified postings list for the document search index corresponding to the updated set of stored fields; and

updating the list of postings of the document search index at least partially based on the modified postings list.

25. The system of claim 24, the system further comprising a transformation indexer, coupled to receive the plurality of documents to be updated and the set of update requests comprising at least one transformation, the transformation indexer generates an inverted index of transformations associated with the plurality of documents to be updated.

26. The system of claim 24 wherein a set of postings and a set of stored fields associated with a document are represented by a set of multiple indexes within the document search index.

27. The system of claim 24 wherein updating a set of stored fields associated with the plurality of documents to be updated comprises:

generating an inverted index of transformations associated with the plurality of documents to be updated; and

updating a second set of stored fields of a document, within the plurality of documents, by performing a method comprising the steps of;

copying an unmodified stored field, within the second set of stored fields, into a new index segment; and

updating a modified stored field, within the second set of stored fields, by applying at least one transformation to the modified stored field and writing the modified stored field into the new index segment.

28. The system of claim 24 wherein generating a modified postings list comprises creating an inverted index of modified postings associated with the plurality of documents to be updated.

29. The system of claim 24 wherein updating the list of postings comprises:

copying a posting into a new index segment in response to the posting not being on the modified postings list; and

writing a modified posting into the new index segment in response to the modified posting being on the modified postings list.