Index replication using crawl modification information

• US 7,945,533 B2
• Filed: 02/23/2007
• Issued: 05/17/2011
• Est. Priority Date: 03/01/2006
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Bhavsar, Meeten, Chang, Thomas H., Delgado, Joaquin A., Yang, Chi-Ming, Krishnaprasad, Muralidhar
• Primary Examiner(s): LeRoux; Etienne P
• Assistant Examiner(s): Nguyen; Cindy
• Application Number:
  US11/710,100
• Publication Number:
  US 20070208716A1
• Time in Patent Office: 1,544 Days
• Field of Search: 707/3, 707/610, 707/611, 707/620, 707/637, 707/638, 707/615, 709/248
• US Class Current: 707/637
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Systems, methodologies, media, and other embodiments associated with index replication using crawl modification information are described. One exemplary system embodiment includes an enterprise search system comprising a target search system comprising an index logic that uses modified crawl information related to items associated with sources to maintain an index that supports searching of the items; and, a crawl search system comprising a pipeline processor configured to receive modified crawl information related to the items and to propagate the modified crawl information to the target system.

Claims

1. An enterprise search system, comprising:

a target search system comprising an index logic, stored in a non-transitory computer-readable medium, that is configured to receive modified crawl information from a crawl search system and to use the modified crawl information to maintain an index that supports searching of the items, where the modified crawl information is related to items associated with sources; and

,

the crawl search system comprising a pipeline processor configured to receive the modified crawl information related to the items and to propagate the modified crawl information to the target search system,

where the modified crawl information describes changes in the items gathered by the crawl search system and comprises at least one of changes to an item, metadata or security information.

2. The enterprise search system of claim 1, where the index functionally replicates a primary index of the crawl search system in a non-transitory computer-readable medium.

3. The enterprise search system of claim 2, where the index and the primary index employ different storage mechanisms.

4. The enterprise search system of claim 1, further comprising a plurality of target search systems, where the index of each target system independently functionally replicates a primary index of the crawl search system by using the modified crawl information to update the index.

5. The enterprise search system of claim 1, further comprising a plurality of target search systems, where the indexes of the target systems collectively functionally replicate a primary index of the crawl search system by using the modified crawl information to update the index;

and

the pipeline processor selectively forwarding the modified crawl information to a subset of the plurality of target search systems based, at least in part, on a type of the items, where a subset is less than the plurality.

6. The enterprise search system of claim 1, where the items comprise at least one of documents, files, web pages, emails spread sheets and/or databases.

7. The enterprise search system of claim 1, further comprising a crawler logic configured to re-index a modified item marked for re-indexing by the crawl search system to provide modified crawl information that describes the modifications determined from the re-indexing to the pipeline processor.

8. An enterprise search system, comprising:

a plurality of target search systems, each target search system comprising an index logic that uses modified crawl information related to items associated with sources to update an index that supports searching of the items where the modified crawl information describes changes in the items gathered by a crawl search system;

the crawl search system comprising;

a pipeline processor configured to receive modified crawl information related to the items and to propagate the modified crawl information to the plurality of target systems; and

,

a crawler logic configured to access the items to determine a modification to the items and selectively mark items for re-indexing if there is a modification, the crawler logic further configured to provide modified crawl information to the pipeline processor.

9. The enterprise search system of claim 8, where each of the plurality of target search systems is configured to process only a particular type of item and the pipeline processor propagates only the particular type of item to a search system associated with the particular type.

10. The enterprise search system of claim 8, where each of the plurality of target search systems is configured to process only items associated with a particular source.

11. The enterprise search system of claim 8, where each of the plurality of target search systems is configured to process only items associated with one or more entities.

12. The enterprise search system of claim 8, where the index of each target system independently functionally replicates a primary index of the crawl search system.

13. The enterprise search system of claim 8, where the indexes of the target systems collectively functionally replicate a primary index of the crawl search system.

14. A distributed enterprise crawl system, comprising:

a first search system and a second search system, each search system comprising;

an index logic that uses modified crawl information related to items associated with sources to maintain an index that supports searching of the items;

a pipeline processor configured to receive modified crawl information related to the items and to propagate the modified crawl information to the other search system; and

,

a crawler logic configured to access the items to determine a modification to the items, the crawler logic further configured to provide modified crawl information that describes changes in the items gathered by the crawler logic to the pipeline processor and to the index of the particular search system,

where the first search system is configured to access items associated with a first source and the second search system is configured to access items associated with a second source.

15. The distributed enterprise crawl system of claim 14, where the index of each search system independently functionally replicates the index of the other search system.

16. A computer implemented method for replicating search information, comprising:

crawling items associated with sources;

identifying changes to the items;

broadcasting information regarding the identified changes to a plurality of target search systems, where each of the plurality of target search systems is configured to process only items associated with a particular source; and

,

independently updating, in a non-transitory computer-readable medium, an index associated with each target search system based on the broadcasted identified changes where each target search system updates the associated index using identified changes associated with a particular source.

17. The method of claim 16, where the broadcasting includes collecting the identified changes and broadcasting the identified changes to the plurality of target search systems, where the identified changes to an item are a less-than-whole portion of the item.

18. The method of claim 16, where identifying changes to the items includes re-indexing items that have been marked by the crawling.

19. The method of claim 16, where the index of each target system independently functionally replicates a primary index of a crawl search system.

20. The method of claim 16, where the indexes of the target systems collectively functionally replicate a primary index of a crawl search system.

21. A non-transitory computer-readable medium for providing processor executable instructions that when executed cause a computer to perform a method for replicating search information in a computing system comprising a plurality of search systems and a plurality of crawlers, the method comprising:

distributing crawling assignments of a plurality of information sources between the plurality of crawlers;

crawling and identifying, by each of the crawlers, changes to items from one or more assigned information sources;

broadcasting, by each of the crawlers, the identified changes to the plurality of search systems, where each of the plurality of target search systems is configured to process only items associated with a particular source; and

independently updating, by each of the search systems, an index to the plurality of information sources using the identified changes broadcasted from each of the crawlers where each target search system updates an associated index using identified changes associated with a particular source that are broadcasted by the crawlers.