SEMANTICALLY AGGREGATED INDEX IN AN INDEXER-AGNOSTIC INDEX BUILDING SYSTEM

• US 20120179684A1
• Filed: 01/12/2011
• Published: 07/12/2012
• Est. Priority Date: 01/12/2011
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Alba, Alfredo, Ercegovac, Vuk, Griffin, Thomas D., Rao, Jun, DeLuca, Chad E., Singh, Asim V., Wang, Kevin B.
• Granted Patent: US 9,104,749 B2
• Time in Patent Office: Days
• US Class Current: 707/738
• CPC Class Codes: G06F 16/192 Implementing virtual folder structures
• CPC Class Codes: G06F 16/328 Management therefor
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification

Abstract

A computer program product for an indexer-agnostic index building system includes a computer readable storage medium to store a computer readable program, wherein the computer readable program, when executed on a computer, causes the computer to perform operations for creating a semantically aggregated index. The operations include: extracting documents from a data source, wherein each document includes a data object; distributing the documents to a plurality of processing nodes within the system; for each node: indexing the data objects for each document into fields using semantic rules; and grouping indexed data objects for related fields by: classifying the documents into logical groups based on the semantic rules; and creating a searchable index shard for related logical groups.

Claims

1. A computer program product for an indexer-agnostic index building system, comprising:

a computer readable storage medium to store a computer readable program, wherein the computer readable program, when executed on a computer, causes the computer to perform operations for creating a semantically aggregated index, the operations comprising;

extracting documents from a data source, wherein each document includes a data object;

distributing the documents to a plurality of processing nodes within the system;

for each node indexing the data objects for each document into fields using semantic rules; and

grouping indexed data objects for related fields by;

classifying the documents into logical groups based on the semantic rules; and

creating a searchable index shard for related logical groups.

2. The computer program product of claim 1, wherein a distribution load of the documents is approximately balanced among the processing nodes after distributing the documents.

3. The computer program product of claim 1, wherein the data objects comprise a full text of the document.

4. The computer program product of claim 1, wherein the semantic rules comprise a rule based on at least one of the group consisting of:

country of origin, language, topic, and metadata for the data objects in the document.

5. The computer program product of claim 1, wherein each processing node contributes a portion of the index shard.

6. The computer program product of claim 1, wherein the operations are implemented using distributed computing software, wherein distributing the documents to a plurality of nodes comprises distributing the documents to a mapper at each processing node, and grouping the indexed data objects comprises grouping the data objects at a reducer for each index shard created by the system.

7. The computer program product of claim 1, wherein the computer program product, when executed on the computer, causes the computer to perform additional operations, comprising:

determining a scope of the semantically aggregated index based on a custom user query.

8. The computer program product of claim 1, wherein the semantically aggregated index comprises a plurality of index shards, wherein the computer program product, when executed on the computer, causes the computer to perform additional operations, comprising:

processing each index shard in parallel across the processing nodes.

9. A method for creating a semantically aggregated index in an indexer-agnostic index building system, the method comprising:

extracting documents from a data source, wherein each document includes a data object;

distributing the documents to a plurality of processing nodes within the system;

for each node indexing the data objects for each document into fields using semantic rules; and

grouping indexed data objects for related fields by;

classifying the documents into logical groups based on the semantic rules; and

creating a searchable index shard for related logical groups.

10. The method of claim 9, wherein a distribution load of the documents is generally balanced among the processing nodes after distributing the documents.

11. The method of claim 9, wherein the data objects comprise a full text of each document.

12. The method of claim 9, wherein the semantic rules comprise a rule based on at least one of the group consisting of:

country of origin, language, topic, and metadata for the data objects in the document.

13. The method of claim 9, wherein each processing node contributes a portion of the index shard.

14. The method of claim 9, wherein the operations are implemented using distributed computing software, wherein distributing the documents to a plurality of nodes comprises distributing the documents to a mapper at each processing node, and grouping the indexed data objects comprises grouping the data objects at a reducer for each index shard created by the system.

15. An indexer-agnostic index building system, comprising:

a plurality of processing nodes connected to a computer readable storage medium; and

a processor configured to perform operations, the operations comprising;

extracting documents from the data source and storing the documents on the computer readable storage medium, wherein each document includes at least one data object;

distributing the documents to the processing nodes;

for each node indexing the data objects for each document into fields using semantic rules; and

grouping indexed data objects for related fields by;

classifying the documents into logical groups based on the semantic rules; and

creating a searchable index shard for related logical groups.

16. The system of claim 15, wherein a distribution load of the documents is generally balanced among the processing nodes after distributing the documents.

17. The system of claim 15, wherein the data objects comprise a full text of each document.

18. The system of claim 15, wherein the semantic rules comprise a rule based on at least one of the group consisting of:

country of origin, language, topic, and metadata for the data objects in the document.

19. The system of claim 15, wherein each processing node contributes a portion of the index shard.

20. The system of claim 15, wherein the operations are implemented using distributed computing software, wherein distributing the documents to a plurality of nodes comprises distributing the documents to a mapper at each processing node, and grouping the indexed data objects comprises grouping the data objects at a reducer for each index shard created by the system.