DISTRIBUTED EVENT PROCESSING

• US 20140244650A1
• Filed: 12/20/2011
• Published: 08/28/2014
• Est. Priority Date: 11/04/2011
• Status: Abandoned Application

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: Micro Focus LLC (Open Text Corporation)
• Inventors: Zhou, Yizheng, Huang, Wei, Aguilar-Macias, Hector, Wiser, David Earl, Weston, Michael Scott
• Application Number:
  US14/352,121
• Publication Number:
  US 20140244650A1
• Time in Patent Office: Days
• US Class Current: 707/737
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 43/12 Network monitoring probes

Abstract

A distributed event processing method includes providing a plurality of connectors. Each provided connector is configured to acquire event data from an assigned data source, partition acquired event data into clusters, and divide each cluster into chunks. The method also includes collecting the chunks from the plurality of connectors and storing the chunks to a data file that can be queried.

Claims

1. A distributed event processing method, comprising:

providing a plurality of connectors each connector configured to acquire event data from an assigned data source, partition acquired event data into clusters, and divide each duster into chunks,

collecting the chunks from the plurality of connectors; and

storing the chunks to a data file that can be queried.

2. The method of claim 1, wherein providing comprises providing a plurality of connectors, each connector configured to:

partition by partitioning the acquired event data into clusters, each cluster including rows of event data segmented into columns of event fields;

divide by dividing each cluster into chunks where each chunk includes the event fields of a particular column of that cluster.

3. The method of claim 2, wherein providing comprises providing a plurality of connectors, each connector configured to divide by dividing each cluster into compressed chunks.

4. The method of claim 2, wherein:

providing comprises providing a plurality of connectors, each connector configured to divide each cluster into chunks wherein each chunk is associated with metadata identifying that chunk and an attribute of the chunk, the associated metadata included in or otherwise linked to its corresponding chunk; and

collecting comprises collecting the chunks and associated metadata from the plurality of connectors.

5. The method of claim 4 wherein storing comprises merging the collected chunks and storing the merged chunks to the data file and maintaining an index for the data file from the collected metadata.

6. A non-transitory computer readable medium including instructions that when executed cause a processor to:

collect chunks from a plurality of connectors each configured to acquire event data from an assigned data source, partition acquired event data into clusters, and divide each cluster into chunks, and

store the chunks to a data file that can be queried.

7. The medium of claim 6, wherein each cluster partitioned by the plurality of connectors includes rows of event data divided into columns of event fields, and wherein the instructions, when executed, cause a processor to collect chunks from the plurality of connectors, wherein each collected chunk includes the event fields of a particular column of the cluster from which it was divided.

8. The medium of claim 7, wherein each chunk is associated with metadata identifying that chunk and an attribute of the chunk, the associated metadata included in or otherwise linked to that chunk, and wherein the instructions, when executed, cause the processor to collect the chunks and associated metadata from the plurality of connectors.

9. The medium of claim 8 wherein the instructions, when executed, cause the processor to:

merge the collected chunks;

store the merged chunks to the data file;

maintain an index for the data file utilizing the collected metadata.

10. The medium of claim 9, wherein the instructions, when executed, cause the processor to examine the index to identify chunks in the data file that are relevant to a query.

11. A distributed event processing system, comprising a plurality of connectors and a storage manager, wherein:

each connector is configured to acquire event data from an assigned data source, partition acquired event data into clusters, and divide each cluster into chunks, and

the storage manager is configured to collect the chunks from the plurality of connectors and store the collected chunks to a data file that can be queried.

12. The system of claim 11, wherein each cluster can be represented by a table having a plurality of rows each representing an event and including a plurality of event fields, each connector being configured to divide by dividing each cluster into chunks where each chunk includes the event fields defining a particular column of that cluster.

13. The system of claim 12 wherein each connector configured to divide each cluster into chunks such that each chunk is associated with metadata identifying that chunk and an attribute of the chunk, the associated metadata included in or otherwise linked to its corresponding chunk;

and

The storage manager is configured to collecting the chunks and associated metadata from the plurality of connectors.

14. The system of claim 13 wherein the storage manager is configured to:

merging the collected chunks;

store the merged chunks to the data file; and

maintain an index for the data file from the collected metadata.

15. The system of claim 14, wherein the storage manager is configured to examine the index to identify chunks in the data file that are relevant to a query and to return the identified chunks or data included in the identified chunks in response to the query.