CLUSTERING EVENT DATA BY MULTIPLE TIME DIMENSIONS

• US 20140359771A1
• Filed: 01/26/2012
• Published: 12/04/2014
• Est. Priority Date: 12/28/2007
• Status: Active Grant

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: entIT Software LLC (Open Text Corporation)
• Inventors: Tang, Wenting, Orayani, Marylou, Dash, Debabrata
• Granted Patent: US 9,853,986 B2
• Time in Patent Office: Days
• US Class Current: 726/23
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 16/335 Filtering based on additional data, e.g. user or group profiles filtering in web context G06F16/9535, G06F16/9536
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: G06F 2221/2151 Time stamp
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1441 Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002

Abstract

Systems and methods for processing log data are provided. A set of data chunks is determined. Each data chunk is associated with a set of events, which are grouped according to a primary time dimension field of each event of the set of events. A metadata structure is determined for each of the data chunks. The metadata structure includes comprises a range of the primary time dimension field of all of the events in the data chunk and a range of a secondary time dimension field of all of the events in the data chunk. A subset of the data chunks is selected. A data chunk associated with at least one event of the plurality of events is generated according to the secondary time dimension field of the at least one event.

Claims

1. A method for processing log data, the method comprising:

determining, by a computing device, a set of data chunks, each data chunk includes a set of events clustered according to a primary time dimension field of each event of the set of events;

for each data chunk of the set of data chunks, determining a metadata structure that comprises a range of the primary time dimension field of all of the events in the data chunk and a range of a secondary time dimension field of all of the events in the data chunk;

selecting a subset of the data chunks;

disassembling the subset of data chunks into a plurality of events; and

generating a data chunk including at least one event of the plurality of events, the event is clustered in the data chunk according to the secondary time dimension field of the at least one event.

2. The method of claim 1, wherein the range of the primary time dimension field comprises minimum value of the primary time dimension field of all of the events in the data chunk and a maximum value of the primary time dimension field of all of the events in the data chunk, and wherein the range of the secondary time dimension field comprises a minimum value of the secondary time dimension field of all of the events in the data chunk and a maximum value of the secondary time dimension field of all of the events in the data chunk.

3. The method of claim 1, wherein selecting the subset comprises:

for each data chunk in the set of data chunks,

determining a density level of the chunk; and

determining, based on the density level of the chunk, the subset comprises the data chunk if the data chunk is a sparse chunk or a dense chunk.

4. The method of claim 3, wherein determining the density level comprises:

determining a number of events associated with the chunk; and

dividing the number of events by a range of the secondary time dimension field of the events of the chunk.

5. The method of claim 3, further comprising:

comparing a range of the secondary time dimension field of the chunk to a dense time range threshold;

determining the chunk is a dense chunk if the range meets the dense time range threshold;

comparing the range of the secondary time dimension field of the chunk to a sparse time range threshold; and

determining the chunk is a sparse chunk if the range meets the sparse time range threshold.

6. The method of claim 5, further comprising:

comparing the range of the secondary time dimension field of the chunk to a range of the secondary time dimension field of the sparse chunk; and

determining the chunk is an overlap chunk if the range of the chunk overlaps with the range of the sparse chunk.

7. The method of claim 1, wherein a plurality of data chunks are generated, and wherein the density level of the data chunk is balanced among the plurality of generated data chunks.

8. The method of claim 1, wherein the primary time dimension field is an event receipt time.

9. The method of claim 1, wherein the secondary time dimension field is an event occurrence time.

10. The method of claim 1, further comprising:

storing the generated chunk in a datafile in a read-optimized store; and

updating the metadata structure to include information about the generated chunk.

11. The method of claim 1, wherein the metadata structure further comprises a location of each chunk of the set of chunks in a write-optimized store, and a location of the generated chunk in a read-optimized store.

12. The method of claim 1, further comprising:

receiving a search query that includes a set of search terms;

identifying at least one search term, from the set of search terms, that concerns event time information that is contained in the metadata structure; and

searching the metadata structure by comparing the identified search term to the minimum value of the primary time dimension field and to the minimum value of the secondary time dimension field.

13. The method of claim 12, further comprising:

identifying a data chunk that satisfies the search terms; and

retrieving the identified data chunk from a read-optimized store.

14. A system for processing log data, comprising:

a receiving module to generate a set of data chunks, each data chunk includes a set of events clustered according to a primary time dimension field of each event of the set of events;

a chunks table to maintain, for each data chunk of the set of data chunks, a metadata structure that comprises a range of the primary time dimension field of all of the events in the data chunk and a range of a secondary time dimension field of all of the events in the data chunk;

a read-optimized store to store the set of data chunks;

a write-optimized store; and

a clustering module to select a subset of the data chunks, and generate a data chunk using events of the subset, wherein the events of the subset are grouped according to the secondary time dimension field.

15. A non-transitory computer-readable medium storing a plurality of instructions to control a data processor to process log data, the plurality of instructions comprising instructions that cause the data processor to:

determine a set of data chunks, each data chunk includes a set of events clustered according to a primary time dimension field of each event of the set of events;

for each data chunk of the set of data chunks, determine a metadata structure that comprises a range of the primary time dimension field of all of the events in the data chunk and a range of a secondary time dimension field of all of the events in the data chunk;

select a subset of the data chunks;

disassemble the subset of data chunks into a plurality of events; and

generate a data chunk including at least one event of the plurality of events clustered in the data chunk according to the secondary time dimension field of the at least one event.