Storing log data efficiently while supporting querying
First Claim
1. A method for processing log data, comprising:
- receiving log data that comprises a plurality of events, wherein an event includes a set of fields, and wherein a field stores a value; and
for each event in the plurality of events;
storing the event in a set of buffers, wherein each field value of the event is stored in a different buffer;
identifying a first value stored in a first field of the event;
identifying a first minimum value that indicates a minimum value of the first field of all of the events stored in the buffers, wherein the first minimum value is stored in a metadata structure that comprises information about contents of the buffers;
determining whether the first minimum value exceeds the first value; and
responsive to determining that the first minimum value exceeds the first value, updating the metadata structure by replacing the first minimum value with the first value.
11 Assignments
0 Petitions
Accused Products
Abstract
A logging system includes an event receiver and a storage manager. The receiver receives log data, processes it, and outputs a column-based data “chunk.” The manager receives and stores chunks. The receiver includes buffers that store events and a metadata structure that stores metadata about the contents of the buffers. Each buffer is associated with a particular event field and includes values from that field from one or more events. The metadata includes, for each “field of interest,” a minimum value and a maximum value that reflect the range of values of that field over all of the events in the buffers. A chunk is generated for each buffer and includes the metadata structure and a compressed version of the buffer contents. The metadata structure acts as a search index when querying event data. The logging system can be used in conjunction with a security information/event management (SIEM) system.
50 Citations
21 Claims
-
1. A method for processing log data, comprising:
-
receiving log data that comprises a plurality of events, wherein an event includes a set of fields, and wherein a field stores a value; and for each event in the plurality of events; storing the event in a set of buffers, wherein each field value of the event is stored in a different buffer; identifying a first value stored in a first field of the event; identifying a first minimum value that indicates a minimum value of the first field of all of the events stored in the buffers, wherein the first minimum value is stored in a metadata structure that comprises information about contents of the buffers; determining whether the first minimum value exceeds the first value; and responsive to determining that the first minimum value exceeds the first value, updating the metadata structure by replacing the first minimum value with the first value. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19)
-
-
20. A computer program product for processing log data, the computer program product comprising a machine-readable storage medium containing computer program code for performing a method, the method comprising:
-
receiving log data that comprises a plurality of events, wherein an event includes a set of fields, and wherein a field stores a value; and for each event in the plurality of events; storing the event in a set of buffers, wherein each field value of the event is stored in a different buffer; identifying a first value stored in a first field of the event; identifying a first minimum value that indicates a minimum value of the first field of all of the events stored in the buffers, wherein the first minimum value is stored in a metadata structure that comprises information about contents of the buffers; determining whether the first minimum value exceeds the first value; and responsive to determining that the first minimum value exceeds the first value, updating the metadata structure by replacing the first minimum value with the first value.
-
-
21. A system for processing log data, comprising:
-
a machine-readable storage medium containing computer program code for performing a method, the method comprising; receiving log data that comprises a plurality of events, wherein an event includes a set of fields, and wherein a field stores a value; and for each event in the plurality of events; storing the event in a set of buffers, wherein each field value of the event is stored in a different buffer; identifying a first value stored in a first field of the event; identifying a first minimum value that indicates a minimum value of the first field of all of the events stored in the buffers, wherein the first minimum value is stored in a metadata structure that comprises information about contents of the buffers; determining whether the first minimum value exceeds the first value; and responsive to determining that the first minimum value exceeds the first value, updating the metadata structure by replacing the first minimum value with the first value; and a processor configured to execute the computer program code stored by the machine-readable medium.
-
Specification