Storing log data efficiently while supporting querying to assist in computer network security
First Claim
1. A computer-implemented method for processing log data, comprising:
- receiving log data that comprises a plurality of events, an event including one or more fields; and
for each event in the plurality of events;
storing, in a buffer, the event;
identifying a first value of a first field of the event;
identifying a first minimum value that indicates a minimum value of the first field of all of the events stored in the buffer, wherein the first minimum value is stored in a metadata structure that comprises information about contents of the buffer;
determining whether the first minimum value exceeds the first value; and
responsive to determining that the first minimum value exceeds the first value, updating the metadata structure by replacing the first minimum value with the first value.
11 Assignments
0 Petitions
Accused Products
Abstract
A logging system includes an event receiver and a storage manager. The receiver receives log data, processes it, and outputs a data “chunk.” The manager receives data chunks and stores them so that they can be queried. The receiver includes buffers that store events and a metadata structure that stores metadata about the contents of the buffers. The metadata includes a unique identifier associated with the receiver, the number of events in the buffers, and, for each “field of interest,” a minimum value and a maximum value that reflect the range of values of that field over all of the events in the buffers. A chunk includes the metadata structure and a compressed version of the contents of the buffers. The metadata structure acts as a search index when querying event data. The logging system can be used in conjunction with a security information/event management (SIEM) system.
40 Citations
18 Claims
-
1. A computer-implemented method for processing log data, comprising:
-
receiving log data that comprises a plurality of events, an event including one or more fields; and for each event in the plurality of events; storing, in a buffer, the event; identifying a first value of a first field of the event; identifying a first minimum value that indicates a minimum value of the first field of all of the events stored in the buffer, wherein the first minimum value is stored in a metadata structure that comprises information about contents of the buffer; determining whether the first minimum value exceeds the first value; and responsive to determining that the first minimum value exceeds the first value, updating the metadata structure by replacing the first minimum value with the first value. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16)
-
-
17. A computer program product for processing log data, the computer program product comprising a non-transitory computer-readable medium containing computer program code for performing a method, the method comprising:
-
receiving log data that comprises a plurality of events, an event including one or more fields; and for each event in the plurality of events; storing, in a buffer, the event; identifying a first value of a first field of the event; identifying a first minimum value that indicates a minimum value of the first field of all of the events stored in the buffer, wherein the first minimum value is stored in a metadata structure that comprises information about contents of the buffer; determining whether the first minimum value exceeds the first value; and responsive to determining that the first minimum value exceeds the first value, updating the metadata structure by replacing the first minimum value with the first value.
-
-
18. An apparatus for processing log data, comprising:
-
a processor; and a computer-readable storage medium coupled to the processor and storing instructions executed by the processor to perform operations comprising; receiving log data that comprises a plurality of events, an event including one or more fields; for each event in the plurality of events; storing, in a buffer, the event; identifying a first value of a first field of the event; identifying a first minimum value that indicates a minimum value of the first field of all of the events stored in the buffer, wherein the first minimum value is stored in a metadata structure that comprises information about contents of the buffer; determining whether the first minimum value exceeds the first value; and responsive to determining that the first minimum value exceeds the first value, updating the metadata structure by replacing the first minimum value with the first value.
-
Specification