STORING LOG DATA EFFICIENTLY WHILE SUPPORTING QUERYING

• US 20100011031A1
• Filed: 09/04/2009
• Published: 01/14/2010
• Est. Priority Date: 12/28/2006
• Status: Active Grant

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: Arcsight Incorporated (HP Inc.)
• Inventors: Huang, Wei, Yu, Bin, Zhou, Yizheng, Beedgen, Christian F., Tang, Wenting
• Granted Patent: US 9,166,989 B2
• Time in Patent Office: Days
• US Class Current: N/A
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: G06F 2221/034 Test or assess a computer or a system
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection

Abstract

A logging system includes an event receiver and a storage manager. The receiver receives log data, processes it, and outputs a column-based data “chunk.” The manager receives and stores chunks. The receiver includes buffers that store events and a metadata structure that stores metadata about the contents of the buffers. Each buffer is associated with a particular event field and includes values from that field from one or more events. The metadata includes, for each “field of interest,” a minimum value and a maximum value that reflect the range of values of that field over all of the events in the buffers. A chunk is generated for each buffer and includes the metadata structure and a compressed version of the buffer contents. The metadata structure acts as a search index when querying event data. The logging system can be used in conjunction with a security information/event management (SIEM) system.

Claims

1. A method for processing log data, comprising:

receiving log data that comprises a plurality of events, wherein an event includes a set of fields, and wherein a field stores a value; and

for each event in the plurality of events;

storing the event in a set of buffers, wherein each field of the event is associated with a different buffer; and

updating a metadata structure that comprises information about contents of the buffers,

wherein information about contents of the buffers includes a first minimum value that reflects a minimum value of a first field of all of the events stored in the buffers.

2. The method of claim 1, wherein information about contents of the buffers further includes a first maximum value that reflects a maximum value of the first field of all of the events stored in the buffers.

3. The method of claim 1, wherein information about contents of the buffers further includes a second minimum value that reflects a minimum value of a second field of all of the events stored in the buffers.

4. The method of claim 1, wherein information about contents of the buffers further includes a number of events stored in the buffers.

5. The method of claim 1, further comprising:

in response to a first trigger condition, generating one data chunk for each buffer in the set of buffers, wherein the data chunk is based on contents of the metadata structure and further based on contents of the buffer.

6. The method of claim 5, wherein the first trigger condition is based on a buffer usage threshold or based on a timeout window.

7. The method of claim 5, further comprising:

in response to a second trigger condition, reclaiming storage space used by one of the data chunks.

8. The method of claim 7, wherein the second trigger condition is based on a retention policy associated with the data chunk.

9. The method of claim 7, wherein the second trigger condition is based on a disk-space usage threshold or based on a maximum time to retain the data chunk.

10. The method of claim 5, further comprising:

in response to a second trigger condition, archiving one of the data chunks.

11. The method of claim 10, wherein the second trigger condition is based on archival criteria associated with the data chunk.

12. The method of claim 10, wherein the second trigger condition comprises a user command.

13. The method of claim 1, further comprising:

for each event in the plurality of events;

determining when the event was received; and

storing, in an additional buffer, a timestamp that reflects when the event was received.

14. The method of claim 1, wherein storing the event in the set of buffers comprises storing each field value of the event in the buffer associated with that field.

15. The method of claim 1, further comprising generating one data chunk for each buffer in the set of buffers, wherein the data chunk includes contents of the metadata structure and a compressed version of contents of the buffer.

16. The method of claim 15, wherein the data chunk further includes a file signature or a version identifier.

17. The method of claim 15, wherein the data chunk further includes a message digest of contents of the buffer.

18. The method of claim 15, further comprising:

receiving a search query that includes a set of one or more search terms;

identifying one or more first search terms, from the set of search terms, that concern information that is contained in the metadata structure; and

searching one or more data chunks by comparing, for each data chunk, the identified first search terms to contents of the metadata structure included within the data chunk.

19. The method of claim 18, further comprising:

for each data chunk that satisfies the identified first search terms;

identifying one or more second search terms, from the set of search terms, that concern the field associated with the buffer that is associated with the data chunk;

comparing the identified second search terms to an uncompressed version of contents of the buffer included within the data chunk.

20. The method of claim 19, further comprising:

for each field value that satisfies the identified second search terms;

identifying an index location identifier that is associated with the field value;

for each buffer in the set of buffers;

accessing the data chunk associated with the buffer; and

identifying a field value in the data chunk that matches the index location identifier; and

assembling the identified field values into an event.

21. A computer program product for processing log data, the computer program product comprising a machine-readable storage medium containing computer program code for performing a method, the method comprising:

receiving log data that comprises a plurality of events, wherein an event includes a set of fields, and wherein a field stores a value; and

for each event in the plurality of events;

storing the event in a set of buffers, wherein each field of the event is associated with a different buffer; and

updating a metadata structure that comprises information about contents of the buffers, wherein information about contents of the buffers includes a first minimum value that reflects a minimum value of a first field of all of the events stored in the buffers.

22. A system for processing log data, comprising:

a machine-readable storage medium containing computer program code for performing a method,

the method comprising;

receiving log data that comprises a plurality of events, wherein an event includes a set of fields, and wherein a field stores a value; and

for each event in the plurality of events;

storing the event in a set of buffers, wherein each field of the event is associated with a different buffer; and

updating a metadata structure that comprises information about contents of the buffers, wherein information about contents of the buffers includes a first minimum value that reflects a minimum value of a first field of all of the events stored in the buffers; and

a processor configured to execute the computer program code stored by the machine-readable medium.

23. A method for processing events, wherein an event includes multiple fields, and wherein a field stores a value, comprising:

receiving a set of events;

generating a row-based chunk that includes the set of events and metadata about the set of events; and

generating a column-based chunk that includes metadata about the set of events and, for each

event in the set of events, a value of a particular field,

wherein the metadata about the set of events includes a first minimum value that reflects a minimum value of a first field over all of the events in the set of events.

24. The method of claim 23, further comprising generating a second column-based chunk that includes metadata about the set of events and, for each event in the set of events, a value of an additional field.

25. The method of claim 23, wherein the row-based chunk is associated with a first retention policy, and wherein the column-based chunk is associated with a different retention policy.

26. A method for processing events, wherein an event includes multiple fields, and wherein a field stores a value, comprising:

receiving a first set of events;

generating a row-based chunk that includes the first set of events and metadata about the first set of events, wherein the metadata about the first set of events includes a first minimum value that reflects a minimum value of a first field over all of the events in the first set of events;

receiving a second set of events; and

generating a column-based chunk that includes metadata about the second set of events and, for each event in the second set of events, a value of a particular field, wherein the metadata about the second set of events includes a second minimum value that reflects a minimum value of a second field over all of the events in the second set of events.

27. The method of claim 26, wherein the metadata about the first set of events further includes a first maximum value that reflects a maximum value of the first field over all of the events in the first set of events.

28. The method of claim 26, wherein the metadata about the first set of events further includes a third minimum value that reflects a minimum value of a third field over all of the events in the first set of events.

29. The method of claim 26, further comprising generating a second column-based chunk that includes metadata about the second set of events and, for each event in the second set of events, a value of an additional field, wherein the metadata about the second set of events includes a second minimum value that reflects a minimum value of a second field over all of the events in the second set of events.

30. A method for searching a set of events according to a search query, wherein an event includes multiple fields, and wherein a field stores a value, and wherein the search query indicates a desired value and one field of the multiple fields, the method comprising:

accessing a first column-based chunk that includes, for each event in the set of events, a value of the indicated field and an associated index location identifier;

identifying a value in the first column-based chunk that matches the desired value;

identifying the index location identifier associated with the identified value;

accessing a second column-based chunk that includes, for each event in the set of events, a table location identifier and an associated index location identifier;

identifying the table location identifier in the second column-based chunk that is associated with the identified index location identifier;

accessing a row-based chunk that includes each event in the set of events; and

identifying the event in the row-based chunk that is associated with the identified table location identifier.

31. A method for selecting an execution strategy for a query, comprising:

estimating a selectivity of the query'"'"'s predicates;

determining a number of fields involved in the query;

responsive to the selectivity being low and the number of columns being low, selecting a column-only strategy; and

responsive to the selectivity being high and the number of columns being high, selecting a row-and-column strategy.