STORING LOG DATA EFFICIENTLY WHILE SUPPORTING QUERYING TO ASSIST IN COMPUTER NETWORK SECURITY

• US 20080162592A1
• Filed: 12/28/2007
• Published: 07/03/2008
• Est. Priority Date: 12/28/2006
• Status: Active Grant

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: Arcsight Incorporated (HP Inc.)
• Inventors: Huang, Wei, Tang, Wenting, Beedgen, Christian F.
• Granted Patent: US 9,031,916 B2
• Time in Patent Office: Days
• US Class Current: 707/202
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 2201/835 Timestamp
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: H04L 41/0686 Additional information in the notification, e.g. enhancement of specific meta-data
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

A logging system includes an event receiver and a storage manager. The receiver receives log data, processes it, and outputs a data “chunk.” The manager receives data chunks and stores them so that they can be queried. The receiver includes buffers that store events and a metadata structure that stores metadata about the contents of the buffers. The metadata includes a unique identifier associated with the receiver, the number of events in the buffers, and, for each “field of interest,” a minimum value and a maximum value that reflect the range of values of that field over all of the events in the buffers. A chunk includes the metadata structure and a compressed version of the contents of the buffers. The metadata structure acts as a search index when querying event data. The logging system can be used in conjunction with a security information/event management (SIEM) system.

Claims

1. A method for processing log data, comprising:

receiving log data that comprises a plurality of events, an event including one or more fields; and

for each event in the plurality of events;

storing, in a buffer, the event; and

updating a metadata structure that comprises information about contents of the buffer, wherein information about contents of the buffer includes a first minimum value that reflects a minimum value of a first field of all of the events stored in the buffer.

2. The method of claim 1, wherein information about contents of the buffer further includes a first maximum value that reflects a maximum value of the first field of all of the events stored in the buffer.

3. The method of claim 1, wherein information about contents of the buffer further includes a second minimum value that reflects a minimum value of a second field of all of the events stored in the buffer.

4. The method of claim 1, wherein information about contents of the buffer further includes a number of events stored in the buffer.

5. The method of claim 1, further comprising:

in response to a first trigger condition, generating a data chunk based on contents of the metadata structure and further based on contents of the buffer

6. The method of claim 5, wherein the first trigger condition is based on a buffer usage threshold or based on a timeout window.

7. The method of claim 5, further comprising:

in response to a second trigger condition, reclaiming storage space used by the data chunk.

8. The method of claim 7, wherein the second trigger condition is based on a retention policy associated with the data chunk.

9. The method of claim 7, wherein the second trigger condition is based on a disk-space usage threshold or based on a maximum time to retain the chunk.

10. The method of claim 1, further comprising:

for each event in the plurality of events;

determining when the event was received; and

storing, in the buffer, a timestamp that reflects when the event was received.

11. The method of claim 1, wherein storing the event in the buffer comprises appending the event to content of the buffer.

12. The method of claim 1, further comprising generating a data chunk that includes contents of the metadata structure and a compressed version of contents of the buffer.

13. The method of claim 12, wherein the data chunk further includes a file signature or a version identifier.

14. The method of claim 12, wherein the data chunk further includes a message digest of contents of the buffer.

15. The method of claim 12, further comprising:

receiving a search query that includes a set of one or more search terms;

identifying one or more search terms, from the set of search terms, that concern information that is contained in the metadata structure; and

searching one or more data chunks by comparing, for each data chunk, the identified search terms to contents of the metadata structure included within the data chunk.

16. The method of claim 15, further comprising:

for each data chunk that satisfies the identified search terms;

disassembling the data chunk into a plurality of events; and

comparing, for each event in the plurality of events, the set of search terms to the event.

17. A computer program product for processing log data, the computer program product comprising a computer-readable medium containing computer program code for performing a method, the method comprising:

receiving log data that comprises a plurality of events, an event including one or more fields; and

for each event in the plurality of events;

storing, in a buffer, the event; and

updating a metadata structure that comprises information about contents of the buffer, wherein information about contents of the buffer includes a first minimum value that reflects a minimum value of a first field of all of the events stored in the buffer.

18. An apparatus for processing log data, comprising:

a receiving module configured to receive log data that comprises a plurality of events, an event including one or more fields;

a buffer module configured to store, for each event in the plurality of events, the event in a buffer; and

a metadata module configured to update, for each event in the plurality of events, a metadata structure that comprises information about contents of the buffer, wherein information about contents of the buffer includes a first minimum value that reflects a minimum value of a first field of all of the events stored in the buffer.