Compressed journaling in event tracking files for metadata recovery and replication

• US 8,589,403 B2
• Filed: 02/28/2011
• Issued: 11/19/2013
• Est. Priority Date: 02/28/2011
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Marquardt, David Ryan, Blank, Mitchell Neuman Jr., Sorkin, Stephen
• Primary Examiner(s): Trujillo, James
• Assistant Examiner(s): VU, THONG H
• Application Number:
  US13/037,279
• Publication Number:
  US 20120221576A1
• Time in Patent Office: 995 Days
• Field of Search: 707/610, 707/201, 707/646, 707/100, 707/737, 707/692, 715/200, 706/25, 1/1
• US Class Current: 707/741
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs

Abstract

Embodiments are directed towards employing compressed journaling for event tracking files for metadata recovery and replication. Event data and related metadata are received from one or more client devices. When a feature within the received metadata is detected that is previously unwritten to a journal, then the previously unwritten feature is written to the journal. Further, any feature is detected for the received event data that is determined to be different from a feature associated with an immediately preceding event data that is written in the journal, then the detected different feature is identified in the journal. In one embodiment, the identification employs writing to the journal an effective feature record that may employ indices identifying the different feature. The received event data is also written to the journal and may further employ string arguments to minimize recording of redundant information into the journal.

Claims

1. A computer-implemented method, comprising:

receiving raw data on a computing device;

determining boundaries that divide the raw data into a set of events;

assigning a time stamp to each event in the set of events;

identifying a subset of events in the set of events;

compressing the raw data that includes the subset of events;

storing the compressed raw data that includes the subset of events;

determining a compression offset for the compressed raw data that includes the subset of events, wherein the compression offset indicates a location of the compressed raw data that includes the subset of events;

storing the compression offset in a compression index;

associating one or more uncompressed offsets with the compression offset, wherein each uncompressed offset includes information for identifying one of the events in the raw data that includes the subset of events;

receiving an indication to retrieve a particular event in the subset of events;

using the compression index to identify the compression offset indicating the location of the compressed raw data that includes the particular event; and

using the compression offset and an associated uncompressed offset to locate the particular event.

2. The method of claim 1, wherein using the compression offset and the associated uncompressed offset to locate the particular event further comprises:

retrieving the compressed raw data that includes the subset of events containing the particular event from the location specified by the compression offset; and

decompressing the compressed raw data that includes the subset of events containing the particular event.

3. The method of claim 1, wherein using the compression offset and the associated uncompressed offset to locate the particular event further comprises:

retrieving the compressed raw data that includes the subset of events containing the particular event from the location specified by the compression offset;

decompressing the compressed raw data that includes the subset of events containing the particular event; and

using the uncompressed offset to locate the particular event in the raw data that includes the subset of events, wherein the uncompressed offset is applied to the raw data obtained by decompressing the compressed raw data.

4. The method of claim 1, further comprising:

identifying a second subset of events in the set of events;

compressing the raw data associated with the second subset of events;

storing the compressed raw data associated with the second subset of events;

determining a compression offset for the compressed raw data associated with the second subset of events, wherein the compression offset for the compressed raw data associated with the second subset of events indicates a location of the compressed raw data associated with the second subset of events; and

storing the compression offset for the compressed raw data associated with the second subset of events in the compression index.

5. The method of claim 1, further comprising:

storing in a search index a set of keywords and a set of associated uncompressed offsets, wherein each uncompressed offset includes information for identifying an event that contains the associated keyword in the raw data that includes the subset of events;

receiving a selection of a keyword; and

retrieving from the search index an uncompressed offset for identifying an event that contains the selected keyword.

6. The method of claim 1, wherein storing the compressed raw data that includes the subset of events comprises storing the compressed raw data in a journal.

7. A computer-implemented system, comprising:

one or more processors; and

one or more non-transitory computer-readable storage mediums containing instructions configured to cause the one or more processors to perform operations including;

receiving raw data;

determining boundaries that divide the raw data into a set of events;

assigning a time stamp to each event in the set of events;

identifying a subset of events in the set of events;

compressing the raw data that includes the subset of events;

storing the compressed raw data that includes the subset of events;

determining a compression offset for the compressed raw data that includes the subset of events, wherein the compression offset indicates a location of the compressed raw data that includes the subset of events;

storing the compression offset in a compression index;

associating one or more uncompressed offsets with the compression offset, wherein each uncompressed offset includes information for identifying one of the events in the raw data that includes the subset of events;

receiving an indication to retrieve a particular event in the subset of events;

using the compression index to identify the compression offset indicating the location of the compressed raw data that includes the particular event; and

using the compression offset and an associated uncompressed offset to locate the particular event.

8. The system of claim 7, wherein using the compression offset and the associated uncompressed offset to locate the particular event further comprises:

retrieving the compressed raw data that includes the subset of events containing the particular event from the location specified by the compression offset; and

decompressing the compressed raw data that includes the subset of events containing the particular event.

9. The system of claim 7, wherein using the compression offset and the associated uncompressed offset to locate the particular event further comprises:

retrieving the compressed raw data that includes the subset of events containing the particular event from the location specified by the compression offset;

decompressing the compressed raw data that includes the subset of events containing the particular event; and

using the uncompressed offset to locate the particular event in the raw data that includes the subset of events, wherein the uncompressed offset is applied to the raw data obtained by decompressing the compressed raw data.

10. The system of claim 7, wherein the one or more non-transitory computer-readable storage mediums contain further instructions configured to cause the one or more processors to perform operations including:

identifying a second subset of events in the set of events;

compressing the raw data associated with the second subset of events;

storing the compressed raw data associated with the second subset of events;

determining a compression offset for the compressed raw data associated with the second subset of events, wherein the compression offset for the compressed raw data associated with the second subset of events indicates a location of the compressed raw data associated with the second subset of events; and

storing the compression offset for the compressed raw data associated with the second subset of events in the compression index.

11. The system of claim 7, wherein the one or more non-transitory computer-readable storage mediums contain further instructions configured to cause the one or more processors to perform operations including:

storing in a search index a set of keywords and a set of associated uncompressed offsets, wherein each uncompressed offset includes information for identifying an event that contains the associated keyword in the raw data that includes the subset of events;

receiving a selection of a keyword; and

retrieving from the search index an uncompressed offset for identifying an event that contains the selected keyword.

12. The system of claim 7, wherein storing the compressed raw data that includes the subset of events comprises storing the compressed raw data in a journal.

13. A computer-program product, tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause a data processing apparatus to:

receive raw data;

determine boundaries that divide the raw data into a set of events;

assign a time stamp to each event in the set of events;

identify a subset of events in the set of events;

compress the raw data that includes the subset of events;

store the compressed raw data that includes the subset of events;

determine a compression offset for the compressed raw data that includes the subset of events, wherein the compression offset indicates a location of the compressed raw data that includes the subset of events;

store the compression offset in a compression index;

associate one or more uncompressed offsets with the compression offset, wherein each uncompressed offset includes information for identifying one of the events in the raw data that includes the subset of events;

receive an indication to retrieve a particular event in the subset of events;

use the compression index to identify the compression offset indicating the location of the compressed raw data that includes the particular event; and

use the compression offset and an associated uncompressed offset to locate the particular event.

14. The computer-program product of claim 13, wherein using the compression offset and the associated uncompressed offset to locate the particular event further comprises:

retrieving the compressed raw data that includes the subset of events containing the particular event from the location specified by the compression offset; and

decompressing the compressed raw data that includes the subset of events containing the particular event.

15. The computer-program product of claim 13, wherein using the compression offset and the associated uncompressed offset to locate the particular event further comprises:

retrieving the compressed raw data that includes the subset of events containing the particular event from the location specified by the compression offset;

decompressing the compressed raw data that includes the subset of events containing the particular event; and

using the uncompressed offset to locate the particular event in the raw data that includes the subset of events, wherein the uncompressed offset is applied to the raw data obtained by decompressing the compressed raw data.

16. The computer-program product of claim 13, further including instructions configured to cause the data processing apparatus to:

identify a second subset of events in the set of events;

compress the raw data associated with the second subset of events;

store the compressed raw data associated with the second subset of events;

determine a compression offset for the compressed raw data associated with the second subset of events, wherein the compression offset for the compressed raw data associated with the second subset of events indicates a location of the compressed raw data associated with the second subset of events; and

store the compression offset for the compressed raw data associated with the second subset of events in the compression index.

17. The computer-program product of claim 13, further including instructions configured to cause the data processing apparatus to:

store in a search index a set of keywords and a set of associated uncompressed offsets, wherein each uncompressed offset includes information for identifying an event that contains the associated keyword in the raw data that includes the subset of events;

receive a selection of a keyword; and

retrieve from the search index an uncompressed offset for identifying an event that contains the selected keyword.

18. The computer-program product of claim 13, wherein storing the compressed raw data that includes the subset of events comprises storing the compressed raw data in a journal.