Systems and methods for concurrent summarization of indexed data

• US 10,229,150 B2
• Filed: 04/23/2015
• Issued: 03/12/2019
• Est. Priority Date: 04/23/2015
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Marquardt, David, Wang, Xiaowei
• Primary Examiner(s): Truong, Dennis
• Application Number:
  US14/694,797
• Publication Number:
  US 20160314163A1
• Time in Patent Office: 1,419 Days
• Field of Search: 707737, 707704, 718102, 718103, 718107
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2343 Locking methods, e.g. distributed locking or locking implementation details
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/245 Query processing

Abstract

Provided are systems and methods for concurrent summarization of indexed data. In some embodiments, two or more summary processes can be executed concurrently (e.g., in parallel) by an indexer to generate summaries for respective subsets of indexed data (e.g., partitions or buckets of indexed data) managed by the indexer.

Claims

1. A method comprising:

receiving raw machine-generated data;

segmenting the raw machine-generated data into a set of time-stamped event records;

indexing and storing the set of time-stamped event records in two or more partitions of event records;

receiving a first summary request;

executing, in response to the receiving the first summary request, a first process comprising writing, to a first summary directory, first summary data for a first partition of the two or more partitions;

receiving a second summary request; and

executing, in response to the receiving the second summary request, a second process comprising;

determining that the first summary directory is locked to inhibit processes other than the first process from writing summary data to the first summary directory; and

in response to the determining that the first summary directory is locked;

identifying a second partition of the two or more partitions;

locking a second summary directory to inhibit processes other than the second process from writing summary data to the second summary directory;

determining second summary data for the second partition;

writing the second summary data to the second summary directory, wherein the second summary data comprises an indication that an entirety of contents of the second partition is summarized and a summary of data in the second partition; and

unlocking the second summary directory.

2. The method of claim 1, wherein the first process further comprises:

identifying the first partition of the two or more partitions;

locking the first summary directory to inhibit processes other than the first process

from writing summary data to the first summary directory;

determining first summary data for the first partition;

writing the first summary data to the first summary directory; and

unlocking the first summary directory.

3. The method of claim 1, wherein the first process and the second process are executed in parallel.

4. The method of claim 1, wherein the two or more partitions are stored in a memory of an indexer.

5. The method of claim 1, wherein the two or more partitions, the first summary directory and the second summary directory are stored in a memory of an indexer.

6. The method of claim 1, wherein each of the two or more partitions is associated with a timespan and comprises time-stamped event records each having a respective timestamp corresponding to a respective time in the timespan.

7. The method of claim 1, wherein the two or more partitions and the first summary directory and the second summary directory are stored in a memory of an indexer, and wherein the first and second summary requests are received by the indexer from one or more search heads.

8. The method of claim 1, wherein the two or more partitions and the first summary directory and the second summary directory are stored in a memory of an indexer, and wherein the first process and the second process are executed in parallel by one or more processors of the indexer.

9. The method of claim 1, wherein the first and second summary request each comprise a scheduled request for summarizing the indexed event records stored in the two or more partitions.

10. The method of claim 1 further comprising determining that less than a given number of processes are currently being executed, and wherein at least one of the first summary request and the second summary request is generated in response to the determining that less than a the given number of processes are currently being executed.

11. The method of claim 1 further comprising determining that at least a given amount of time has passed since initiating execution of a prior summary request, wherein at least one of the first summary request and the second summary request is generated in response to the determining that at least a given amount of time has passed since initiating execution of a prior summary request.

12. The method of claim 1, further comprising:

receiving, from an entity, a request for summary data; and

providing, to the entity, the first summary data and the second summary data,

wherein the entity is configured to generate a result based at least in part on the contents of the first summary data and the contents of the second summary data.

13. The method of claim 1, further comprising performing report acceleration based at least in part on the contents of the first summary data and the contents of the second summary data.

14. The method of claim 1, wherein the first summary data comprises one or more first pre-computed aggregates, wherein the second summary data comprises one or more second pre-computed aggregates, and further comprising:

receiving, from an entity, a request for summary data; and

providing, to the entity, the first summary data and the second summary data,

wherein the entity is configured to generate a report comprising an aggregate determined based at least in part on the one or more first pre-computed aggregates and the one or more second pre-computed aggregates.

15. The method of claim 1, further comprising performing data model acceleration based at least in part on the contents of the first summary data and the contents of the second summary data.

16. The method of claim 1, wherein the first summary data comprises a first subset of values of fields of events corresponding to a data model, wherein the second summary data comprises a second subset of values of fields of events corresponding to the data model, and further comprising:

receiving, from an entity, a request for summary data; and

providing, to the entity, the first summary data and the second summary data, wherein the entity is configured to generate a set of values fields of events for the data model determined based at least in part on the first subset of values of fields and the second subset of values of fields.

17. The method of claim 1, further comprising:

receiving a search request;

generating search results based at least in part on the first summary data and the second summary data; and

providing the search results in response to the search request.

18. The method of claim 1, wherein the first summary directory comprises a first summary file.

19. The method of claim 1, wherein the second summary directory comprises a second summary file.

20. The method of claim 1, wherein the second process further comprises determining that the second summary directory is unlocked.

21. The method of claim 1, wherein the second process further comprises:

determining that the second summary directory does not exist; and

creating the second summary directory.

22. The method of claim 1, wherein the first process further comprises:

identifying a third partition of the two or more partitions;

determining that the third summary directory is unlocked;

locking a third summary directory to inhibit processes other than the first process from writing summary data to the third summary directory;

determining third summary data for the third partition;

writing the third summary data to the third summary directory; and

unlocking the third summary directory.

23. The method of claim 1, further comprising executing, in parallel with the first process, a third process comprising writing, to a third summary directory, third summary data for the first partition.

24. The method of claim 1, wherein the first process further comprises:

identifying a third partition of the two or more partitions;

determining that third summary data of a third summary directory associated with the third partition indicates that summarization of data in the third partition is complete; and

in response to the determining that the third summary data indicates that summarization of the data in the third partition is complete, not writing summary data to the third summary directory.

25. The method of claim 1, wherein the first process further comprises:

identifying a third partition of the two or more partitions;

determining that third summary data of a third summary directory associated with the third partition indicates that summarization of a portion of the data in the third partition is incomplete; and

in response to the determining that the third summary data indicates that summarization of a portion of the data in the third partition is incomplete;

locking the third summary directory to inhibit processes other than the first process from writing summary data to the third summary directory;

determining third summary data for the portion of the data in the third partition;

writing the third summary data to the third summary directory; and

unlocking the third summary directory.

26. The method of claim 1, wherein the first summary data comprises an indication that less than an entirety of the contents of the first partition is summarized.

27. The method of claim 1, further comprising searching the set of time-stamped event records using a late binding schema.

28. A system comprising:

one or more processors; and

one or more memories comprising program instructions stored thereon that are executable by the one or more processors to cause the system to perform a method comprising;

receiving, at a computing device, raw machine-generated data;

segmenting the raw machine-generated data into a set of time-stamped event records;

indexing and storing the set of time-stamped event records in two or more partitions of event records;

receiving a first summary request;

executing, in response to the receiving a first summary request, a first process comprising writing, to a first summary directory, first summary data for a first partition of the two or more partitions;

receiving a second summary request; and

executing, in response to the receiving the second summary request, a second process comprising;

determining that the first summary directory is locked to inhibit processes other than the first process from writing summary data to the first summary directory; and

in response to determining that the first summary directory is locked;

identifying a second partition of the two or more partitions;

locking a second summary directory to inhibit processes other than the second process from writing summary data to the second summary directory;

determining second summary data for the second partition;

writing the second summary data to the second summary directory,

wherein the second summary data comprises an indication that an entirety of contents of the second partition is summarized and a summary of data in the second partition; and

unlocking the second summary directory.

29. One or more non-transitory computer-readable medium comprising program instructions stored thereon that are executable by one or more processors to implement a method comprising:

receiving, at a computing device, raw machine-generated data;

segmenting the raw machine-generated data into a set of time-stamped event records;

indexing and storing the set of time-stamped event records in two or more partitions of event records;

receiving a first summary request; and

executing, in response to the receiving a first summary request, a first process comprising writing, to a first summary directory, first summary data for a first partition of the two or more partitions;

receiving a second summary request;

executing, in response to the receiving the second summary request, a second process comprising;

determining that the first summary directory is locked to inhibit processes other than the first process from writing summary data to the first summary directory; and

in response to the determining that the first summary directory is locked;

identifying a second partition of the two or more partitions;

locking a second summary directory to inhibit processes other than the second process from writing summary data to the second summary directory;

determining second summary data for the second partition;

writing the second summary data to the second summary directory,

wherein the second summary data comprises an indication that an entirety of contents of the second partition is summarized and a summary of data in the second partition; and

unlocking the second summary directory.