Time series search in primary and secondary memory

• US 9,594,789 B2
• Filed: 01/30/2015
• Issued: 03/14/2017
• Est. Priority Date: 10/05/2006
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Baum, Michael Joseph, Carasso, R. David, Das, Robin Kumar, Greene, Rory, Hall, Bradley, Mealy, Nicholas Christian, Murphy, Brian Philip, Sorkin, Stephen Phillip, Stechert, Andre David, Swan, Erik M.
• Primary Examiner(s): Uddin, Mohammed R
• Application Number:
  US14/611,170
• Publication Number:
  US 20150149480A1
• Time in Patent Office: 774 Days
• Field of Search: 707/746, 707/750, 707/754, 707/752, 707/753, 707/999.007
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2291 User-Defined Types; Storage management thereof
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Methods and apparatus consistent with the invention provide the ability to organize, index, search, and present time series data based on searches. Time series data are sequences of time stamped records occurring in one or more usually continuous streams, representing some type of activity. In one embodiment, time series data is organized into discrete events with normalized time stamps and the events are indexed by time and keyword. A search is received and relevant event information is retrieved based in whole or in part on the time indexing mechanism, keyword indexing mechanism, or statistical indices calculated at the time of the search.

Claims

1. A method for building a searchable data store, comprising:

electronically receiving machine data produced by devices in an information processing environment;

segmenting the machine data into a plurality of events by determining a beginning and ending of each event in the plurality of events in the machine data;

associating a time stamp with each event in the plurality of events by applying an extraction rule to the machine data segmented for each event in order to extract time information to use as the time stamp for that event;

repeatedly generating buckets in volatile, random access memory;

designating a time span for each bucket of the generated buckets;

inserting a time stamped event in the plurality of events into a particular bucket in the generated buckets based at least in part on the associated time stamp and the time span of the particular bucket;

advancing a full bucket to full bucket status that does not accept further events;

transferring the full bucket into non-volatile storage, the full bucket is available for searching;

determining that a full bucket in non-volatile storage has expired; and

based on determining that the full bucket has expired, moving the full bucket out of active status.

2. The method of claim 1, further comprising indexing events in the particular bucket.

3. The method of claim 1, further comprising creating a speculative index for the events in at least one of the generated buckets based on the full bucket.

4. The method of claim 1, further comprising indexing keywords in the events in the particular bucket.

5. The method of claim 1, further comprising indexing keywords in the events in the full bucket.

6. A system that builds a searchable data store, comprising:

a processor and memory coupled to the processor, the memory storing program instructions that when executed cause;

electronically receiving machine data produced by devices in an information processing environment;

segmenting the machine data into a plurality of events by determining a beginning and ending of each event in the plurality of events in the machine data;

associating a time stamp with each event in the plurality of events by applying an extraction rule to the machine data segmented for each event in order to extract time information to use as the time stamp for that event;

repeatedly generating buckets in volatile, random access memory;

designating a time span for each bucket of the generated buckets;

inserting a time stamped event in the plurality of events into a particular bucket in the generated buckets based at least in part on the associated time stamp and the time span of the particular bucket;

advancing a full bucket to full bucket status that does not accept further events;

transferring the full bucket into non-volatile storage, the full bucket is available for searching;

determining that a full bucket in non-volatile storage has expired; and

based on determining that the full bucket has expired, moving the full bucket out of active status.

7. The system of claim 6, further configured to index events in the particular bucket.

8. The system of claim 6, further configured to create a speculative index for the events in at least one of the generated buckets based on the full bucket.

9. The system of claim 6, further configured to index keywords in the events in the particular bucket.

10. The system of claim 6, further configured to index keywords in the events in the full bucket.

11. A computer program product including memory that stores program instructions that, when executed on a computer, cause:

electronically receiving machine data produced by devices in an information processing environment;

segmenting the machine data into a plurality of events by determining a beginning and ending of each event in the plurality of events in the machine data;

associating a time stamp with each event in the plurality of events by applying an extraction rule to the machine data segmented for each event in order to extract time information to use as the time stamp for that event;

repeatedly generating buckets in volatile, random access memory;

designating a time span for each bucket of the generated buckets;

inserting a time stamped event in the plurality of events into a particular bucket in the generated buckets based at least in part on the associated time stamp and the time span of the particular bucket;

advancing a full bucket to full bucket status that does not accept further events;

transferring the full bucket into non-volatile storage, the full bucket is available for searching;

determining that a full bucket in non-volatile storage has expired; and

based on determining that the full bucket has expired, moving the full bucket out of active status.

12. The computer program product of claim 11, further configured to index events in the particular bucket.

13. The computer program product of claim 11, further configured to create a speculative index for the events in at least one of the generated buckets based on the full bucket.

14. The computer program product of claim 11, further configured to index keywords in the events in the particular bucket.

15. The computer program product of claim 11, further configured to index keywords in the events in the full bucket.