Time series search engine

• US 8,112,425 B2
• Filed: 10/05/2007
• Issued: 02/07/2012
• Est. Priority Date: 10/05/2006
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Baum, Michael J., Carasso, David, Das, Robin K., Greene, Rory, Hall, Brad, Mealy, Nick, Murphy, Brian, Sorkin, Stephen, Stechert, Andre, Swan, Erik M.
• Primary Examiner(s): Lovel, Kimberly
• Assistant Examiner(s): Uddin, Mohammed R
• Application Number:
  US11/868,370
• Publication Number:
  US 20080215546A1
• Time in Patent Office: 1,586 Days
• Field of Search: 707/746, 707/752, 707/753, 707/999.007
• US Class Current: 707/746
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2291 User-Defined Types; Storage management thereof
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Methods and apparatus consistent with the invention provide the ability to organize, index, search, and present time series data based on searches. Time series data are sequences of time stamped records occurring in one or more usually continuous streams, representing some type of activity. In one embodiment, time series data is organized into discrete events with normalized time stamps and the events are indexed by time and keyword. A search is received and relevant event information is retrieved based in whole or in part on the time indexing mechanism, keyword indexing mechanism, or statistical indices calculated at the time of the search.

Claims

1. A computer-implemented method for time searching data, comprising:

receiving time series data streams from an information processing environment that includes a plurality of servers;

organizing the time series data stream for subsequent searching by performing actions, including;

determining at least one domain that corresponds to data in the time series data streams, wherein data in the time series data streams is aggregated based on at least the one determined domain;

time stamping the time series data streams to create at least one time stamped event that includes at least a portion of aggregated data from the time series data streams, wherein the time stamped event represents a statistic and a pattern of behavior;

employing at least one feature of the included aggregated data to determine at least one boundary for each time stamped event;

segmenting each time stamped event into a plurality of segments, wherein a segment is a substring of event text;

time indexing the time stamped events to create time bucketed indices based on the time stamps and segments; and

receiving a time series search request; and

executing the time series search request at least in part by searching the time bucketed indices.

2. The method of claim 1 wherein the step of receiving time series data streams comprises receiving time series data streams temporally out of order.

3. The method of claim 1 wherein the step of receiving time series data streams comprises collecting the time series data streams at more than one physical location.

4. The method of claim 1 wherein the step of time stamping the time series data streams comprises:

aggregating the time series data streams into events; and

time stamping the events.

5. The method of claim 4 wherein the step of time stamping the time series data streams comprises:

classifying the events by domain; and

time stamping the events according to their domain.

6. The method of claim 5 wherein the step of time stamping the events according to their domain comprises extracting the time stamp from the event for events that are classified in a domain with a known time stamp format.

7. The method of claim 5 wherein the step of time stamping the events according to their domain comprises interpolating a time stamp for the event for events that are not classified in a domain with a known time stamp format.

8. The method of claim 1 wherein the step of time indexing the time stamped events to create time bucketed indices comprises assigning the time stamped events to time buckets according to their time stamps.

9. The method of claim 8 wherein at least two of the time buckets have different time durations.

10. The method of claim 8 wherein time buckets are instantiated using a lazy allocation policy.

11. The method of claim 8 wherein the step of time indexing the time stamped events comprises:

creating hot indexes, which are not searchable and do not persist; and

converting hot indexes into warm indexes when the hot indexes are at capacity, the warm indexes being searchable and persistent.

12. The method of claim 11 further comprising merging multiple warm indices within a time bucket to form a new warm index.

13. The method of claim 1 wherein the step of time indexing the time stamped events comprises creating new indices based on segments in completed indices.

14. The method of claim 1 wherein the time series search request is expressed in a time series search language.

15. The method of claim 1 wherein the step of executing the time series search request comprises creating meta events.

16. The method of claim 1 wherein the step of executing the time series search request at least in part by searching the time bucketed indices comprises:

creating sub-searches for different time buckets; and

executing the sub-searches across the corresponding time bucketed indices.

17. The method of claim 1 further comprises displaying search results organized by time.

18. The method of claim 17 wherein the step of displaying search results organized by time comprises zooming the display according to user input.

19. The method of claim 1 wherein the time series data streams comprise server logs.

20. The method of claim 1 wherein the time series data streams comprise transaction records.

21. The method of claim 1 wherein the time series data streams comprise recorded measurements.

22. The method of claim 1 further comprising storing the time-stamped events in a data store, each stored event having an address.

23. The method of claim 22 wherein the indices include a list of segments, a list of event addresses, and associations between the segments and event addresses.

24. The method of claim 1, wherein the segments comprise overlapping segments.

25. An apparatus for a time series search engine for time searching data, comprising:

a memory for storing data and instructions;

a processor for executing instructions that perform actions, including;

receiving time series data streams from an information processing environment that includes a plurality of servers;

organizing the time series data stream for subsequent searching by performing actions, including;

determining at least one domain that corresponds to data in the time series data streams, wherein data in the time series data streams is aggregated based on at least the one determined domain;

time stamping the time series data streams to create at least one time stamped event that includes at least a portion of aggregated data from the time series data streams, wherein the time stamped event represents a statistic and a pattern of behavior;

employing at least one feature of the included aggregated data to determine at least one boundary for each time stamped event;

segmenting each time stamped event into a plurality of segments, wherein a segment is a substring of event text;

time indexing the time stamped events to create time bucketed indices based on the time stamps and segments; and

receiving a time series search request; and

executing the time series search request at least in part by searching the time bucketed indices.

26. The time series search engine of claim 25 wherein the time stamping the time series data streams comprises:

aggregating the time series data streams into events; and

time stamping the events.

27. The time series search engine of claim 26 wherein the time stamping the time series data streams comprises:

classifying the events by domain; and

time stamping the events according to their domain.

28. The time series search engine of claim 25 wherein the time indexing the time stamped events to create time bucketed indices comprises assigning the time stamped events to time buckets according to their time stamps.

29. The time series search engine of claim 28 wherein time buckets are instantiated using a lazy allocation policy.

30. The time series search engine of claim 28 wherein the time indexing the time stamped events comprises:

creating hot indexes, which are not searchable and do not persist; and

converting hot indexes into warm indexes when the hot indexes are at capacity, the warm indexes being searchable and persistent.

31. The time series search engine of claim 25 wherein the executing the time series search request at least in part by searching the time bucketed indices comprises:

creating sub-searches for different time buckets; and

executing the sub-searches across the corresponding time bucketed indices.

32. The time series search engine of claim 25 wherein the time series data streams comprise server logs.

33. A computer program product for time searching data, the computer program product stored on a non-transitory computer readable storage medium and including instructions for causing a computer system to execute a method for time searching data, the method comprising the actions of:

receiving time series data streams from an information processing environment that includes a plurality of servers;

organizing the time series data stream for subsequent searching by performing actions, including;

determining at least one domain that corresponds to data in the time series data streams, wherein data in the time series data streams is aggregated based on at least the one determined domain;

time stamping the time series data streams to create at least one time stamped event that includes at least a portion of aggregated data from the time series data streams, wherein the time stamped event represents a statistic and a pattern of behavior;

employing at least one feature of the included aggregated data to determine at least one boundary for each time stamped event;

segmenting each time stamped event into a plurality of segments, wherein a segment is a substring of event text;

time indexing the time stamped events to create time bucketed indices based on the time stamps and segments; and

receiving a time series search request; and

executing the time series search request at least in part by searching the time bucketed indices.