Time Series Search Engine

• US 20080215546A1
• Filed: 10/05/2007
• Published: 09/04/2008
• Est. Priority Date: 10/05/2006
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Murphy, Brian, Sorkin, Stephen, Baum, Michael J., Carasso, David, Hall, Brad, Swan, Erik M., Das, Robin K., Stechert, Andre, Greene, Rory, Mealy, Nick
• Granted Patent: US 8,112,425 B2
• Time in Patent Office: Days
• US Class Current: 707/3
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2291 User-Defined Types; Storage management thereof
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Methods and apparatus consistent with the invention provide the ability to organize, index, search, and present time series data based on searches. Time series data are sequences of time stamped records occurring in one or more usually continuous streams, representing some type of activity. In one embodiment, time series data is organized into discrete events with normalized time stamps and the events are indexed by time and keyword. A search is received and relevant event information is retrieved based in whole or in part on the time indexing mechanism, keyword indexing mechanism, or statistical indices calculated at the time of the search.

Claims

1. A computer-implemented method for time searching data, comprising:

receiving time series data streams;

time stamping the time series data streams to create time stamped events;

time indexing the time stamped events to create time bucketed indices;

receiving a time series search request; and

executing the time series search request at least in part by searching the time bucketed indices.

2. The method of claim 1 wherein the step of receiving time series data streams comprises receiving time series data streams temporally out of order.

3. The method of claim 1 wherein the step of receiving time series data streams comprises collecting the time series data streams at more than one physical location.

4. The method of claim 1 wherein the step of time stamping the time series data streams comprises:

aggregating the time series data streams into events; and

time stamping the events.

5. The method of claim 4 wherein the step of time stamping the time series data streams comprises:

classifying the events by domain; and

time stamping the events according to their domain.

6. The method of claim 5 wherein the step of time stamping the events according to their domain comprises extracting the time stamp from the event for events that are classified in a domain with a known time stamp format.

7. The method of claim 5 wherein the step of time stamping the events according to their domain comprises interpolating a time stamp for the event for events that are not classified in a domain with a known time stamp format.

8. The method of claim 1 wherein the step of time indexing the time stamped events to create time bucketed indices comprises assigning the time stamped events to time buckets according to their time stamps.

9. The method of claim 8 wherein at least two of the time buckets have different time durations.

10. The method of claim 8 wherein time buckets are instantiated using a lazy allocation policy.

11. The method of claim 8 wherein the step of time indexing the time stamped events to create time bucketed indices further comprises segmenting the time stamped events.

12. The method of claim 11 wherein the step of time indexing the time stamped events to create time bucketed indices further comprises creating time bucketed indices based on the time stamps and segments.

13. The method of claim 8 wherein the step of time indexing the time stamped events comprises:

creating hot indexes, which are not searchable and do not persist;

creating warm indexes, which are searchable and persistent but are immutable; and

from time to time, converting hot indexes to warm indexes.

14. The method of claim 8 wherein the step of time indexing the time stamped events comprises speculatively indexing.

15. The method of claim 1 wherein the time series search request is expressed in a time series search language.

16. The method of claim 1 wherein the step of executing the time series search request comprises creating meta events.

17. The method of claim 1 wherein the step of executing the time series search request at least in part by searching the time bucketed indices comprises:

creating sub-searches for different time buckets; and

executing the sub-searches across the corresponding time bucketed indices.

18. The method of claim 1 further comprising:

displaying search results organized by time.

19. The method of claim 18 wherein the step of displaying search results organized by time comprises zooming the display according to user input:

20. The method of claim 1 wherein the time series data streams comprise server logs.

21. The method of claim 1 wherein the time series data streams comprise transaction records.

22. The method of claim 1 wherein the time series data streams comprise recorded measurements.

23. A time series search engine for time searching data, comprising:

means for receiving time series data streams;

means for time stamping the time series data streams to create time stamped events;

means for time indexing the time stamped events to create time bucketed indices;

means for receiving a time series search request; and

means for executing the time series search request at least in part by searching the time bucketed indices.

24. The time series search engine of claim 23 wherein the means for time stamping the time series data streams comprises:

means for aggregating the time series data streams into events; and

means for time stamping the events.

25. The time series search engine of claim 24 wherein the means for time stamping the time series data streams comprises:

means for classifying the events by domain; and

means for time stamping the events according to their domain.

26. The time series search engine of claim 23 wherein the means for time indexing the time stamped events to create time bucketed indices comprises means for assigning the time stamped events to time buckets according to their time stamps.

27. The time series search engine of claim 26 wherein time buckets are instantiated using a lazy allocation policy.

28. The time series search engine of claim 26 wherein the means for time indexing the time stamped events to create time bucketed indices further comprises means for segmenting the time stamped events.

29. The time series search engine of claim 28 wherein the means for time indexing the time stamped events to create time bucketed indices further comprises means for creating time bucketed indices based on the time stamps and segments.

30. The time series search engine of claim 26 wherein the means for time indexing the time stamped events comprises:

means for creating hot indexes, which are not searchable and do not persist;

means for creating warm indexes, which are searchable and persistent but are immutable; and

means for, from time to time, converting hot indexes to warm indexes.

31. The time series search engine of claim 23 wherein the means for executing the time series search request at least in part by searching the time bucketed indices comprises:

means for creating sub-searches for different time buckets; and

means for executing the sub-searches across the corresponding time bucketed indices.

32. The time series search engine of claim 23 wherein the time series data streams comprise server logs.

33. A computer readable medium storing instructions for causing a computer system to execute a method for time searching data, the method comprising:

receiving time series data streams;

time stamping the time series data streams to create time stamped events;

time indexing the time stamped events to create time bucketed indices;

receiving a time series search request; and

executing the time series search request at least in part by searching the time bucketed indices.