Time series search engine

• US 8,990,184 B2
• Filed: 10/30/2012
• Issued: 03/24/2015
• Est. Priority Date: 10/05/2006
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Baum, Michael Joseph, Carasso, R. David, Das, Robin Kumar, Greene, Rory, Hall, Bradley, Mealy, Nicholas Christian, Murphy, Brian Philip, Sorkin, Stephen Phillip, Stechert, Andre David, Swan, Erik M.
• Primary Examiner(s): Uddin, Mohammed R
• Application Number:
  US13/664,186
• Publication Number:
  US 20130060783A1
• Time in Patent Office: 875 Days
• Field of Search: 707/746, 707/750, 707/754, 707/999.007
• US Class Current: 707/711
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2291 User-Defined Types; Storage management thereof
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Methods and apparatus consistent with the invention provide the ability to organize, index, search, and present time series data based on searches. Time series data are sequences of time stamped records occurring in one or more usually continuous streams, representing some type of activity. In one embodiment, time series data is organized into discrete events with normalized time stamps and the events are indexed by time and keyword. A search is received and relevant event information is retrieved based in whole or in part on the time indexing mechanism, keyword indexing mechanism, or statistical indices calculated at the time of the search.

Claims

1. A method for building a searchable data store, comprising:

repeatedly generating buckets in memory to receive events,

respective buckets designated with time spans to receive the events based on event time stamps, and

maintaining a record of designated time spans assigned to respective buckets for time keyed event retrieval;

electronically receiving machine data produced by devices in an information processing environment, separating the machine data into events at boundaries between contiguous portions of the machine data, applying the time stamps to the events, and inserting a respective event into a respective bucket based at least in part on a respective applied time stamp;

wherein the machine data included in the events includes textual data;

wherein the events are indexed;

wherein applying the respective time stamp to the respective event includes applying an extraction rule to extract time information to use as the time stamp from the textual data included in the respective event;

wherein the respective bucket receiving the respective event is a hot bucket; and

advancing a filled hot bucket to warm bucket status that does not accept further events;

wherein at least events in warm buckets are searchable.

2. The method of claim 1, further including, for an event lacking extractable time information, interpolating time information to use as a time stamp for the event from known times extracted from portions of the machine data surrounding the machine data included in the event.

3. The method of claim 1, farther comprising indexing events in a warm bucket.

4. The method of claim 1, further comprising creating a speculative index for events in a hot bucket.

5. The method of claim 1, further comprising associating keywords with events in a warm bucket.

6. The method of claim 1, further comprising associating keywords with events in a hot bucket.

7. A system that builds a searchable data store, comprising:

a processor and memory coupled to the processor, the memory storing program instructions that when executed cause;

repeatedly generating buckets in memory to receive events,

respective buckets designated with time spans to receive the events based on event time stamps, and

maintaining a record of designated time spans assigned to respective buckets for time keyed event retrieval;

electronically receiving machine data produced by devices in an information processing environment, separating the machine data into events at boundaries between contiguous portions of the machine data, applying the time stamps to the events, and inserting a respective event into a respective bucket based at least in part on a respective applied time stamp;

wherein the machine data included in the events includes textual data;

wherein the events are indexed;

wherein applying the respective time stamp to the respective event includes applying an extraction rule to extract time information to use as the time stamp from the textual data included in the respective event;

wherein the respective bucket receiving the respective event is a hot bucket; and

advancing a filled hot bucket to warm bucket status that does not accept further events;

wherein at least events in warm buckets are searchable.

8. The system of claim 7, further including, for an event lacking extractable time information, interpolating time information to use as a time stamp for the event from known times extracted from portions of the machine data surrounding the machine data included in the event.

9. The system of claim 7, further comprising indexing events in a warm bucket.

10. The system of claim 7, further comprising creating a speculative index for events in a hot bucket.

11. The system of claim 7, further comprising associating keywords with events in a warm bucket.

12. The system of claim 7, further comprising associating keywords with events in a hot bucket.

13. A computer program product including memory that stores program instructions that, when executed on a computer, cause:

repeatedly generating buckets in memory to receive events,

respective buckets designated with time spans to receive the events based on event time stamps, and

maintaining a record of designated time spans assigned to respective buckets for time keyed event retrieval;

electronically receiving machine data produced by devices in an information processing environment, separating the machine data into events at boundaries between contiguous portions of the machine data, applying the time stamps to the events, and inserting a respective event into a respective bucket based at least in part on a respective applied time stamp;

wherein the machine data included in the events includes textual data;

wherein the events are indexed;

wherein applying the respective time stamp to the respective event includes applying an extraction rule to extract time information to use as the time stamp from the textual data included in the respective event;

wherein the respective bucket receiving the respective event is a hot bucket; and

advancing a filled hot bucket to warm bucket status that does not accept further events;

wherein at least events in warm buckets are searchable.

14. The computer program product of claim 13, further comprising instructions to cause, for an event lacking extractable time information, interpolating time infbrmation to use as a time stamp for the event from known times extracted from portions of the machine data surrounding the machine data included in the event.

15. The computer program product of claim 13, further comprising instructions to cause indexing events in a warm bucket.

16. The computer program product of claim 13, further comprising instructions to cause creating a speculative index for events in a hot bucket.

17. The computer program product of claim 13, further comprising instructions to cause associating keywords with events in a warm bucket.

18. The computer program product claim 13, further comprising instructions to cause associating keywords with events in a hot bucket.