Time series search with interpolated time stamp

• US 9,002,854 B2
• Filed: 01/18/2012
• Issued: 04/07/2015
• Est. Priority Date: 10/05/2006
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Baum, Michael J., Carasso, David, Das, Robin K., Greene, Rory, Hall, Brad, Mealy, Nick, Murphy, Brian, Sorkin, Stephen, Stechert, Andre, Swan, Erik M.
• Primary Examiner(s): Uddin, Mohammed R
• Application Number:
  US13/353,135
• Publication Number:
  US 20120117079A1
• Time in Patent Office: 1,175 Days
• Field of Search: 707/746, 707/750, 707/754, 707/999.007
• US Class Current: 707/746
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2291 User-Defined Types; Storage management thereof
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Methods and apparatus consistent with the invention provide the ability to organize, index, search, and present time series data based on searches. Time series data are sequences of time stamped records occurring in one or more usually continuous streams, representing some type of activity. In one embodiment, time series data is organized into discrete events with normalized time stamps and the events are indexed by time and keyword. A search is received and relevant event information is retrieved based in whole or in part on the time indexing mechanism, keyword indexing mechanism, or statistical indices calculated at the time of the search.

Claims

1. A computer-implemented method for searching data, the method comprising:

gathering, using a computing device, a stream of data from an information processing environment;

separating the stream of data into a plurality of events, each event including a respective portion of the stream of data;

for each event of the plurality of events, determining an associated time stamp representing a time for the event;

wherein determining the associated time stamp for an event of the plurality of events comprises time information to use in the time stamp from known times corresponding to portions of the stream of data surrounding the portion of the stream of data included in the event;

assigning each event of the plurality of events to a bucket having an associated time range that includes the time represented by the time stamp for the event;

receiving a search query that includes a time criterion and a second criterion for selection of events, the second criterion relating to a segment within the events, the segment identified by an extraction rule for extracting a subportion of data from the portion of the stream of data included in an event, the extraction rule using a pattern to identify boundaries of the subportion of data being extracted;

identifying one or more buckets that each have an associated time range, at least one time in each associated time range satisfying the time criterion;

based on examining events only in the identified one or more buckets, identifying a set of events that match the time criterion and have a segment that matches the second criterion;

determining a result based on the set of events; and

causing the result to be displayed.

2. The method of claim 1, wherein the extraction rule comprises a regular expression.

3. The method of claim 1, wherein the extraction rule is based on character delimiters.

4. The method of claim 1, wherein the extraction rule is based on punctuation delimiters.

5. The method of claim 1, wherein gathering the stream of data comprises employing a script to collect the stream of data and deliver it to a computing device configured for separating the stream of data into the plurality of events.

6. The method of claim 1, wherein causing the result to be displayed comprises causing display of the set of events that match the time criterion and have the segment that matches the second criterion.

7. The method of claim 1, wherein the result determined from the set of events includes a statistic or a pattern of behavior.

8. The method of claim 1, further comprising determining a rule used for separating the stream of data into the plurality of events, wherein the determination of the rule is based on one or more of:

a source of the stream of data, a domain of the stream of data, a type of the stream of data, a signature of the stream of data, a punctuation analysis of the stream of data, or an analysis of repeating patterns within the stream of data.

9. The method of claim 1, wherein determining the associated time stamp for an event of the plurality of events comprises extracting time information from the portion of the stream of data included in the event to use in the time stamp.

10. The method of claim 1, wherein determining the associated time stamp for an event of the plurality of events comprises using in the time stamp a time of receipt for the portion of the stream of data that is included in the event.

11. The method of claim 1, wherein the stream of data includes a server log, a portion of a log file, a transaction record, a recorded measurement, message bus traffic, network data, network management data, an output of an application, or an output of a technology.

12. The method of claim 1, wherein multiple buckets are associated with time ranges that include at least one time that satisfies the time criterion, and wherein the method further comprises:

determining a sequence to search the multiple buckets;

determining that a current set of events identified as meeting the time criterion is sufficient before searching all of the multiple buckets; and

terminating searching the multiple buckets prior to completing the sequence.

13. The method of claim 1, wherein the result includes a count of a number of events for which both the time criterion and the second criterion is satisfied.

14. A system for searching time series data, the system comprising:

at least one network interface that enables communication over the network;

one or more memories that store instructions;

one or more processors that execute the instructions to enable performance of actions, including;

gathering, using a computing device, a stream of data from an information processing environment;

separating the stream of data into a plurality of events, each event including a respective portion of the stream of data;

for each event of the plurality of events, determining an associated time stamp representing a time for the event;

wherein determining the associated time stamp for an event of the plurality of events comprises interpolatin time information to use in the time stamp from known times corresponding to portions of the stream of data surrounding the portion of the stream of data included in the event;

assigning each event of the plurality of events to a bucket having an associated time range that includes the time represented by the time stamp for the event;

receiving a search query that includes a time criterion and a second criterion for selection of events, the second criterion relating to a segment within the events, the segment identified by an extraction rule for extracting a subportion of data from the portion of the stream of data included in an event, the extraction rule using a pattern to identify boundaries of the subportion of data being extracted;

identifying one or more buckets that each have an associated time range, at least one time in each associated time range satisfying the time criterion;

based on examining events only in the identified one or more buckets, identifying a set of events that match the time criterion and have a segment that matches the second criterion;

determining a result based on the set of events; and

causing the result to be displayed.

15. The system of claim 14, wherein the extraction rule comprises a regular expression.

16. The system of claim 14, wherein the extraction rule is based on character delimiters.

17. The system of claim 14, wherein the extraction rule is based on punctuation delimiters.

18. The system of claim 14, wherein gathering the stream of data comprises employing a script to collect the stream of data and deliver it to a computing device configured for separating the stream of data into the plurality of events.

19. The system of claim 14, wherein causing the result to be displayed comprises causing display of the set of events that match the time criterion and have the segment that matches the second criterion.

20. The system of claim 14, wherein the result determined from the set of events includes a statistic or a pattern of behavior.

21. A processor readable non-volatile storage media that includes instructions for searching data, wherein execution of the instructions by one or more processors enable actions, the actions comprising:

gathering, using a computing device, a stream of data from an information processing environment;

separating the stream of data into a plurality of events, each event including a respective portion of the stream of data;

for each event of the plurality of events, determining an associated time stamp representing a time for the event;

wherein determining the associated time stamp for an event of the plurality of events comprises time information to use in the time stamp from known times corresponding to portions of the stream of data surrounding the portion of the stream of data included in the event;

assigning each event of the plurality of events to a bucket having an associated time range that includes the time represented by the time stamp for the event;

receiving a search query that includes a time criterion and a second criterion for selection of events, the second criterion relating to a segment within the events, the segment identified by an extraction rule for extracting a subportion of data from the portion of the stream of data included in an event, the extraction rule using a pattern to identify boundaries of the subportion of data being extracted;

identifying one or more buckets that each have an associated time range, at least one time in each associated time range satisfying the time criterion;

based on examining events only in the identified one or more buckets, identifying a set of events that match the time criterion and have a segment that matches the second criterion;

determining a result based on the set of events; and

causing the result to be displayed.

22. The media of claim 21, wherein the extraction rule comprises a regular expression.

23. The media of claim 21, wherein the extraction rule is based on character delimiters.

24. The media of claim 21, wherein the extraction rule is based on punctuation delimiters.

25. The media of claim 21, wherein gathering the stream of data comprises employing a script to collect the stream of data and deliver it to a computing device configured for separating the stream of data into the plurality of events.

26. The media of claim 21, wherein causing the result to be displayed comprises causing display of the set of events that match the time criterion and have the segment that matches the second criterion.