APPLICATION OF SEARCH POLICIES TO SEARCHES ON EVENT DATA STORED IN PERSISTENT DATA STRUCTURES

• US 20170046403A1
• Filed: 11/01/2016
• Published: 02/16/2017
• Est. Priority Date: 10/05/2006
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Swan, Erik M., Carasso, R. David, Das, Robin Kumar, Greene, Rory, Hall, Bradley, Mealy, Nicholas Christian, Murphy, Brian Philip, Sorkin, Stephen Phillip, Stechert, Andre David, Baum, Michael Joseph
• Granted Patent: US 10,262,018 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2291 User-Defined Types; Storage management thereof
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Methods and apparatus consistent with the invention provide the ability to organize, index, search, and present time series data based on searches. Time series data are sequences of time stamped records occurring in one or more usually continuous streams, representing some type of activity. In one embodiment, time series data is organized into discrete events with normalized time stamps and the events are indexed by time and keyword. A search is received and relevant event information is retrieved based in whole or in part on the time indexing mechanism, keyword indexing mechanism, or statistical indices calculated at the time of the search.

Claims

1. A method, comprising:

receiving raw data from a plurality of sources;

creating a plurality of events based on the raw data;

associating a time stamp with each event in the plurality of events;

indexing each time stamped event in the plurality of events;

creating persistent data structures for storing the plurality of events, wherein each persistent data structure corresponds to a specific time interval, wherein events stored in a particular persistent data structure have associated time stamps that fall within a particular time interval corresponding to the particular persistent data structure;

searching events in one or more persistent data structures according to a received search query, wherein the searching the events is halted upon satisfying one or more search policies.

2. The method of claim 1, wherein the one or more search policies indicate a maximum number of events to be returned in response to a search query.

3. The method of claim 1, wherein the one or more search policies indicate a maximum amount of memory used for a response to a search query.

4. The method of claim 1, further comprising:

determining a time stamp for each event of the plurality of events to create a plurality of time stamped events.

5. The method of claim 1, wherein each event in the plurality of events is created using feature extraction to detect the beginning and ending of events within the raw data.

6. The method of claim 1, wherein a beginning of each subsequent event in the raw data is detected to determine boundaries between events in the raw data.

7. The method of claim 1, wherein the raw data is segmented into events by examining punctuation within the raw data.

8. The method of claim 1, wherein the received raw data includes machine data.

9. One or more non-transitory computer-readable storage media, storing one or more sequences of instructions, which when executed by one or more processors cause performance of:

receiving raw data from a plurality of sources;

creating a plurality of events based on the raw data;

associating a time stamp with each event in the plurality of events;

indexing each time stamped event in the plurality of events;

creating persistent data structures for storing the plurality of events, wherein each persistent data structure corresponds to a specific time interval, wherein events stored in a particular persistent data structure have associated time stamps that fall within a particular time interval corresponding to the particular persistent data structure;

searching events in one or more persistent data structures according to a received search query, wherein the searching the events is halted upon satisfying one or more search policies.

10. The one or more non-transitory computer-readable storage media of claim 9, wherein the one or more search policies indicate a maximum number of events to be returned in response to a search query.

11. The one or more non-transitory computer-readable storage media of claim 9, wherein the one or more search policies indicate a maximum amount of memory used for a response to a search query.

12. The one or more non-transitory computer-readable storage media of claim 9, further comprising:

determining a time stamp for each event of the plurality of events to create a plurality of time stamped events.

13. The one or more non-transitory computer-readable storage media of claim 9, wherein each event in the plurality of events is created using feature extraction to detect the beginning and ending of events within the raw data.

14. The one or more non-transitory computer-readable storage media of claim 9, wherein a beginning of each subsequent event in the raw data is detected to determine boundaries between events in the raw data.

15. The one or more non-transitory computer-readable storage media of claim 9, wherein the raw data is segmented into events by examining punctuation within the raw data.

16. The one or more non-transitory computer-readable storage media of claim 9, wherein the received raw data includes machine data.

17. An apparatus, comprising:

a raw data receiver, implemented at least partially in hardware, that receives raw data from a plurality of sources;

an event creator, implemented at least partially in hardware, that creates a plurality of events based on the raw data;

a time stamp processor, implemented at least partially in hardware, that associates a time stamp with each event in the plurality of events;

an event indexer, implemented at least partially in hardware, that indexes each time stamped event in the plurality of events;

a persistent data structure creation device, implemented at least partially in hardware, that creates persistent data structures for storing the plurality of events, wherein each persistent data structure corresponds to a specific time interval, wherein events stored in a particular persistent data structure have associated time stamps that fall within a particular time interval corresponding to the particular persistent data structure;

an event search device, implemented at least partially in hardware, that searches events in one or more persistent data structures according to a received search query, wherein the event search device halts searching the events upon satisfying one or more search policies.

18. The apparatus of claim 17, wherein the one or more search policies indicate a maximum number of events to be returned in response to a search query.

19. The apparatus of claim 17, wherein the one or more search policies indicate a maximum amount of memory used for a response to a search query.

20. The apparatus of claim 17, wherein each event in the plurality of events is created using feature extraction to detect the beginning and ending of events within the raw data.