QUERY-INITIATED SEARCH ACROSS SEPARATE STORES FOR LOG DATA AND DATA FROM A REAL-TIME MONITORING ENVIRONMENT

• US 20170139963A1
• Filed: 01/31/2017
• Published: 05/18/2017
• Est. Priority Date: 10/05/2006
• Status: Abandoned Application

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Greene, Rory, Mealy, Nicholas Christian, Sorkin, Stephen Phillip, Swan, Erik M., Das, Robin Kumar, Murphy, Brian Philip, Carasso, R. David, Baum, Michael Joseph, Hall, Bradley, Stechert, Andre David
• Application Number:
  US15/421,068
• Publication Number:
  US 20170139963A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2291 User-Defined Types; Storage management thereof
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Methods and apparatus consistent with the invention provide the ability to organize, index, search, and present time series data based on searches. Time series data are sequences of time stamped records occurring in one or more usually continuous streams, representing some type of activity. In one embodiment, time series data is organized into discrete events with normalized time stamps and the events are indexed by time and keyword. A search is received and relevant event information is retrieved based in whole or in part on the time indexing mechanism, keyword indexing mechanism, or statistical indices calculated at the time of the search.

Claims

1. A computer-implemented method for time searching data, comprising:

obtaining log data generated by at least one component in an information processing environment;

obtaining data that is not log data from a real-time monitoring environment;

storing the log data in a first searchable time series data store;

storing the data obtained from the real-time monitoring environment in a second searchable time series data store;

receiving a search query in a search language; and

based on the search query, searching the log data and the data obtained from the real-time monitoring environment.

2. The computer-implemented method of claim 1, wherein the data obtained from the real-time monitoring environment includes sensor data.

3. The computer-implemented method of claim 1, wherein the data obtained from the real-time monitoring environment includes measurement data.

4. The computer-implemented method of claim 1, wherein the data obtained from the real-time monitoring environment includes operational performance data.

5. The computer-implemented method of claim 1, wherein the search query indicates a search for similar data.

6. The computer-implemented method of claim 1, wherein the search query indicates a search for related data.

7. The computer-implemented method of claim 1, wherein the search query indicates a search for data within a defined time range.

8. The computer-implemented method of claim 1, wherein the search query indicates a search for data having a particular frequency of distribution.

9. The computer-implemented method of claim 1, wherein the search query indicates a search for data having a particular pattern of occurrence.

10. The computer-implemented method of claim 1, wherein searching the log data and the data obtained from the real-time monitoring environment includes searching for data within a defined time range specified in the search query.

11. The computer-implemented method of claim 1, further comprising causing display of results of the search.

12. The computer-implemented method of claim 1, further comprising providing results of the search through an application program interface (API).

13. The computer-implemented method of claim 1, wherein the log data comes from two or more sources.

14. The computer-implemented method of claim 1, wherein the data obtained from the real-time monitoring environment comes from two or more sources.

15. The computer-implemented method of claim 1, wherein at least some of the data obtained from the real-time monitoring environment is obtained synchronously.

16. The computer-implemented method of claim 1, wherein at least some of the data obtained from the real-time monitoring environment is obtained asynchronously.

17. The computer-implemented method of claim 1, wherein at least some of the data obtained from the real-time monitoring environment is obtained synchronously and at least some of the data obtained from the real-time monitoring environment is obtained asynchronously.

18. The computer-implemented method of claim 1, further comprising timestamping the log data prior to storing the log data in the first searchable time series data store.

19. The computer-implemented method of claim 1, wherein the obtaining log data comprises collecting the log data at more than one physical location.

20. The computer-implemented method of claim 1, wherein storing the log data comprises:

aggregating the log data into events;

time stamping the events; and

storing the events in the first searchable time series data store.

21. The computer-implemented method of claim 1, wherein storing the log data comprises:

aggregating the log data into events;

time stamping the events; and

storing the events in the first searchable time series data store in chronological order based on the time stamping.

22. The computer-implemented method of claim 1, wherein storing the log data comprises:

aggregating the log data into events;

classifying the events by domain;

time stamping the events based on the domain; and

storing the events in the first searchable time series data store.

23. The computer-implemented method of claim 1, wherein storing the log data comprises:

aggregating the log data into events;

classifying the events by domain;

interpolating a time stamp for at least one event of the events that is not classified in a domain with a known time stamp format;

time stamping the at least one event based on the interpolating; and

storing the at least one event in the first searchable time series data store.

24. The computer-implemented method of claim 1, wherein storing the log data comprises:

aggregating the log data into events using extraction to detect the beginning and ending of the events; and

storing the events in the first searchable time series data store.

25. The computer-implemented method of claim 1, wherein storing the log data comprises:

aggregating the log data into events using machine learning to identify boundaries between events; and

storing the events in the first searchable time series data store.

26. The computer-implemented method of claim 1, wherein storing the log data comprises:

aggregating the log data into events;

time stamping the events; and

combining a group of events into a hot index, which is not searchable and does not persist; and

converting the hot index into a warm index when the hot index is at capacity, the warm index being stored in the first searchable time series data store.

27. A system comprising:

a memory; and

a processing device coupled with the memory to;

obtain log data generated by at least one component in an information processing environment,

obtain data from a real-time monitoring environment,

store the log data in a first searchable time series data store,

store the data obtained from the real-time monitoring environment in a second searchable time series data store, and

receive a search query in a search language, and

based on the search query, searching the log data and the data obtained from the real-time monitoring environment.

28. The system of claim 27, wherein to store the log data, the processing device is coupled with the memory to:

aggregate the log data into events;

time stamp the events; and

store the events in one or more time buckets in the first searchable time series data store based on the time stamp.

29. A non-transitory computer-readable medium encoding instructions thereon that, in response to execution by one or more processing devices, cause the one or more processing devices to perform operations comprising:

obtaining log data generated by at least one component in an information processing environment;

obtaining data from a real-time monitoring environment;

storing the log data in a first searchable time series data store;

storing the data obtained from the real-time monitoring environment in a second searchable time series data store;

receiving a search query in a search language; and

based on the search query, searching the log data and the data obtained from the real-time monitoring environment.

30. The computer-readable readable medium of claim 29, wherein storing the log data comprises:

aggregating the log data into events;

time stamping the events; and

storing the events in the first searchable time series data store.