Log data time stamp extraction and search on log data real-time monitoring environment

• US 9,928,262 B2
• Filed: 07/27/2017
• Issued: 03/27/2018
• Est. Priority Date: 10/05/2006
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Baum, Michael Joseph, Carasso, R. David, Das, Robin Kumar, Greene, Rory, Hall, Bradley, Mealy, Nicholas Christian, Murphy, Brian Philip, Sorkin, Stephen Phillip, Stechert, Andre David, Swan, Erik M.
• Primary Examiner(s): Arjomandi, Noosha
• Application Number:
  US15/661,268
• Publication Number:
  US 20170337231A1
• Time in Patent Office: 243 Days
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2291 User-Defined Types; Storage management thereof
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Methods and apparatus consistent with the invention provide the ability to organize, index, search, and present time series data based on searches. Time series data are sequences of time stamped records occurring in one or more usually continuous streams, representing some type of activity. In one embodiment, time series data is stored as discrete events time stamps. A search is received and relevant event information is retrieved based in whole or in part on the time stamp, a keyword indexing mechanism, or statistical indices calculated at the time of the search.

Claims

1. A computer-implemented method, comprising:

obtaining log data generated by at least one component in an information processing environment;

obtaining data that is not log data from a real-time monitoring environment;

identifying boundaries within the log data that separate the log data into sections, each section of log data reflecting activity in the information processing environment that occurred at a particular time;

for each section of log data,

extracting a time stamp from the section, and

storing in a searchable time series data store at least a portion of log data in the section in association with the extracted time stamp for that section;

storing the data that is not log data in the searchable time series data store; and

executing a search on the log data and the data that is not log data in the searchable time series data store.

2. The computer-implemented method of claim 1, wherein the data that is not log data includes sensor data.

3. The computer-implemented method of claim 1, wherein the data that is not log data includes measurement data.

4. The computer-implemented method of claim 1, wherein the data that is not log data includes operational performance data.

5. The computer-implemented method of claim 1, wherein executing the search includes executing the search to find similar data.

6. The computer-implemented method of claim 1, wherein executing the search includes executing the search to find related data.

7. The computer-implemented method of claim 1, wherein executing the search includes executing the search to find within a defined time range both the log data and the data that is not log data.

8. The computer-implemented method of claim 1, wherein executing the search includes executing the search over a defined time range.

9. The computer-implemented method of claim 1, wherein executing the search includes executing the search to look for a frequency of distribution.

10. The computer-implemented method of claim 1, wherein executing the search includes executing the search to look for a pattern of occurrence.

11. The computer-implemented method of claim 1, further comprising causing display of results of the search.

12. The computer-implemented method of claim 1, wherein executing the search includes executing the search to find within a defined time range both the log data and the data that is not log data, and wherein the computer-implemented method further comprises causing display of results of the search.

13. The computer-implemented method of claim 1, further comprising providing results of the search through an application program interface (API).

14. The computer-implemented method of claim 1, wherein the log data comes from two or more sources.

15. The computer-implemented method of claim 1, wherein the data that is not log data comes from two or more sources.

16. The computer-implemented method of claim 1, wherein at least some of the data that is not log data is obtained synchronously.

17. The computer-implemented method of claim 1, wherein at least some of the data that is not log data is obtained asynchronously.

18. The computer-implemented method of claim 1, wherein at least some of the data that is not log data is obtained synchronously and at least some of the data obtained from the real-time monitoring environment is obtained asynchronously.

19. The computer-implemented method of claim 1, wherein the obtaining log data comprises collecting the log data at more than one physical location.

20. The computer-implemented method of claim 1, wherein the portions of log data are stored in chronological order based on the time stamp extracted from each section.

21. The computer-implemented method of claim 1, further comprising classifying the sections by domain, wherein the time stamp for each section is extracted based on the domain.

22. The computer-implemented method of claim 1, wherein the boundaries are identified using extraction.

23. The computer-implemented method of claim 1, wherein the boundaries are identified using machine learning.

24. The computer-implemented method of claim 1, further comprising:

combining a group of the sections into a hot index, which is not searchable and does not persist; and

converting the hot index into a warm index when the hot index is at capacity, the warm index being stored in the searchable time series data store.

25. A system comprising:

a memory; and

a processing device coupled with the memory to;

obtain log data generated by at least one component in an information processing environment;

obtain data that is not log data from a real-time monitoring environment;

identify boundaries within the log data that separate the log data into sections, each section of log data reflecting activity in the information processing environment that occurred at a particular time;

for each section of log data,

extract a time stamp from the section, and

store in a searchable time series data store at least a portion of log data in the section in association with the extracted time stamp for that section;

store the data that is not log data in the searchable time series data store; and

execute a search on the log data and the data that is not log data in the searchable time series data store.

26. The system of claim 25, wherein the portions of log data are stored in chronological order based on the time stamp extracted from each section.

27. The system of claim 25, wherein the processing device is further coupled with the memory to classify the sections by domain, wherein the time stamp for each section is extracted based on the domain.

28. A non-transitory computer-readable medium encoding instructions thereon that, in response to execution by one or more processing devices, cause the one or more processing devices to:

obtain log data generated by at least one component in an information processing environment;

obtain data that is not log data from a real-time monitoring environment;

identify boundaries within the log data that separate the log data into sections, each section of log data reflecting activity in the information processing environment that occurred at a particular time;

for each section of log data,

extract a time stamp from the section, and

store in a searchable time series data store at least a portion of log data in the section in association with the extracted time stamp for that section;

store the data that is not log data in the searchable time series data store; and

execute a search on the log data and the data that is not log data in the searchable time series data store.

29. The non-transitory computer-readable medium of claim 28, wherein the portions of log data are stored in chronological order based on the time stamp extracted from each section.

30. The non-transitory computer-readable medium of claim 28, wherein the instructions thereon further cause the one or more processing devices to classify the sections by domain, wherein the time stamp for each section is extracted based on the domain.