Storing events associated with a time stamp extracted from log data and performing a search on the events and data that is not log data

• US 10,740,313 B2
• Filed: 01/30/2018
• Issued: 08/11/2020
• Est. Priority Date: 10/05/2006
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Baum, Michael Joseph, Carasso, R. David, Das, Robin Kumar, Greene, Rory, Hall, Bradley, Mealy, Nicholas Christian, Murphy, Brian Philip, Sorkin, Stephen Phillip, Stechert, Andre David, Swan, Erik M.
• Primary Examiner(s): Arjomandi, Noosha
• Application Number:
  US15/883,588
• Publication Number:
  US 20180173739A1
• Time in Patent Office: 924 Days
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2291 User-Defined Types; Storage management thereof
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Methods and apparatus consistent with the invention provide the ability to organize, index, search, and present time series data based on searches. Time series data are sequences of time stamped records occurring in one or more usually continuous streams, representing some type of activity. In one embodiment, time series data is stored as discrete events time stamps. A search is received and relevant event information is retrieved based in whole or in part on the time stamp, a keyword indexing mechanism, or statistical indices calculated at the time of the search.

Claims

1. A computer-implemented method for time searching data, comprising:

obtaining log data generated by at least one component in an information processing environment;

obtaining data that is not log data from a real-time monitoring environment;

storing in a data store, a plurality of events, wherein each event is based on at least a portion of the log data and is associated with a time stamp extracted from the log data;

storing the data that is not log data in the data store; and

executing a search on the plurality of events and the data that is not log data in the data store.

2. The computer-implemented method of claim 1, wherein the time stamp is extracted from the portion of the log data upon which the event is based.

3. The computer-implemented method of claim 1, wherein the event includes the at least a portion of the log data.

4. The computer-implemented method of claim 1, wherein the data obtained from the real-time monitoring environment includes sensor data.

5. The computer-implemented method of claim 1, wherein the data obtained from the real-time monitoring environment includes measurement data.

6. The computer-implemented method of claim 1, wherein executing the search includes executing the search to find similar data.

7. The computer-implemented method of claim 1, wherein executing the search includes executing the search to find related data.

8. The computer-implemented method of claim 1, wherein executing the search includes executing the search to find within a defined time range both the log data and the data obtained from the real-time monitoring environment.

9. The computer-implemented method of claim 1, wherein executing the search includes executing the search over a defined time range.

10. The computer-implemented method of claim 1, wherein executing the search includes executing the search to look for a frequency of distribution.

11. The computer-implemented method of claim 1, wherein executing the search includes executing the search to look for a pattern of occurrence.

12. The computer-implemented method of claim 1, wherein executing the search includes executing the search to find within a defined time range both the log data and the data obtained from the real-time monitoring environment, and wherein the computer-implemented method further comprises causing display of results of the search.

13. The computer-implemented method of claim 1, wherein at least one of the log data or the data obtained from the real-time monitoring environment comes from two or more sources.

14. The computer-implemented method of claim 1, wherein at least some of the data obtained from the real-time monitoring environment is obtained synchronously.

15. The computer-implemented method of claim 1, wherein at least some of the data obtained from the real-time monitoring environment is obtained asynchronously.

16. The computer-implemented method of claim 1, wherein the obtaining log data comprises collecting the log data at more than one physical location.

17. The computer-implemented method of claim 1, wherein storing the plurality of events comprises:

identifying boundaries within the log data that separate the log data into sections, wherein the at least a portion of the log data corresponds to a section of the sections; and

storing at least a portion of each section as an event of the plurality of events in the data store.

18. The computer-implemented method of claim 1, wherein storing the plurality of events comprises:

identifying boundaries within the log data that separate the log data into sections;

extracting a time stamp from each section, wherein the at least a portion of the log data corresponds to the each section and the time stamp extracted from the log data corresponds to the time stamp extracted from the each section; and

storing at least a portion of each section as an event of the plurality of events in the data store.

19. The computer-implemented method of claim 1, wherein storing the plurality of events comprises:

identifying boundaries within the log data that separate the log data into sections;

extracting a time stamp from each section, wherein the at least a portion of the log data corresponds to the each section and the time stamp extracted from the log data corresponds to the time stamp extracted from the each section; and

storing the plurality of events in chronological order based on the time stamp extracted from the each section.

20. The computer-implemented method of claim 1, wherein storing the plurality of events comprises:

aggregating the log data into the plurality of events;

time stamping the plurality of events; and

storing the plurality of events in the data store.

21. The computer-implemented method of claim 1, wherein storing the plurality of events comprises:

aggregating the log data into the plurality of events using extraction to detect a beginning and ending of the plurality of events; and

storing the plurality of events in the data store.

22. The computer-implemented method of claim 1, wherein storing the plurality of events comprises:

aggregating the log data into the plurality of events using machine learning to identify boundaries between events; and

storing the plurality of events in the data store.

23. The computer-implemented method of claim 1, wherein storing the plurality of events comprises:

aggregating the log data into the plurality of events;

time stamping the plurality of events;

combining a group of the plurality of events into a hot index, which is not searchable and does not persist; and

converting the hot index into a warm index when the hot index is at capacity, the warm index being stored in the data store.

24. The computer-implemented method of claim 1, wherein the search identifies a time range, the method further comprising:

storing a plurality of indices, each index associated with a group of events of the plurality of events and identifying a period of time associated with the group of events,

wherein executing the search on the log data comprises dividing the search into a plurality of sub-searches based on an identification of one or more indices of the plurality of indices that identify a period of time that satisfies the time range.

25. The computer-implemented method of claim 1, further comprising storing an index associated with a group of events of the plurality of events, the index comprising a plurality of tokens associated with the group of events, wherein each token of the plurality of tokens is associated with at least one event of the group of events.

26. The computer-implemented method of claim 1, further comprising storing an index associated with a group of events of the plurality of events, the index comprising:

a plurality of tokens associated with the group of events, wherein each token of the plurality of tokens is associated with at least one event of the group of events, and

an address for each event of the group of events.

27. The computer-implemented method of claim 1, further comprising storing an index associated with a group of events of the plurality of events, the index comprising:

a plurality of tokens associated with the group of events, wherein each token of the plurality of tokens is associated with at least one event of the group of events,

an address for each event of the group of events, and

a posting list identifying an association between each token of the plurality of tokens with the address of the at least one event of the group of events.

28. The computer-implemented method of claim 1, further comprising:

storing in volatile memory an index associated with a group of events of the plurality of events, the index comprising;

a plurality of tokens associated with the group of events, wherein each token of the plurality of tokens is associated with at least one event of the group of events,

an address for each event of the group of events, and

a posting list identifying an association between each token of the plurality of tokens with the address of the at least one event of the group of events; and

storing the index in non-volatile memory based on a determination that at least one of a threshold period of time is satisfied or a size of the index satisfies a threshold size.

29. A computing system, comprising:

memory; and

one or more processing devices coupled to the memory and configured to;

obtain log data generated by at least one component in an information processing environment;

obtain data that is not log data from a real-time monitoring environment;

store in a data store, a plurality of events, wherein each event is based on at least a portion of the log data and is associated with a time stamp extracted from the log data;

store the data that is not log data in the data store; and

execute a search on the plurality of events and the data that is not log data in the data store.

30. Non-transitory computer readable media comprising computer-executable instructions that, when executed by a computing system of a data intake and query system, cause the computing system to:

obtain log data generated by at least one component in an information processing environment;

obtain data that is not log data from a real-time monitoring environment;

store in a data store, a plurality of events, wherein each event is based on at least a portion of the log data and is associated with a time stamp extracted from the log data;

store the data that is not log data in the data store; and

execute a search on the plurality of events and the data that is not log data in the data store.