Determining Timestamps To Be Associated With Events In Machine Data

• US 20160070736A1
• Filed: 10/30/2015
• Published: 03/10/2016
• Est. Priority Date: 10/05/2006
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Swan, Erik M., Carasso, R. David, Greene, Rory, Mealy, Nicholas Christian, Sorkin, Stephen Phillip, Stechert, Andre David, Baum, Michael Joseph, Das, Robin Kumar, Hall, Bradley, Murphy, Brian Philip
• Granted Patent: US 9,922,065 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2291 User-Defined Types; Storage management thereof
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Methods and apparatus consistent with the invention provide the ability to organize, index, search, and present time series data based on searches. Time series data are sequences of time stamped records occurring in one or more usually continuous streams, representing some type of activity. In one embodiment, time series data is organized into discrete events with normalized time stamps and the events are indexed by time and keyword. A search is received and relevant event information is retrieved based in whole or in part on the time indexing mechanism, keyword indexing mechanism, or statistical indices calculated at the time of the search.

Claims

1. A method, comprising:

aggregating a collection of machine data stored on at least one storage device into a set of events, wherein the machine data associated with at least a subset of the events includes time information;

extracting a timestamp for each event in the subset of events that includes time information;

for each event that does not contain time information in the associated machine data, determining a time stamp from at least one other event in the collection of machine data; and

associating the determined time stamp with the corresponding event,

wherein the method is performed by one or more computing devices.

2. The method of claim 1, wherein the set of events are in chronological order.

3. The method of claim 1, wherein determining a time stamp further comprises interpolating time stamps of one or more events preceding and following the event.

4. The method of claim 1, wherein the time stamp for each event that does not contain time information is determined using linear interpolation.

5. The method of claim 1, wherein when an event does not contain time information and the event preceding or following the event also does not contain time information, the time stamp is determined by using the next respective preceding or following event.

6. The method of claim 1, wherein aggregating the machine data into a set of events further comprises using extraction to detect the beginning and ending of events within the machine data.

7. The method of claim 1, wherein aggregating the machine data into a set of events includes identifying a domain for the machine data.

8. The method of claim 1, wherein aggregating the machine data into a set of events includes identifying boundaries between events using machine learning.

9. The method of claim 1, wherein aggregating the machine data into a set of events includes analyzing a portion of the machine data to detect the beginning and ending of events within the machine data.

10. The method of claim 1, wherein each of the events in the set of events includes an unaltered portion of machine data.

11. The method of claim 1, further comprising determining the boundaries between events by detecting the beginning of each subsequent event.

12. The method of claim 1, further comprising indexing the events based on the determined time stamp.

13. The method of claim 1, wherein the collection of machine data is in a stream, and aggregating into events includes determining a rule for separating the stream based on one or more of:

a source of the stream of data, a domain of the stream of data, a type of the stream of data, a signature of the stream of data, a punctuation analysis of the stream of data, or an analysis of repeating patterns within the stream of data.

14. The method of claim 1, wherein the collection of machine data is part of a server log, a portion of a log file, a transaction record, a recorded measurement, message bus traffic, network data, network management data, or an output of an application.

15. The method of claim 1, further comprising assigning each event in the set of events to a bucket having an associated time range that includes the time represented by the time stamp for the event.

16. The method of claim 1, wherein the collection of machine data is from logs from a plurality of different sources and the format associated with each source is determined so as to automatically extract the timestamp for each event in the subset of events that includes time information.

17. The method of claim 1, further comprising dividing event data in each event into segments.

18. The method of claim 1, wherein the events are time stamped events indexed according to time to facilitate search queries on the events using time-based operators.

19. The method of claim 1, further comprising performing entity extraction to identify semantic entities within the machine data of the time stamped events.

20. A non-transitory, computer-readable storage medium storing instructions, an execution of which in a computer system causes the computer system to perform operations comprising:

aggregating a collection of machine data stored on at least one storage device into a set of events, wherein the machine data associated with at least a subset of the events includes time information;

extracting a timestamp for each event in the subset of events that includes time information;

for each event that does not contain time information in the associated machine data, determining a time stamp from at least one other event in the collection of machine data; and

associating the determined time stamp with the corresponding event.

21. The computer-readable storage medium of claim 20, wherein determining a time stamp further comprises interpolating time stamps of one or more events preceding and following the event.

22. The computer-readable storage medium of claim 20, wherein the time stamp for each event that does not contain time information is determined using linear interpolation.

23. The computer-readable storage medium of claim 20, wherein when an event does not contain time information and the event preceding or following the event also does not contain time information, the time stamp is determined by using the next respective preceding or following event.

24. The computer-readable storage medium of claim 20, wherein aggregating the machine data into a set of events further comprises using extraction to detect the beginning and ending of events within the machine data.

25. A computer system comprising:

computer memory for storing machine data; and

a processor for;

aggregating a collection of machine data stored on at least one storage device into a set of events, wherein the machine data associated with at least a subset of the events includes time information;

extracting a timestamp for each event in the subset of events that includes time information;

for each event that does not contain time information in the associated machine data, determining a time stamp from at least one other event in the collection of machine data; and

associating the determined time stamp with the corresponding event.

26. The computer system of claim 25, wherein determining a time stamp further comprises interpolating time stamps of one or more events preceding and following the event.

27. The computer system of claim 25, wherein the time stamp for each event that does not contain time information is determined using linear interpolation.

28. The computer system of claim 25, wherein when an event does not contain time information and the event preceding or following the event also does not contain time information, the time stamp is determined by using the next respective preceding or following event.

29. The computer system of claim 25, wherein aggregating the machine data into a set of events includes using extraction to detect the beginning and ending of events within the machine data.

30. The computer system of claim 25, wherein aggregating the machine data into a set of events includes identifying boundaries between events using machine learning.