Normalization of time stamps for event data

• US 9,514,175 B2
• Filed: 01/26/2016
• Issued: 12/06/2016
• Est. Priority Date: 10/05/2006
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Swan, Erik M., Carasso, R. David, Das, Robin Kumar, Greene, Rory, Hall, Bradley, Mealy, Nicholas Christian, Murphy, Brian Philip, Sorkin, Stephen Phillip, Stechert, Andre David, Baum, Michael Joseph
• Primary Examiner(s): Uddin, Mohammed R
• Application Number:
  US15/007,176
• Publication Number:
  US 20160140238A1
• Time in Patent Office: 315 Days
• Field of Search: 707/746, 707/752, 707/753, 707/E17.002, 707/E17.083, 707/999.006
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2291 User-Defined Types; Storage management thereof
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Methods and apparatus consistent with the invention provide the ability to organize, index, search, and present time series data based on searches. Time series data are sequences of time stamped records occurring in one or more usually continuous streams, representing some type of activity. In one embodiment, time series data is organized into discrete events with normalized time stamps and the events are indexed by time and keyword. A search is received and relevant event information is retrieved based in whole or in part on the time indexing mechanism, keyword indexing mechanism, or statistical indices calculated at the time of the search.

Claims

1. A method, comprising:

detecting whether time information is present in an event that includes a portion of raw data, the event being derived from raw data that includes the portion of raw data, the raw data pertaining to performance or security aspects of one or more information technology systems;

in response to detecting that the time information is present in the event;

determining a time zone in the detected time information;

generating an offset by normalizing the detected time information using the determined time zone;

generating a time stamp based on the offset; and

associating the generated time stamp with the event, thereby enabling the event to be searched using the time stamp;

in response to detecting that the time information is not present in the event;

interpolating a time stamp for the event using at least one event preceding the event and at least one event succeeding the event; and

associating the interpolated time stamp with the event;

wherein the method is performed by one or more computing devices.

2. The method of claim 1, wherein associating the generated time stamp further comprises modifying the time information in the event to the generated time stamp.

3. The method of claim 1, wherein the event is among a plurality of events organized from the raw data, each event in the plurality of events includes a portion of the raw data.

4. The method of claim 1, further comprising:

transforming the raw data into a plurality of time stamped events;

wherein the event is among the plurality of time stamped events.

5. The method of claim 1, further comprising:

transforming the raw data into a plurality of time stamped events;

wherein the event is among the plurality of time stamped events;

receiving a search query;

executing the search query across the plurality of time stamped events.

6. The method of claim 1, further comprising:

transforming the raw data into a plurality of time stamped events;

wherein the event is among the plurality of time stamped events;

indexing the plurality of time stamped events;

receiving a search query;

executing the search query across the plurality of time stamped events.

7. The method of claim 1, further comprising:

transforming the raw data into a plurality of time stamped events;

wherein the event is among the plurality of time stamped events;

indexing the plurality of time stamped events;

receiving a search query;

executing the search query across the plurality of indexed time stamped events.

8. The method of claim 1, further comprising:

identifying a domain for the event;

wherein the detecting whether time information is present in the event further comprises;

determining a location of the time information within the event using the identified domain.

9. The method of claim 1, wherein the event is among a plurality of events organized from the raw data, each event in the plurality of events includes a portion of the raw data, wherein the raw data includes machine data.

10. The method of claim 1, wherein the event is among a plurality of events organized from the raw data, each event in the plurality of events includes a portion of the raw data, wherein the raw data includes unstructured data.

11. An apparatus, comprising:

an event processor, implemented at least partially in hardware, that detects whether time information is present in an event that includes a portion of raw data, the event being derived from raw data that includes the portion of raw data, the raw data pertaining to performance or security aspects of one or more information technology systems;

in response to the event processor detecting that the time information is present in the event, the event processor;

determines a time zone in the detected time information;

generates an offset by normalizing the detected time information using the determined time zone;

generates a time stamp based on the offset; and

associates the generated time stamp with the event, thereby enabling the event to be searched using the time stamp;

in response to the event processor detecting that the time information is not present in the event, the event processor;

interpolates a time stamp for the event using at least one event preceding the event and at least one event succeeding the event; and

associates the interpolated time stamp with the event.

12. The apparatus of claim 11, wherein the event processor further:

modifies the time information in the event to the generated time stamp.

13. The apparatus of claim 11, wherein the event is among a plurality of events organized from the raw data, each event in the plurality of events includes a portion of the raw data.

14. The apparatus of claim 11, wherein the event processor further:

transforms the raw data into a plurality of time stamped events;

wherein the event is among the plurality of time stamped events;

indexes the plurality of time stamped events;

receives a search query;

executes the search query across the plurality of time stamped events.

19. The apparatus of claim 11, wherein the event is among a plurality of events organized from the raw data, each event in the plurality of events includes a portion of the raw data, wherein the raw data includes machine data.

15. One or more non-transitory computer-readable storage media, storing one or more sequences of instructions, which when executed by one or more processors cause performance of:

detecting whether time information is present in an event that includes a portion of raw data, the event being derived from raw data that includes the portion of raw data, the raw data pertaining to performance or security aspects of one or more information technology systems;

in response to detecting that the time information is present in the event;

determining a time zone in the detected time information;

generating an offset by normalizing the detected time information using the determined time zone;

generating a time stamp based on the offset; and

associating the generated time stamp with the event, thereby enabling the event to be searched using the time stamp;

in response to detecting that the time information is not present in the event;

interpolating a time stamp for the event using at least one event preceding the event and at least one event succeeding the event; and

associating the interpolated time stamp with the event.

16. The one or more non-transitory computer-readable storage media of claim 15, wherein associating the generated time stamp further comprises modifying the time information in the event to the generated time stamp.

17. The one or more non-transitory computer-readable storage media of claim 15, wherein the event is among a plurality of events organized from the raw data, each event in the plurality of events includes a portion of the raw data.

18. The one or more non-transitory computer-readable storage media of claim 15, further comprising:

transforming the raw data into a plurality of time stamped events;

wherein the event is among the plurality of time stamped events;

indexing the plurality of time stamped events;

receiving a search query;

executing the search query across the plurality of time stamped events.

20. The one or more non-transitory computer-readable storage media of claim 15, wherein the event is among a plurality of events organized from the raw data, each event in the plurality of events includes a portion of the raw data, wherein the raw data includes machine data.