Aggregation and display of search results from multi-criteria search queries on event data

• US 9,922,066 B2
• Filed: 01/27/2016
• Issued: 03/20/2018
• Est. Priority Date: 10/05/2006
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Swan, Erik M., Carasso, R. David, Das, Robin Kumar, Greene, Rory, Hall, Bradley, Mealy, Nicholas Christian, Murphy, Brian Philip, Sorkin, Stephen Phillip, Stechert, Andre David, Baum, Michael Joseph
• Primary Examiner(s): Uddin, Mohammed R
• Application Number:
  US15/008,428
• Publication Number:
  US 20160140128A1
• Time in Patent Office: 783 Days
• Field of Search: 707725, 707752, 707753, 707746, 707750, 707754, 707999006
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2291 User-Defined Types; Storage management thereof
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Methods and apparatus consistent with the invention provide the ability to organize, index, search, and present time series data based on searches. Time series data are sequences of time stamped records occurring in one or more usually continuous streams, representing some type of activity. In one embodiment, time series data is organized into discrete events with normalized time stamps and the events are indexed by time and keyword. A search is received and relevant event information is retrieved based in whole or in part on the time indexing mechanism, keyword indexing mechanism, or statistical indices calculated at the time of the search.

Claims

1. A method, comprising:

creating, in real-time, a plurality of searchable events from machine data as the machine data is collected in real-time from one or more data sources, each event in the plurality of searchable events is segmented from the machine data and includes an associated portion of the machine data and an associated timestamp derived from the machine data;

dividing the plurality of events into sets of events that are organized by time;

indexing the timestamped events;

hashing each event in the sets of events, wherein each event is tested for duplication using its associated hash value, wherein an event having a hash value that is a duplicate of an existing hash value is removed;

as the plurality of events are being created in real-time, receiving a search query that includes at least a time criterion, a second criterion for selection of events, and a page value;

generating a result set for an event search query by executing the event search query across the plurality of events, the event search query includes the time criterion and the second criterion for selection of events, the result set includes events that match the time criterion and have an associated portion of the machine data that fulfills the second criterion for selection of events;

sorting the result set according to time;

causing display of a plurality of aggregated display lines, wherein each aggregated display line among the plurality of aggregated display lines is a summary of one or more search results among the set of search results that have features that satisfy a particular interval among a plurality of intervals and the page value, each interval among the plurality of intervals fitting within a display page.

2. The method of claim 1, wherein the second criterion for selection of events is a keyword.

3. The method of claim 1, wherein one or more events in the set of search results is assigned a keyword relevance ranking.

4. The method of claim 1, further comprising:

receiving a duration of the time for the sets of events from a user.

5. The method of claim 1, further comprising:

displaying an interactive paging of the set of search results.

6. The method of claim 1, wherein the machine data includes raw log data.

7. The method of claim 1, wherein the machine data includes unstructured data.

8. An apparatus, comprising:

a machine data transformation device, implemented at least partially in hardware, that creates, in real-time, a plurality of searchable events from machine data as the machine data is collected in real-time from one or more data sources, each event in the plurality of searchable events is segmented from the machine data and includes an associated portion of the machine data and an associated timestamp derived from the machine data;

wherein the machine data transformation device divides the plurality of events into sets of events that are organized by time;

wherein the machine data transformation device indexes the timestamped events;

wherein the machine data transformation device hashes each event in the sets of events, wherein each event is tested for duplication using its associated hash value, wherein an event having a hash value that is a duplicate of an existing hash value is removed;

a search receiver, implemented at least partially in hardware, that, as the plurality of events are being created in real-time, receives a search query that includes at least a time criterion, a second criterion for selection of events, and a] page value;

a search result generator, implemented at least partially in hardware, that generates a result set for an event search query by executing the event search query across the plurality of events, the event search query includes the time criterion and the second criterion for selection of events, the result set includes events that match the time criterion and have an associated portion of the machine data that fulfills the second criterion for selection of events;

a search result sorter, implemented at least partially in hardware, that sorts the result set according to time;

a display formatter, implemented at least partially in hardware, that causes display of a plurality of aggregated display lines, wherein each aggregated display line among the plurality of aggregated display lines is a summary of one or more search results among the set of search results that have features that satisfy a particular interval among a plurality of intervals and the page value, each interval among the plurality of intervals fitting within a display page.

9. The apparatus of claim 8, wherein the second criterion is a keyword.

10. The apparatus of claim 8, wherein one or more events in the set of search results is assigned a keyword relevance ranking.

11. The apparatus of claim 8, further comprising:

a user input receiver, implemented at least partially in hardware, that receives a duration of the time for the sets of events from a user.

12. The apparatus of claim 8, wherein the machine data includes raw log data.

13. The apparatus of claim 8, wherein the display formatter displays an interactive paging of the set of search results.

14. The apparatus of claim 8, wherein the machine data includes unstructured data.

15. One or more non-transitory computer-readable storage media, storing one or more sequences of instructions, which when executed by one or more processors cause performance of:

creating, in real-time, a plurality of searchable events from machine data as the machine data is collected in real-time from one or more data sources, each event in the plurality of searchable events is segmented from the machine data and includes an associated portion of the machine data and an associated timestamp derived from the machine data;

dividing the plurality of events into sets of events that are organized by time;

indexing the timestamped events;

hashing each event in the sets of events, wherein each event is tested for duplication using its associated hash value, wherein an event having a hash value that is a duplicate of an existing hash value is removed;

as the plurality of events are being created in real-time, receiving a search that includes at least a time criterion, a second criterion for selection of events, and a] page value;

generating a result set for an event search query by executing the event search query across the plurality of events, the event search query includes the time criterion and the second criterion for selection of events, the result set includes events that match the time criterion and have an associated portion of the machine data that fulfills the second criterion for selection of events;

sorting the result set according to time;

causing display of a plurality of aggregated display lines, wherein each aggregated display line among the plurality of aggregated display lines is a summary of one or more search results among the set of search results that have features that satisfy a particular interval among a plurality of intervals and the page value, each interval among the plurality of intervals fitting within a display page.

16. The one or more non-transitory computer-readable storage media of claim 15, wherein the second criterion is a keyword.

17. The one or more non-transitory computer-readable storage media of claim 15, further comprising:

receiving a duration of the time for the sets of events from a user.

18. The one or more non-transitory computer-readable storage media of claim 15, wherein the one or more sequences of instructions, which when executed by the one or more processors cause further performance of:

displaying an interactive paging of the set of search results.

19. The one or more non-transitory computer-readable storage media of claim 15, wherein the machine data includes raw log data.

20. The one or more non-transitory computer-readable storage media of claim 15, wherein the machine data includes unstructured data.