Extracting unique field values from event fields

• US 10,061,821 B2
• Filed: 07/31/2016
• Issued: 08/28/2018
• Est. Priority Date: 03/14/2011
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Zhang, Steve Yu, Sorkin, Stephen Phillip
• Primary Examiner(s): Dang, Thanh-Ha
• Application Number:
  US15/224,657
• Publication Number:
  US 20160342601A1
• Time in Patent Office: 758 Days
• Field of Search: 707725
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/24 Querying
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/24553 of query operations
• CPC Class Codes: G06F 16/24554 Unary operations; Data partitioning operations
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06F 16/90328 using search space presentation or visualization, e.g. category or range presentation and selection
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: H04L 41/0604 using filtering, e.g. reduction of information by using priority, element types, position or time
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method, system, and processor-readable storage medium are directed towards generating a report derived from data, such as event data, stored on a plurality of distributed nodes. In one embodiment the analysis is generated using a “divide and conquer” algorithm, such that each distributed node analyzes locally stored event data while an aggregating node combines these analysis results to generate the report. In one embodiment, each distributed node also transmits a list of event data references associated with the analysis result to the aggregating node. The aggregating node may then generate a global ordered list of data references based on the list of event data references received from each distributed node. Subsequently, in response to a user selection of a range of global event data, the report may dynamically retrieve event data from one or more distributed nodes for display according to the global order.

Claims

1. A method, comprising:

receiving a query for searching a set of field searchable events stored in a data store, the set of field searchable events indicative of security or performance aspects of one or more information technology systems;

receiving a query for searching a set of field searchable events stored in a data store, the set of field searchable events indicative of security or performance aspects of one or more information technology systems;

causing concurrent display of (i) the number corresponding to how many unique values exist for the field along with a field name by which the field can be referenced in a query, and (ii) one or more other field names that each correspond to a field existing in one or more events in the subset of events and, for each other field name of the one or more other field names, an associated number corresponding to how many unique values exist for the corresponding field in the subset of events, thereby improving the performance and efficiency of communicating information from complex search results to a user;

wherein the method is performed by one or more computing devices.

2. The method of claim 1, wherein each event in the set of field searchable events is associated with a timestamp.

3. The method of claim 1, wherein each event in the set of field searchable events includes machine data reflecting activity in the one or more information.

4. The method of claim 1, wherein at least one event in the set of field searchable events includes log data reflecting activity in the one or more information technology systems.

5. The method of claim 1, wherein at least one event in the set of field searchable events includes unstructured data.

6. The method of claim 1, wherein the query for searching the set of field searchable events includes a criterion for evaluating values for a field that exists in one or more events in the set of field searchable events.

7. The method of claim 1, wherein the query for searching the set of field searchable events includes a criterion requiring that matching events have a particular keyword.

8. The method of claim 1, wherein each of the events in the set of field searchable events is associated with a timestamp, and wherein the query for searching the set of field searchable events is associated with a time range into which matching events must fall.

9. The method of claim 1, further comprising:

causing display of information about one or more events that have the field and that are in the subset of events.

10. The method of claim 1, further comprising:

causing display of information about two or more events that have the field and that are in the subset events, wherein the two or more events that have the field are displayed in a sorted order corresponding to timestamps associated with the two or more events.

11. The method of claim 1, further comprising:

causing display of at least a second number corresponding to how many unique values exist among the subset of events for a second field that exists in one or more events of the subset of events.

12. The method of claim 1, wherein the display of the number corresponding to how many unique values exist for the field along with the field name is displayed in a field picker.

13. The method of claim 1, wherein the display of the number corresponding to how many unique values exist for the field along with the field name by which the field can be referenced in the query is displayed in a field picker.

14. The method of claim 1, wherein the display of the number corresponding to how many unique values exist for the field along with the field name by which the field can be referenced in the query is displayed in a field picker, and wherein the method further comprises:

causing display of information about one or more events in the subset of events;

receiving from a user a selection of a field name through the field picker;

hiding from the display at least one previously displayed event that does not contain the selected field.

15. The method of claim 1, wherein the display of the number corresponding to how many unique values exist for the field along with the field name by which the field can be referenced in the query is displayed in a field picker for specifying field criteria to further filter the subset of the events.

16. The method of claim 1, further comprising:

causing display of a histogram that indicates how many events in the subset of events are associated with a timestamp falling within each of a plurality of time ranges.

17. The method of claim 1, wherein causing display of the number corresponding to how many unique values exist for the field along with the field name by which the field can be referenced in the query causes display of the number and the field name in a vicinity of one or more other displayed numbers and their associated field names.

18. An apparatus, comprising:

a query receiver, implemented at least partially in hardware, that receives a query for searching a set of field searchable events stored in a data store, the set of field searchable events indicative of security or performance aspects of one or more information technology systems;

a query executor, implemented at least partially in hardware, that executes the query against the set of field searchable events to generate a subset of events;

a field identifier, implemented at least partially in hardware, that identifying a field that exists in one or more events of the subset of events;

a subsystem, implemented at least partially in hardware, that determines a number corresponding to how many unique values exist for the field among the subset of events;

a subsystem, implemented at least partially in hardware, that causes concurrent display of (i) the number corresponding to how many unique values exist for the field along with a field name by which the field can be referenced in a query, and (ii) one or more other field names that each correspond to a field existing in one or more events in the subset of events and, for each other field name of the one or more other field names, an associated number corresponding of how many unique values exist for the corresponding field in the subset of events, thereby improving the performance and efficiency of communicating information from complex search results to a user.

19. The apparatus of claim 18, wherein each event in the set of field searchable events is associated with a timestamp.

20. The apparatus of claim 18, wherein each event in the set of field searchable events includes at least one of:

machine data reflecting activity in an information technology environment, log data reflecting activity in an information technology environment, or unstructured data.

21. The apparatus of claim 18, wherein the query for searching the set of field searchable events includes a criterion for evaluating values for a field that exists in one or more events in the set of field searchable events.

22. The apparatus of claim 18, wherein each of the events in the set of field searchable events is associated with a timestamp, and wherein the query for searching the set of field searchable events is associated with a time range into which matching events must fall.

23. The apparatus of claim 18, further comprising:

a subsystem, implemented at least partially in hardware, that causes display of information about two or more events that have the field and that are in the subset events, wherein the two or more events that have the field are displayed in a sorted order corresponding to timestamps associated with the two or mare events.

24. One or more non-transitory computer-readable storage media, storing software instructions, which when executed by one or more processors cause performance of:

receiving a query for searching a set of field searchable events stored in a data store, the set of field searchable events indicative of security or performance aspects of one or more information technology systems;

executing the query against the set of field searchable events to generate a subset of events;

identifying a field that exists in one or more events of the subset of events;

determining a number corresponding to how many unique values exist for the field among the subset of events;

causing concurrent display of (i) the number corresponding to how many unique values exist for the field along with a field name by which the field can be referenced in a query, and (ii) one or more other field names that each correspond to a field existing in one or more events in the subset of events and, for each other field name of the one or more other field names, an associated number corresponding of how many unique values exist for the corresponding field in the subset of events, thereby improving the performance and efficiency of communicating information from complex search results to a user.

25. The one or more non-transitory computer-readable storage media of claim 24, wherein each event in the set of field searchable events is associated with a timestamp.

26. The one or more non-transitory computer-readable storage media of claim 24, wherein each event in the set of field searchable events includes at least one of:

machine data reflecting activity in an information technology environment, log data reflecting activity in an information technology environment, or unstructured data.

27. The one or more non-transitory computer-readable storage media of claim 24, wherein the query for searching the set of field searchable events includes a criterion for evaluating values for a field that exists in one or more events in the set of field searchable events.

28. The one or more non-transitory computer-readable storage media of claim 24, wherein each of the events in the set of field searchable events is associated with a timestamp, and wherein the query for searching the set of field searchable events is associated with a time range into which matching events must fall.

29. The one or more non-transitory computer-readable storage media of claim 24, wherein the instructions, when executed by the one or more computing devices, further cause performance of:

causing display of information about two or more events that have the field and that are in the subset events, wherein the two or more events that have the field are displayed in a sorted order corresponding to timestamps associated with the two or more events.

30. The one or more non-transitory computer-readable storage media of claim 24, wherein the instructions, when executed by the one or more computing devices, further cause performance of:

causing display of at least a second number corresponding to how many unique values exist among the subset of events for a second field that exists in one or more events of the subset of events.