Identifying correlations between log data and network packet data

• US 10,268,652 B2
• Filed: 10/31/2016
• Issued: 04/23/2019
• Est. Priority Date: 07/31/2014
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Cormier, Michael E., Thackrey, William E., Cox, Earl D.
• Primary Examiner(s): Lin, Shew Fen
• Application Number:
  US15/339,467
• Publication Number:
  US 20170046445A1
• Time in Patent Office: 904 Days
• CPC Class Codes: G06F 11/30 Monitoring
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 16/2468 Fuzzy queries
• CPC Class Codes: G06F 16/319 Inverted lists
• CPC Class Codes: G06F 16/86 Mapping to a database
• CPC Class Codes: G06F 16/901 Indexing; Data structures therefor; Storage structures for retrieval from the web G06F16/951
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06F 16/95 Retrieval from the web
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications

Abstract

The disclosed embodiments relate to a system that facilitates performing searches based on qualitative search terms. During operation, the system receives a query that applies a qualitative search term to an attribute of data items in a set of data items. While executing the query, the system processes each data item in the set of data items by extracting an attribute value from the data item and then using a concept-mapping to determine a compatibility index for the attribute value, wherein the concept-mapping associates each attribute value with a numerical compatibility index that indicates a compatibility between the attribute value and the qualitative search term. Finally, the system uses the compatibility index as a factor in determining whether to include the data item in a set of query results.

Claims

1. A computer-implemented method, comprising:

obtaining log data generated by at least one component in an information technology (IT) environment, and network packet data generated by at least one component in the IT environment;

generating, based on the log data and the network packet data, event data including log data events and network packet data events;

receiving, by a query processor, a query that includes search criteria including;

an attribute of at least one log data event and at least one network packet data event; and

a plurality of qualitative search terms to be applied to the attribute using an aggregate compatibility index for the event data,

wherein the plurality of qualitative search terms each comprises an adjective describing the attribute, and

wherein the aggregate compatibility index is based on an ordering of associated qualitative search terms in the query; and

executing the query to identify query results that satisfy the search criteria by;

for each event in at least a subset of the log data events and the network packet events;

for each qualitative search term of the plurality of qualitative search terms;

determining a compatibility index value for an attribute value in the event with the qualitative search term, and

multiplying the compatibility index value by a corresponding weight to obtain an intermediate value for the qualitative search term, the corresponding weight matching the order of the qualitative search term in the query, the order being with respect to the query;

generating the aggregate compatibility index for the event by averaging the intermediate value across the plurality of qualitative search terms, and

including the event in the query results when the aggregate compatibility index satisfies a threshold,

wherein the query results include at least one log data event of the event data and at least one network packet data event of the event data that have a relationship, and

wherein the query results reflect performance activity associated with the IT environment.

2. The computer-implemented method of claim 1, further comprising, for each event of the event data:

determining a time stamp for the event;

associating the time stamp with the event; and

storing the event and the time stamp in a field-searchable data store.

3. The computer-implemented method of claim 1, wherein executing the query further comprises:

using a late-binding schema to extract values from events of the event data, wherein the late-binding schema is a schema applied to stored events during execution of the query; and

using the extracted values to identify the query results that satisfy the search criteria and the plurality of qualitative search terms.

4. The computer-implemented method of claim 1, further causing display, on a graphical interface, of an indication of the query results.

5. The computer-implemented method of claim 1, wherein at least one of the components in the IT environment includes one or more of a web server, an application server, a host, a database, a firewall, a router, an operating system, a software application, a mobile device, and a sensor.

6. The computer-implemented method of claim 1, wherein the search criteria include a time range, and wherein the query results are associated with time stamps in the time range.

7. The computer-implemented method of claim 1, wherein the log data includes one or more of:

an operating system log, an application server log, a web server log, a firewall log, a software application log, and an activity log.

8. The computer-implemented method of claim 1, wherein the query is received by a search head, and wherein executing the query includes the search head sending the query to an indexer which searches an associated field-searchable data store for events responsive to the query.

9. The computer-implemented method of claim 1, wherein the query is received by a search head, and wherein executing the query includes the search head sending the query to a plurality of indexers each of which searches a respective associated field-searchable data store for events responsive to the query.

10. The computer-implemented method of claim 1, wherein the query is structured as a map-reduce operation, and wherein the map-reduce operation is performed by a distributed set of indexers each having an associated field-searchable data store.

11. The computer-implemented method of claim 1, wherein the at least one component in the IT environment from which the log data is obtained is different from the at least one component in the IT environment from which the network data is obtained.

12. The computer-implemented method of claim 1, wherein the at least one component in the IT environment from which the log data is obtained is the same as the at least one component in the IT environment from which the network data is obtained.

13. The computer-implemented method of claim 1, wherein the relationship identifies events relating to a same component in the IT environment.

14. The computer-implemented method of claim 1, wherein the relationship identifies events associated with respective time stamps in a time range identified in the query.

15. The computer-implemented method of claim 1, further comprising:

storing the event data as events in a field-searchable data store; and

wherein executing the query includes;

extracting the attribute value from events stored in the field-searchable data store; and

using a concept-mapping function to determine the capability index value associated with each extracted attribute value, the compatibility index value indicating a compatibility between the extracted attribute value and the plurality of qualitative search terms.

16. The computer-implemented method of claim 1, further comprising:

storing the event data as events in a field-searchable data store; and

wherein executing the query includes;

extracting the attribute value from events stored in the field-searchable data store;

using a concept-mapping function to determine the compatibility index value associated with each extracted attribute value, the compatibility index value indicating a compatibility between the extracted attribute value and the plurality of qualitative search terms; and

wherein the concept-mapping function is part of a context comprising a set of concept-mappings associated with a set of qualitative search terms for the attribute; and

wherein using the concept-mapping function to determine the compatibility index involves using the set of concept-mappings to determine a set of compatibility indices for the set of qualitative search terms associated with the attribute.

17. The computer-implemented method of claim 1, further comprising:

storing the event data as events in a field-searchable data store; and

wherein executing the query includes;

extracting the attribute value from events stored in the field-searchable data store; and

using a concept-mapping function to determine the compatibility index value associated with each extracted attribute value, the compatibility index value indicating a compatibility between the extracted attribute value and the plurality of qualitative search terms,

wherein the concept-mapping function is a class-specific concept-mapping function, and

wherein prior to using the concept-mapping function, the method further comprises selecting the concept-mapping function based on class information obtained from one or more fields in the event data.

18. The computer-implemented method of claim 1, further comprising:

storing the event data as events in a field-searchable data store; and

wherein executing the query includes;

extracting the attribute value from events stored in the field-searchable data store; and

using a concept-mapping function to determine the compatibility index value associated with each extracted attribute value, the compatibility index value indicating a compatibility between the extracted attribute value and the plurality of qualitative search terms,

wherein the concept-mapping function is stored as an array of data values representing a function that maps attribute values to corresponding compatibility indices.

19. The computer-implemented method of claim 1, further comprising:

storing the event data as events in a field-searchable data store; and

wherein executing the query includes;

extracting the attribute value from events stored in the field-searchable data store; and

using a concept-mapping function to determine the compatibility index value associated with each extracted attribute value, the compatibility index value indicating a compatibility between the extracted attribute value and the plurality of qualitative search terms,

wherein the concept-mapping function is part of a context comprising a set of concept-mappings associated with a set of qualitative search terms for the attribute, and

wherein the method further comprises updating the context based on instructions contained in the context.

20. The computer-implemented method of claim 1, wherein the ordering is defined by one of the plurality of qualitative search terms appearing earlier in the query being more important than one of the plurality of qualitative search terms appearing later in the query.

21. A non-transitory computer-readable storage medium storing instructions which, when executed by one or more processors, cause performance of operations comprising:

obtaining log data generated by at least one component in an information technology (IT) environment, and network packet data generated by at least one component in the IT environment;

generating, based on the log data and the network packet data, event data including log data events and network packet data events;

receiving a query that includes search criteria including;

an attribute of at least one log data event and at least one network packet data event; and

a plurality of qualitative search terms to be applied to the attribute using an aggregate compatibility index for the event data,

wherein the plurality of qualitative search terms each comprises an adjective describing the attribute, and

wherein the aggregate compatibility index is based on an ordering of associated qualitative search terms in the query; and

executing the query to identify query results that satisfy the search criteria by;

for each event in at least a subset of the log data events and the network packet events;

for each qualitative search term of the plurality of qualitative search terms;

determining a compatibility index value for an attribute value in the event with the qualitative search term, and

multiplying the compatibility index value by a corresponding weight to obtain an intermediate value for the qualitative search term, the corresponding weight matching the order of the qualitative search term in the query, the order being with respect to the query;

generating the aggregate compatibility index for the event by averaging the intermediate value across the plurality of qualitative search terms, and

including the event in the query results when the aggregate compatibility index satisfies a threshold,

wherein the query results include at least one log data event of the event data and at least one network packet data event of the event data that have a relationship, and

wherein the query results reflect performance activity associated with the IT environment.

22. The non-transitory computer-readable storage medium of claim 21, wherein the instructions which, when executed by the one or more processors, further cause performance of operations comprising:

for each event of the event data;

determining a time stamp for the event;

associating the time stamp with the event; and

storing the event and the time stamp in a field-searchable data store.

23. The non-transitory computer-readable storage medium of claim 21, wherein executing the query further comprises:

using a late-binding schema to extract values from events of the event data, wherein the late-binding schema is a schema applied to stored events during execution of the query; and

using the extracted values to identify the query results that satisfy the search criteria and the plurality of qualitative search terms.

24. The non-transitory computer-readable storage medium of claim 21, wherein the instructions which, when executed by the one or more processors, further cause display, on a graphical interface, of an indication of the query results.

25. The non-transitory computer-readable storage medium of claim 21, wherein the instructions which, when executed by the one or more processors, further cause performance of operations comprising:

storing the event data as events in a field-searchable data store; and

wherein executing the query includes;

extracting the attribute value from events stored in the field-searchable data store; and

using a concept-mapping function to determine the capability index value associated with each extracted attribute value, the compatibility index value indicating a compatibility between the extracted attribute value and the plurality of qualitative search terms.

26. The non-transitory computer-readable storage medium of claim 21, wherein the ordering is defined by one of the plurality of qualitative search terms appearing earlier in the query being more important than one of the plurality of qualitative search terms appearing later in the query.

27. An apparatus, comprising:

one or more processors;

a non-transitory computer-readable storage medium coupled to the one or more processors, the computer-readable storage medium storing instructions which, when executed by the one or more processors, causes the apparatus to;

obtain log data generated by at least one component in an information technology (IT) environment, and network packet data generated by at least one component in the IT environment;

generate, based on the log data and the network packet data, event data including log data events and network packet data events;

receive a query that includes search criteria including;

an attribute of at least one log data event and at least one network packet data event; and

a plurality of qualitative search terms to be applied to the attribute using an aggregate compatibility index for the event data,

wherein the plurality of qualitative search terms each comprises an adjective describing the attribute, and

wherein the aggregate compatibility index is based on an ordering of associated qualitative search terms in the query; and

execute the query to identify query results that satisfy the search criteria by;

for each event in at least a subset of the log data events and the network packet events;

for each qualitative search term of the plurality of qualitative search terms;

 

determining a compatibility index value for an attribute value in the event with the qualitative search term, and

 

multiplying the compatibility index value by a corresponding weight to obtain an intermediate value for the qualitative search term, the corresponding weight matching the order of the qualitative search term in the query, the order being with respect to the query;

generating the aggregate compatibility index for the event by averaging the intermediate value across the plurality of qualitative search terms, and

including the event in the query results when the aggregate compatibility index satisfies a threshold,

wherein the query results include at least one log data event of the event data and at least one network packet data event of the event data that have a relationship, and

wherein the query results reflect performance activity associated with the IT environment.

28. The apparatus of claim 27, wherein the instructions which, when executed by the one or more processors, further causes the apparatus to:

for each event of the event data;

determine a time stamp for the event;

associate the time stamp with the event; and

store the event and the time stamp in a field-searchable data store.

29. The apparatus of claim 27, wherein executing the query further comprises:

using a late-binding schema to extract values from events of the event data, wherein the late-binding schema is a schema applied to stored events during execution of the query; and

using the extracted values to identify the query results that satisfy the search criteria and the plurality of qualitative search terms.

30. The apparatus of claim 27, wherein the instructions which, when executed by the one or more processors, further causes the apparatus to:

store the event data as events in a field-searchable data store; and

wherein executing the query includes;

extracting the attribute value from events stored in the field-searchable data store; and

using a concept-mapping function to determine the capability index value associated with each extracted attribute value, the compatibility index value indicating a compatibility between the extracted attribute value and the plurality of qualitative search terms.

31. The apparatus of claim 27, wherein the ordering is defined by one of the plurality of qualitative search terms appearing earlier in the query being more important than one of the plurality of qualitative search terms appearing later in the query.