Analyzing log streams based on correlations between data structures of defined node types

• US 9,779,005 B2
• Filed: 06/24/2014
• Issued: 10/03/2017
• Est. Priority Date: 06/24/2014
• Status: Active Grant

• Current Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Original Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Inventors: Gukal, Sreenivas, Vyas, Sanjay
• Primary Examiner(s): Boccio, Vincent F
• Application Number:
  US14/313,075
• Publication Number:
  US 20150370842A1
• Time in Patent Office: 1,197 Days
• Field of Search: 707709, 707711, 707749, 707750, 707751
• CPC Class Codes: G06F 11/3051 Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs verification or detection of system hardware configuration G06F11/2247
• CPC Class Codes: G06F 11/3072 where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06F 11/321 Display for diagnostics, e.g. diagnostic result display, self-test user interface
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 17/40 Data acquisition and logging for input to computer G06F3/00
• CPC Class Codes: G06F 2201/86 Event-based monitoring

Abstract

A method by a log stream analysis computer includes identifying records of log streams within a log repository containing a defined term. The log streams are generated by respective software sources executed by the host nodes. Similarity values are determined to indicate similarity between content of the records containing the defined term. A term node is generated to contain a data structure that identifies the defined term and lists identities of the records and corresponding ones of the similarity values. Related log stream analysis computers are disclosed.

Claims

1. A method by a log stream analysis computer comprising:

identifying a set of records within log streams within a log repository containing a defined term and which are all constrained to being within a time period, wherein the log streams are generated by respective software sources executed by the host nodes characterizing performance data captured by the respective software sources over the time period;

determining similarity values that indicate similarity between content of the records containing the defined term in the time period;

generating a term node comprising a data structure that identifies the defined term and lists identities of the records containing the defined term in the time period and corresponding ones of the similarity values in the time period;

identifying hosts within the log streams;

for each of a plurality of the hosts not corresponding to an existing host node, generating a host node comprising a data structure that identifies the host and lists an identity of a hardware configuration of the host, an identity of the software source of the log stream, and a defined type of the software source; and

determining correlation between records of the log streams within the log repository based on content of the data structures of the host nodes.

2. The method of claim 1 further comprising:

determining correlation between the records of the log streams within the log repository based on content of the data structure of the term node and a defined correlation rule.

3. The method of claim 1 further comprising:

repeating for each of a plurality of defined terms, the identifying records, the determining similarity values, and the generating a term node, wherein different ones of the term nodes correspond to different ones of the defined terms; and

determining correlation between content of records of log streams within the log repository based on content of the data structure of the term nodes and a defined correlation rule.

4. The method of claim 3, further comprising:

providing information based on content of the data structures of a plurality of term nodes for display on a display device;

receiving a selection of one of the term nodes displayed on the display device; and

determining correlation between records of the log streams within the log repository based on comparison of content of the data structure of the selected one of the term nodes to content of the data structures of other term nodes.

5. The method of claim 4, further comprising:

selecting another term node based on the comparison of content of the data structure of the selected one of the term nodes to content of the data structure of the other term node satisfying a defined correlation rule; and

providing information based on content of the data structure of the other term node for display on the display device.

6. The method of claim 1 wherein:

determining similarity values comprises calculating min-wise independent permutation locality sensitive hashing (MinHash) values of content of the records containing the defined term; and

generating the term node comprises storing the MinHash values associated with corresponding ones of the records in the data structure of the term node.

7. The method of claim 6 further comprising:

counting a number of occurrences of the defined term in each of the records containing the defined term; and

generating the term node comprises storing the number of occurrences associated with corresponding ones of the records in the data structure of the term node.

8. The method of claim 1,

further comprising based on the defined term not being present in a data dictionary containing lists of terms and corresponding term value identifiers, adding the defined term and a corresponding term value identifier to the data dictionary;

and

wherein the defined term is identified in the data structure by the corresponding term value identifier from the data dictionary.

9. The method of claim 1, further comprising:

providing information based on content of the data structures of a plurality of host nodes for display on a display device;

receiving a selection of one of the host nodes displayed on the display device; and

determining correlation between records of the log streams within the log repository based on comparison of content of the data structure of the selected one of the host nodes to content of the data structures of other host nodes.

10. The method of claim 9, further comprising:

selecting another host node based on the comparison of content of the data structure of the selected one of the host nodes to content of the data structure of the other host node satisfying a defined correlation rule; and

providing information based on content of the data structure of the other host node for display on the display device.

11. A method by a log stream analysis computer comprising:

identifying a set of records within log streams within a log repository containing a defined term and which are all constrained to being within a time period, wherein the log streams are generated by respective software sources executed by the host nodes characterizing performance data captured by the respective software sources over the time period;

determining similarity values that indicate similarity between content of the records containing the defined term in the time period;

generating a term node comprising a data structure that identifies the defined term and lists identities of the records containing the defined term in the time period and corresponding ones of the similarity values in the time period;

identifying software sources within the log streams;

for each of a plurality of the software sources having a defined type not corresponding to an existing source type node, generating a source type node containing a data structure that identifies the defined type of the software source and lists identifiers of records of one of the log streams generated by the software source, identifies the software source, and identifies one of the host nodes executing the software source; and

determining correlation between records of the log streams within the log repository based on content of the data structures of the source type nodes.

12. The method of claim 11, further comprising:

providing information based on content of the data structures of a plurality of source type nodes for display on a display device;

receiving a selection of one of the source type nodes displayed on the display device; and

determining correlation between records of the log streams within the log repository based on comparison of content of the data structure of the selected one of the source type nodes to content of the data structures of other source type nodes.

13. The method of claim 12, further comprising:

selecting another source type node based on the comparison of content of the data structure of the selected one of the source type nodes to content of the data structure of the other source type node satisfying a defined correlation rule; and

providing information based on content of the data structure of the other source type node for display on the display device.

14. A log stream analysis computer comprising:

a processor; and

a memory coupled to the processor, the memory comprising a non-transitory computer readable storage medium having computer readable program code embodied in the medium that when executed by the processor causes the processor to perform operations comprising;

identifying a set of records within log streams within a log repository containing a defined term and which are all constrained to being within a time period, wherein the log streams are generated by respective software sources executed by the host nodes characterizing performance data captured by the respective software sources over the time period;

determining similarity values that indicate similarity between content of the records containing the defined term in the time period;

generating a term node comprising a data structure that identifies the defined term in the time period and lists identities of the records and corresponding ones of the similarity values in the time period;

identifying one of hosts and software sources within the log streams;

responsive to identifying hosts within the log streams, for each of a plurality of the hosts not corresponding to an existing host node, generating a host node containing a data structure that identifies the host and lists an identity of a hardware configuration of the host, an identity of the software source of the log stream, and a defined type of the software source;

responsive to identifying software sources within the log streams, for each of a plurality of the software sources having a defined type not corresponding to an existing source type node, generating a source type node containing a data structure that identifies the defined type of the software source and lists identifiers of records of one of the log streams generated by the software source, identifies the software source, and identifies one of the host nodes executing the software source; and

determining correlation between records of the log streams based on content of the data structures of the term nodes and based on content of one of the data structures of the host nodes and the data structures of the source type nodes.

15. The log stream analysis computer of claim 14, wherein the operations further comprise:

repeating for each of a plurality of defined terms, the identifying records, the determining similarity values, and the generating a term node, wherein different ones of the term nodes correspond to different ones of the defined terms; and

determining correlation between content of records of log streams within the log repository based on content of the data structure of the term nodes and a defined correlation rule.

16. The log stream analysis computer of claim 15, wherein the operations further comprise:

providing information based on content of the data structures of a plurality of term nodes for display on a display device;

receiving a selection of one of the term nodes displayed on the display device; and

determining correlation between records of the log streams within the log repository based on comparison of content of the data structure of the selected one of the term nodes to content of the data structures of other term nodes.

17. The log stream analysis computer of claim 16, wherein the operations further comprise:

selecting another term node based on the comparison of content of the data structure of the selected one of the term nodes to content of the data structure of the other term node satisfying a defined correlation rule; and

providing information based on content of the data structure of the other term node for display on the display device.

18. The log stream analysis computer of claim 15, wherein:

determining similarity values comprises calculating min-wise independent permutation locality sensitive hashing (MinHash) values of content of the records containing the defined term; and

generating the term node comprises storing the MinHash values associated with corresponding ones of the records in the data structure of the term node.

19. The log stream analysis computer of claim 15, wherein the operations further comprise:

generating other ones of the term node for other defined terms contained in records of the log streams.