SCALABLE AND ACCURATE MINING OF CONTROL FLOW FROM EXECUTION LOGS ACROSS DISTRIBUTED SYSTEMS

• US 20170068709A1
• Filed: 09/09/2015
• Published: 03/09/2017
• Est. Priority Date: 09/09/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Dasgupta, Gargi Banerjee, Mandal, Atri, Raghavan, Sriram, Subramanian, Suriya, Nandi, Animesh, Neogi, Anindya
• Granted Patent: US 10,140,287 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/30 Monitoring
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3082 the data filtering being achieved by aggregating or compressing the monitored data
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 40/186 Templates
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

Methods and arrangements for efficiently mining a control flow graph from execution logs of a distributed system. Using at least one text clustering technique, two text clusters are generated from the plurality of execution logs. At least one approximate template is generated based on the at least two text clusters. At least one refined template is created via refining the at least one approximate template using multimodal sequencing. The control flow graph is created based on the at least one refined template. An anomaly is detected in the control flow graph.

Claims

1. A method of efficiently mining a control flow graph from execution logs of a distributed system, said method comprising:

utilizing at least one processor to execute computer code that performs the steps of;

receiving a plurality of execution logs;

generating, using at least one text clustering technique, at least two text clusters, from the plurality of execution logs;

generating at least one approximate template based on the at least two text clusters;

creating at least one refined template via refining the at least one approximate template using multimodal sequencing;

creating the control flow graph, based on the at least one refined template; and

detecting at least one anomaly in the control flow graph.

2. The method of claim 1, wherein the plurality of execution logs comprises raw log lines.

3. The method of claim 1, wherein said creating the control flow graph comprises identifying, with respect to the at least one refined template, at least one temporally co-occurring template.

4. The method of claim 3, wherein said creating the control flow graph further comprises ordering a plurality of temporally co-occurring templates.

5. The method according to claim 1, wherein said detecting comprises:

comparing the created control flow graph to at least one previously stored control flow graph;

identifying, based on said comparing, at least one anomaly; and

wherein the at least one identified anomaly is selected from the group consisting of;

a sequence anomaly and a distribution anomaly.

6. The method according to claim 1, wherein the at least one text-clustering technique comprises a technique selected from the group consisting of:

an edit-distance technique and a dictionary-based logline parameterization.

7. The method according to claim 6, wherein the dictionary-based logline parameterization comprises identifying keywords and parameters for each of the plurality of execution logs.

8. The method according to claim 7, wherein said identifying comprises employing a frequency analysis of words for each of the plurality of execution logs.

9. The method according to claim 1, wherein the multimodal sequencing comprises at least one of:

temporal-vicinity and text.

10. An apparatus for efficiently mining a control flow graph from execution logs of a distributed system, said apparatus comprising:

at least one processor; and

a computer readable storage medium having computer readable program code embodied therewith and executable by the at least one processor, the computer readable program code comprising;

computer readable program code that receives a plurality of execution logs;

computer readable program code that generates, using at least one text clustering technique, at least two text clusters, from the plurality of execution logs;

computer readable program code that creates at least one refined template via refining the at least one approximate template using multimodal sequencing;

computer readable program code that creates the control flow graph, based on the at least one refined template; and

computer readable program code that detects at least one anomaly in the control flow graph.

11. A computer program product to efficiently mine a control flow graph from execution logs of a distributed system, said computer program comprising:

at least one processor; and

at least one processor; and

a computer readable storage medium having computer readable program code embodied therewith and executable by the at least one processor, the computer readable program code comprising;

computer readable program code that receives a plurality of execution logs;

computer readable program code that generates, using at least one text clustering technique, at least two text clusters, from the plurality of execution logs;

computer readable program code that creates at least one refined template via refining the at least one approximate template using multimodal sequencing;

computer readable program code that creates the control flow graph, based on the at least one refined template; and

computer readable program code that detects at least one anomaly in the control flow graph.

12. The computer program product according to claim 11, wherein the plurality of execution logs comprises raw log lines.

13. The computer program product according to claim 11, wherein the creating of the control flow graph comprises, identifying, with respect to the at least one refined template, at least one temporally co-occurring template.

14. The computer program product according to claim 13, wherein the creating the control flow graph further comprises ordering a plurality of temporally co-occurring templates.

15. The computer program product according to claim 14, wherein the detecting comprises:

comparing the created control flow graph to at least one previously stored control flow graph;

identifying, based on said comparing, at least one anomaly; and

wherein the at least one identified anomaly is selected from the group consisting of;

a sequence anomaly and a distribution anomaly.

16. The computer program product according to claim 11, wherein the at least one text-clustering technique comprises a technique selected from the group consisting of:

an edit-distance technique and a dictionary-based logline parameterization.

17. The computer program product according to claim 16, wherein the dictionary-based logline parameterization comprises identifying keywords and parameters for each of the plurality of execution logs.

18. The computer program product according to claim 17, wherein the identifying comprises employing a frequency analysis of words for each of the plurality of execution logs.

19. The computer program product according to claim 11, wherein the multimodal sequencing comprises at least one of:

temporal-vicinity and text.

20. A method of efficiently mining the control flow graph from execution logs of a distributed system, said method comprising:

utilizing at least one processor to execute computer code that performs the steps of;

receiving a plurality of execution logs;

mining at least one template from the plurality of execution logs in the first-phase;

said mining comprising creating at least one template, via employing a two-stage template mining technique;

said first-stage creating approximate-templates via a dictionary based logline transformation in order to attain scalability and

said second-stage refining the mined approximate-templates by leveraging the multimodal (text+temporal-vicinity) signature of each approximate-template; and

generating the control-flow graph between the mined templates in the second-phase via a two-stage technique;

said first-stage creating for each template, the set of its temporally co-occurring templates, referred to as its Nearest-Neighbor-Group, by leveraging the time-series of occurrence of each template; and

said second-stage, in a single-pass of the logstream, determining for each template, its immediate predecessors/successors by tracking predecessors/successors on the projected logstream on the Nearest-Neighbor group of the template, and stitching the mined successors of each template to construct the desired control flow graph.