ANOMALY CLASSIFICATION, ANALYTICS AND RESOLUTION BASED ON ANNOTATED EVENT LOGS

• US 20160283310A1
• Filed: 03/24/2015
• Published: 09/29/2016
• Est. Priority Date: 03/24/2015
• Status: Active Grant

• Current Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Original Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Inventors: Greenspan, Steven, Mankovskii, Serguei, Velez-Rojas, Maria
• Granted Patent: US 10,133,614 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0781 Error filtering or prioritizing based on a policy defined by the user or on a policy defined by a hardware/software module, e.g. according to a severity level
• CPC Class Codes: G06F 11/34 Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: H04L 41/00 Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
• CPC Class Codes: H04L 41/064 involving time analysis
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications

Abstract

Operational event loggings and operational alarm productions within a running multiserver data processing system are automatically and repeatedly sampled and co-associated with one another so as to build annotated logs that can be used by post-process analytics for filling in mappings thereof into an anomalies versus parameters mapping space and for keeping track of unusual changes in the mappings or their rates where the unusual changes can be indicative of emerging new problems of significance within the system.

Claims

1. A machine-implemented method for keeping track in an anomalies versus parameters mapping space of previously identified and emerging anomalies of a data processing system, the method comprising:

running a first section of the data processing system where the first section includes a section alarming subsystem and a section behaviors logging subsystem, the section alarming subsystem being configured to generate alarms for alarm-worthy events within the first section, the section behaviors logging subsystem being configured to generate a log of monitored behaviors within the first section;

logically co-associating recently logged behaviors of the generated log produced by the section behaviors logging subsystem with substantially cotemporaneous alarms generated by the section alarming subsystem and recording the logical associations;

building an annotated log comprised of the logically co-associated logged behaviors and substantially cotemporaneous alarms;

using the annotated log to keep track in a corresponding first anomalies versus parameters mapping space of previously identified as routine and emerging anomalies of the first section of the data processing system; and

automatically repeating said co-associating building and using steps while the first section of the data processing system continues to run.

2. The machine-implemented method of claim 1 and further comprising:

creating a behavior mimicking copy of the section alarming subsystem, the behavior mimicking copy having accessible internal logic structures configured to mimic output behaviors of the section alarming subsystem;

3. The machine-implemented method of claim 1 wherein:

the first anomalies versus parameters mapping space is defined by a database storing alarmed sample points (ASP'"'"'s) as points within a multi-parameters coordinate space where each alarmed sample point (ASP) correlates to one or more of cotemporaneous ones of the temporally corresponding generatings of alarms by the section alarming subsystem.

4. The machine-implemented method of claim 3 wherein:

the first anomalies versus parameters mapping space is further defined by the database further storing non-alarmed sample points (NASP'"'"'s) as points within the multi-parameters coordinate space where each non-alarmed sample point (NASP) correlates to an event logged time when there are no generatings of alarms by the section alarming subsystem.

5. The machine-implemented method of claim 3 wherein:

the first section of the data processing system includes a first data processing unit, a first data storage unit and a first data input/output communicating unit; and

the recently logged behaviors of the generated log include a data processing rate of the first data processing unit, a data access rate of the first data storage unit and a data communicating rate of the first data input/output communicating unit.

6. The machine-implemented method of claim 5 wherein:

the section alarming subsystem of the first section generates a low storage access rate alarm when the data access rate of the first data storage unit is below a predetermined first storage access rate threshold.

7. The machine-implemented method of claim 6 wherein:

the section alarming subsystem of the first section generates a high storage access rate alarm when the data access rate of the first data storage unit is above a predetermined second storage access rate threshold.

8. The machine-implemented method of claim 7 wherein:

the section alarming subsystem of the first section does not generate a storage access rate alarm when the data access rate of the first data storage unit is between the predetermined first and second storage access rate thresholds; and

in response to the section alarming subsystem not generating the storage access rate alarms, the first anomalies versus parameters mapping space is defined within the database by corresponding storing of non-alarmed data access sample points (daNASP'"'"'s) as points within the multi-parameters coordinate space where each non-alarmed data access sample point (daNASP) correlates to a time when there are no generatings of data access alarms by the section alarming subsystem.

9. The machine-implemented method of claim 8 wherein:

an emerging data access anomaly is flagged for an area of the first anomalies versus parameters mapping space when that area was previously mapped as having non-alarmed data access sample points (daNASP'"'"'s) and there is a current change wherein the area starts being filled with alarmed data access sample points (daASP'"'"'s) indicative of generatings of data access alarms by the section alarming subsystem for that previously not-alarmed area.

10. The machine-implemented method of claim 3 and further comprising:

cotemporaneous with the running of the first section, running a second section of the data processing system where the first and second sections are operatively influential one at least on the other or the first and second sections are operatively influenced by a third section of the data processing system and where the second section includes a corresponding second section alarming subsystem and a corresponding second section behaviors logging subsystem, the second section alarming subsystem being configured to generate alarms for alarm-worthy events within the second section, the second section behaviors logging subsystem being configured to generate a corresponding second log of monitored behaviors within the second section;

logically co-associating recently logged behaviors of the generated second log produced by the second section behaviors logging subsystem with substantially cotemporaneous alarms generated by the second section alarming subsystem and recording the logical associations of the second section;

building an annotated second log that correlates the recently logged behaviors of the second section to temporally corresponding generatings and non-generatings of alarms by the second section alarming subsystem;

using the second annotated log for keeping track in a corresponding second anomalies versus parameters mapping space of previously identified and emerging anomalies of the second section of the data processing system;

automatically repeating said co-associating, building and using steps for the second section while the second section of the data processing system continues to run; and

automatically correlating changes in one of the first and second anomalies versus parameters mapping spaces with cotemporaneous changes in the other of the anomalies versus parameters mapping spaces.

11. A data processing system configured to keep track in an anomalies versus parameters mapping space of previously identified and emerging anomalies of the data processing system, the data processing system comprising:

a plurality of sections including a first section having a section alarming subsystem and a section behaviors logging subsystem, the section alarming subsystem being configured to generate alarms for alarm-worthy events within the first section, the section behaviors logging subsystem being configured to generate a log of monitored behaviors within the first section;

an annotated log storing database that is configured to indicate correlations for respective ones of the system sections between the recently logged behaviors and temporally corresponding generatings and non-generatings of alarms by the respective section alarming subsystem;

an annotated log builder, coupled to the database and configured to automatically repeatedly for respective ones of the system sections, add to the stored and annotated log additional samples of correlations between recently logged behaviors and temporally corresponding generatings and non-generatings of alarms by the respective section alarming subsystem of the respective section; and

a post-process analytics unit that is operatively coupled to respective ones of the annotated logs stored in the database for the respective sections and is configured to automatically repeatedly map into a first anomalies versus parameters mapping space, sample point indicators indicative of respective coordinates in the first anomalies versus parameters mapping space corresponding to plural parameters associated with each generating and non-generating of an alarm by the section alarming subsystem of a first of the sections in response to the recently logged behaviors of the generated log produced by the section behaviors logging subsystem of that first section;

wherein the post-process analytics unit is configured to flag out abnormal changes over time in the automatically repeatedly mapped first anomalies versus parameters mapping space.

12. The data processing system of claim 11 wherein:

the sample point indicators of the first anomalies versus parameters mapping space include alarmed sample points (ASP'"'"'s) that each indicates as coordinates within a first multi-parameters coordinate space, where each such indicated sample point correlates to one or more of cotemporaneous ones of temporally corresponding generatings of alarms by the section alarming subsystem.

13. The data processing system of claim 12 wherein:

the sample point indicators of the first anomalies versus parameters mapping space further include non-alarmed sample points (NASP'"'"'s) that each indicates as coordinates within a first multi-parameters coordinate space, where each such indicated sample point correlates to one or more of cotemporaneous ones of temporally corresponding non-generatings of alarms by the section alarming subsystem.

14. The data processing system of claim 12 wherein:

the first section of the data processing system includes a first data processing unit, a first data storage unit and a first data input/output communicating unit; and

the recently logged behaviors of the generated log include a data processing rate of the first data processing unit, a data access rate of the first data storage unit and a data communicating rate of the first data input/output communicating unit.

15. The data processing system of claim 14 wherein:

the section alarming subsystem of the first section is configured to generate a low storage access rate alarm when the data access rate of the first data storage unit is below a predetermined first storage access rate threshold.

16. The data processing system of claim 15 wherein:

the section alarming subsystem of the first section is configured to generate a high storage access rate alarm when the data access rate of the first data storage unit is above a predetermined second storage access rate threshold.

17. The data processing system of claim 16 wherein:

the section alarming subsystem of the first section does not generate a storage access rate alarm when the data access rate of the first data storage unit is between the predetermined first and second storage access rate thresholds; and

in response to the section alarming subsystem not generating the storage access rate alarms, the post-process analytics unit is configured to map into the first anomalies versus parameters mapping space, corresponding non-alarmed data access sample points (daNASP'"'"'s) as points within the first multi-parameters coordinate space where each non-alarmed data access sample point (daNASP) correlates to a time when there are no generatings of data access alarms by the section alarming subsystem.

18. A machine-implemented method for adaptively responding to previously identified and emerging anomalies of a data processing system, the method comprising:

running each of plural and intercoupled sections of the data processing system where each first section includes a respective section alarming subsystem and a respective section behaviors logging subsystem, the respective section alarming subsystem being configured to generate alarms for alarm-worthy events within the respective section, the respective section behaviors logging subsystem being configured to generate a log of monitored behaviors within the respective section;

for each running section, co-associating recently logged behaviors of the respectively generated log produced by the respective section behaviors logging subsystem with substantially contemporaneous alarms generated by the section alarming subsystem and recording the co-associations;

for each running section, building a respective annotated log that includes the recorded co-associations between the recently logged respective behaviors and the temporally corresponding generatings and non-generatings of alarms by the respective section alarming subsystem;

for each running section, using the respective annotated log to keep track within a respective anomalies versus parameters mapping space of previously identified and emerging anomalies of the respective section of the data processing system;

automatically repeating said co-associating, building and using steps of each respective section while the respective section of the data processing system continues to run;

keeping track in each of the respective anomalies versus parameters mapping spaces of changes over time in the mapping locations and/or the rates of sample point additions to the mappings; and

adaptively reallocating resources to sections based on the tracked changes within the respective anomalies versus parameters mapping spaces.

19. The machine-implemented method of claim 18 wherein:

the adaptively reallocating of resources includes automatically repeatedly determining for each tracked section if a per unit usage is above a predetermined maximum threshold for such a unit whereby a desired margin of safety is compromised and in response to a determining that the per unit usage is above the corresponding predetermined maximum threshold, adding a further such unit to the respective section.

20. The machine-implemented process of claim 18 and further comprising:

building one or more hierarchical parent annotated logs that include the recorded co-associations between the recently logged respective behaviors and the temporally corresponding generatings and non-generatings of alarms by the respective section alarming subsystems of two or more of the system sections.