Anomaly classification, analytics and resolution based on annotated event logs

• US 10,133,614 B2
• Filed: 03/24/2015
• Issued: 11/20/2018
• Est. Priority Date: 03/24/2015
• Status: Active Grant

• Current Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Original Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Inventors: Mankovskii, Serguei, Greenspan, Steven, Velez-Rojas, Maria
• Primary Examiner(s): Truong, Loan L. T.
• Application Number:
  US14/666,665
• Publication Number:
  US 20160283310A1
• Time in Patent Office: 1,337 Days
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0781 Error filtering or prioritizing based on a policy defined by the user or on a policy defined by a hardware/software module, e.g. according to a severity level
• CPC Class Codes: G06F 11/34 Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: H04L 41/00 Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
• CPC Class Codes: H04L 41/064 involving time analysis
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications

Abstract

Operational event loggings and operational alarm productions within a running multiserver data processing system are automatically and repeatedly sampled and co-associated with one another so as to build annotated logs that can be used by post-process analytics for filling in mappings thereof into an anomalies versus parameters mapping space and for keeping track of unusual changes in the mappings or their rates where the unusual changes can be indicative of emerging new problems of significance within the system.

Claims

1. A machine-implemented method of separately dealing with emerging and possibly not routine anomalies of a data processing system that could be of significance to continuing operations of the data processing system, the data processing system being subdivided into a plurality of sections with each section comprising intercoupled local resources including one or more local data processing units and one or more local data storage units, wherein two or more of the plural sections each respectively includes a respective section behaviors logging subsystem configured to automatically log monitored behaviors within the respective section and a respective section alarming subsystem configured to automatically generate alarms for alarm worthy events within the respective section, wherein said routine anomalies and said emerging and possibly not routine anomalies are not catastrophic failures, the method comprising:

running a first section among said plural sections of the data processing system where the first section includes as its respective section alarming subsystem, a first section alarming subsystem and includes as its respective section behaviors logging subsystem, a first section behaviors logging subsystem, the first section alarming subsystem being configured to generate alarms for non-catastrophic alarm-worthy events detected within the first section, the section behaviors logging subsystem being configured to generate a log of monitored behaviors within the first section;

logically co-associating recently logged behaviors of the generated log produced by the first section behaviors logging subsystem with substantially cotemporaneous alarms generated by the first section alarming subsystem;

building an annotated log comprised of the logically co-associated logged behaviors and the substantially cotemporaneous alarms;

using the annotated log to update a corresponding anomalies versus parameters first mapping space populated by sample points representing previously identified as routine anomalies of the first section of the data processing system by adding recent, alarm-including sample point entries from the annotated log into the first mapping space as recently logged ones of alarmed sample points (ASP'"'"'s);

determining if the recently logged ASP'"'"'s map into a first region of the first mapping space occupied by older ASP'"'"'s associated with the identified as routine anomalies or if the recently logged ASP'"'"'s map into a different region of the first mapping space, where the ASP'"'"'s which map into the different region can represent newly emerging and possibly non-routine anomalies;

automatically repeating said logically co-associating step, said building step, said using step and said determining step while the first section of the data processing system continues to run; and

automatically responding to said determining that the recently logged ASP'"'"'s map into the different region and can thus represent newly emerging and possibly non-routine anomalies that could be of significance to operations of the data processing system, the automatic responding being separate from responses to known-to-be-routine anomalies and separate from responses to detected catastrophic failures.

2. The machine-implemented method of claim 1 and further comprising:

using the annotated log for creating a behavior mimicking model of the first section alarming subsystem, the behavior mimicking model having accessible internal logic structures configured to mimic output behaviors of the first section alarming subsystem.

3. The machine-implemented method of claim 1 wherein:

the anomalies versus parameters first mapping space is defined by a database storing first data representing the alarmed sample points (ASP'"'"'s) as points within a multi-parameters coordinate space where each alarmed sample point (ASP) correlates to one or more of the temporally corresponding generatings of alarms by the first section alarming subsystem.

4. The machine-implemented method of claim 3 wherein:

the anomalies versus parameters first mapping space is further defined by the database further storing second data representing non-alarmed sample points (NASP'"'"'s) as points within the multi-parameters coordinate space where each non-alarmed sample point (NASP) correlates to an event logging time when there are no temporally corresponding generatings of alarms by the first section alarming subsystem.

5. The machine-implemented method of claim 3 wherein:

in addition to its respective one or more local data processing units and its respective one or more local data storage units, the first section of the data processing system includes a first data input/output communicating unit; and

the recently logged behaviors of the generated log include a data processing rate of at least one of the respective local data processing units, a data access rate of at least one of the respective local data storage units and a data communicating rate of the first data input/output communicating unit.

6. The machine-implemented method of claim 5 wherein:

the first section alarming subsystem of the first section generates a low storage access rate alarm when the data access rate of the at least one of the respective local data storage units is below a predetermined first storage access rate threshold.

7. The machine-implemented method of claim 6 wherein:

the first section alarming subsystem of the first section generates a high storage access rate alarm when the data access rate of the at least one of the respective local data storage units is above a predetermined second storage access rate threshold.

8. The machine-implemented method of claim 7 wherein:

the first section alarming subsystem of the first section does not normally generate a storage access rate alarm when the data access rate of the first data storage unit is between the predetermined first and second storage access rate thresholds; and

in response to the section alarming subsystem not generating the storage access rate alarms, the anomalies versus parameters first mapping space is defined within the database by a corresponding storing of non-alarmed data access sample points (daNASP'"'"'s) as points within the multi-parameters coordinate space where each such corresponding non-alarmed data access sample point (daNASP) correlates to a time when there are no generatings of data access alarms by the first section alarming subsystem.

9. The machine-implemented method of claim 8 wherein:

an emerging data access anomaly is flagged for an area of the anomalies versus parameters first mapping space when that area was previously mapped as having only non-alarmed data access sample points (daNASP'"'"'s) and there is a current change wherein the area starts being filled with alarmed data access sample points (daASP'"'"'s) indicative of generatings of data access alarms by the first section alarming subsystem for that previously not-alarmed area of the anomalies versus parameters first mapping space.

10. The machine-implemented method of claim 3 and further comprising:

cotemporaneous with the running of the first section, running a second section of the data processing system where the first and second sections are operatively influential one at least on the other or the first and second sections are operatively influenced by a third section of the data processing system and where the second section includes as its respective section alarming subsystem, a second section alarming subsystem and includes as its respective section behaviors logging subsystem, a second section behaviors logging subsystem, the second section alarming subsystem being configured to generate alarms for alarm-worthy events detected within the second section, the second section behaviors logging subsystem being configured to generate a corresponding second log of monitored behaviors within the second section;

logically co-associating recently logged behaviors of the generated second log produced by the second section behaviors logging subsystem with substantially cotemporaneous alarms generated by the second section alarming subsystem of the second section of the data processing system;

building an annotated second log that correlates the recently logged behaviors of the second section to temporally corresponding generatings and non-generatings of alarms by the second section alarming subsystem;

using the second annotated log to update a corresponding anomalies versus parameters second mapping space of previously identified anomalies of the second section of the data processing system by adding recent, alarm-including sample point entries from the second annotated log into the second mapping space as recently logged ones of alarmed sample points (ASP'"'"'s);

determining if the recently logged ASP'"'"'s map into a respective first region of the second mapping space occupied by older ASP'"'"'s associated with the identified as routine anomalies for the second section or if the recently logged ASP'"'"'s map into a respective different region of the second mapping space, where the ASP'"'"'s which map into the respective different region can represent newly emerging and possibly non-routine anomalies;

automatically repeating for the second section of the data processing system, said respective logically co-associating step, said respective building step, said respective using step and said respective determining step while the second section of the data processing system continues to run;

automatically responding to said determining for the second section that the recently logged ASP'"'"'s map into the different region for the second section and can thus represent newly emerging and possibly non-routine anomalies for the second section that could be of significance to operations of the data processing system, the automatic responding for the second section being separate from responses to known-to-be-routine anomalies and separate from responses to detected catastrophic failures; and

automatically correlating changes in one of the first and second mapping spaces with cotemporaneous changes in the other of the first and second mapping spaces.

11. The machine-implemented method of claim 1 wherein for a case in which said automatic repeating of the logical co-associating step, of the building step and of the using step causes a region within the first mapping space that was previously free of a specific type of logged ASP'"'"'s to start to become filled with the specific type of logged ASP'"'"'s, automatically deeming the previously free region to be a region routinely populated by the specific type of logged ASP'"'"'s.

12. The machine-implemented method of claim 1 wherein the automatic responding to said determining that the recently logged ASP'"'"'s map into the different region comprises:

determining from placement of the recently logged ASP'"'"'s into the different region that the recently logged ASP'"'"'s indicate a problem within the first section that can be ameliorated by re-allocation of a particular type of resource; and

re-allocating the particular resource so as to ameliorate the problem within the first section.

13. The machine-implemented method of claim 1 wherein the re-allocating of the particular resource so as to ameliorate the problem within the first section comprises:

adding one or more storage units to the first section.

14. The machine-implemented method of claim 1 wherein the re-allocating of the particular resource so as to ameliorate the problem within the first section comprises:

adding one or more data processing units to the first section.

15. A data processing system configured to separately deal with emerging and possibly not routine anomalies of the data processing system that could be of significance to continuing operations of the data processing system, wherein said routine anomalies and said emerging and possibly not routine anomalies are not catastrophic failures, the data processing system being subdivided into a plurality of sections with each section comprising intercoupled local resources including one or more local data processing units and one or more local data storage units, wherein two or more of the plural sections each respectively includes a respective section behaviors logging subsystem configured to automatically log monitored behaviors within the respective section and generate a respective local log and each of the two or more sections respectively includes a respective section alarming subsystem configured to automatically generate alarms for alarm worthy events within the respective section, the data processing system comprising:

an annotated logs storing database storing one or more respective annotated logs that respectively indicate correlations for respective ones of the system sections between recently logged behaviors of the respective system sections as recently recorded in the respective local log of the respective section and temporally corresponding generatings and non-generatings of alarms by the respective section alarming subsystems of the respective system sections;

an annotated logs builder, coupled to the database and configured to automatically repeatedly for respective ones of the system sections, add to the respective stored and annotated logs of the respective system sections additional samples of correlations between recently logged behaviors logged in the respective local logs and temporally corresponding generatings and non-generatings of alarms by the respective section alarming subsystems of the respective sections; and

a post-process analytics portion of the data processing system that is operatively coupled to respective ones of the annotated logs stored in the database for the respective sections and is configured to automatically repeatedly map into respective anomalies versus parameters mapping spaces of respective ones of the system sections, sample point indicators indicative of respective coordinates in the respective mapping space corresponding to plural parameters associated with each generating and non-generating of alarms by the respective section alarming subsystem of the respective one among the sections and corresponding to substantially cotemporaneous, recently logged behaviors of the respective local log produced by the section behaviors logging subsystem of that respective section;

wherein the post-process analytics portion is configured to flag out abnormal changes over time in the automatically repeatedly made mappings of the sample point indicators into the respective anomalies versus parameters mapping spaces, where the flagged out abnormal changes include those representing emerging and possibly not routine anomalies that are not catastrophic failures.

16. The data processing system of claim 15 wherein:

the sample point indicators of the anomalies versus parameters mapping spaces include alarmed sample points (ASP'"'"'s) that each indicates an association between its corresponding alarmed entry in the annotated log of the respective system section and cross-correlated coordinates within the respective multi-parameters coordinate space, where each such indicated as-alarmed sample point correlates to one or more of cotemporaneous ones of temporally corresponding generatings of alarms by the respective section alarming subsystem of the respective section.

17. The data processing system of claim 16 wherein:

the sample point indicators of the respective anomalies versus parameters mapping spaces further include non-alarmed sample points (NASP'"'"'s) that each indicates as coordinates within the respective multi-parameters coordinate space, where each such NASP sample point correlates to one or more of cotemporaneous ones of temporally corresponding non-generatings of alarms by the respective section alarming subsystem of the respective section.

18. A machine-implemented method for adaptively and separately responding to emerging and potentially non-routine anomalies of a running data processing system that also experiences routine anomalies, wherein said routine anomalies and said emerging and possibly not routine anomalies are not catastrophic failures, the method comprising:

running each of plural and inter-coupled sections of the data processing system where each respective section includes a respective section alarming subsystem and a respective section behaviors logging subsystem, the respective section alarming subsystem being configured to generate alarms for non-catastrophic alarm-worthy events within the respective section, the respective section behaviors logging subsystem being configured to generate a log of monitored behaviors within the respective section;

for each respective one of the running sections which has not experienced a catastrophic failure and thus is running, respectively co-associating recently logged behaviors of the respectively generated log produced by the respective section behaviors logging subsystem with substantially contemporaneous alarms generated by the respective section alarming subsystem;

for each respective one of the running sections, building a respective annotated log that includes the co-associations made between the recently logged respective behaviors and the respective temporally corresponding generatings and non-generatings of alarms by the respective section alarming subsystem;

for each respective one of the running sections, using the respective annotated log to update a respective anomalies versus parameters mapping space by adding to that respective mapping space recent, alarm-including sample point entries from the respective annotated log as recently logged ones of alarmed sample points (ASP'"'"'s) of the respective section of the data processing system;

for each respective one of the running sections, determining if the recently logged ASP'"'"'s map into a region of the respective mapping space occupied by older ASP'"'"'s associated with identified as routine anomalies or if the recently logged ASP'"'"'s map into a different region of the respective mapping space, where the ASP'"'"'s which map into the different region can represent newly emerging and possibly non-routine anomalies; and

automatically repeating for each respective one of the running sections, said respective co-associating step, said respective building step, said respective using step and said respective determining step;

keeping track in each of the respective anomalies versus parameters mapping spaces of changes over time in the mapping locations of and/or the rates of sample point additions by the respective alarm-including sample point entries to the respective anomalies versus parameters mapping spaces; and

adaptively reallocating resources to respective ones of the running sections based on the tracked changes within the respective anomalies versus parameters mapping spaces.

19. The machine-implemented method of claim 18 wherein:

the adaptively reallocating of resources includes automatically repeatedly determining for each tracked section if a per unit usage is above a predetermined maximum threshold for such a unit such that a predetermined desired margin of safety of the respectively tracked section is compromised and in response to a determining that the per unit usage is above the corresponding predetermined maximum threshold, adding a further such unit to the respective section.

20. The machine-implemented method of claim 18 and further comprising:

building one or more hierarchical parent annotated logs that include the co-associations made between the recently logged respective behaviors and the temporally corresponding generatings and non-generatings of alarms by the respective section alarming subsystems of two or more of the system sections.