PREDICTIVE DIAGNOSIS OF SLA VIOLATIONS IN CLOUD SERVICES BY SEASONAL TRENDING AND FORECASTING WITH THREAD INTENSITY ANALYTICS

• US 20140310714A1
• Filed: 12/17/2013
• Published: 10/16/2014
• Est. Priority Date: 04/11/2013
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Chan, Eric S., Santos, Adriano Covello, Ahad, Rafiul, Choneimy, Adel
• Granted Patent: US 9,495,395 B2
• Time in Patent Office: Days
• US Class Current: 718/102
• CPC Class Codes: G06F 11/008 Reliability or availability analysis
• CPC Class Codes: G06F 11/0715 in a system implementing multitasking multitasking per se G06F9/46
• CPC Class Codes: G06F 11/0718 in an object-oriented system
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0778 Dumping, i.e. gathering error/state information after a fault for later diagnosis
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3003 Monitoring arrangements specially adapted to the computing system or computing system component being monitored
• CPC Class Codes: G06F 11/3037 where the computing system component is a memory, e.g. virtual memory, cache accessing, addressing or allocating within memory systems or architectures G06F12/00; checking stores for correct operation G11C29/00
• CPC Class Codes: G06F 11/34 Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3419 by assessing time
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 11/3471 Address tracing
• CPC Class Codes: G06F 16/1824 implemented using Network-attached Storage [NAS] architecture distributed or networked storage systems G06F3/067; protocols for distributed storage of data in a network H04L67/1097
• CPC Class Codes: G06F 16/21 Design, administration or maintenance of databases
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 2201/865 Monitoring of software
• CPC Class Codes: G06F 2201/88 Monitoring involving counting
• CPC Class Codes: G06F 9/45533 Hypervisors; Virtual machine monitors
• CPC Class Codes: G06F 9/4881 : Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: H04L 41/5009 : Determining service level performance parameters or violations of service level contracts, e.g. violations of agreed response time or mean time between failures [MTBF]

Abstract

Data can be categorized into facts, information, hypothesis, and directives. Activities that generate certain categories of data based on other categories of data through the application of knowledge which can be categorized into classifications, assessments, resolutions, and enactments. Activities can be driven by a Classification-Assessment-Resolution-Enactment (CARE) control engine. The CARE control and these categorizations can be used to enhance a multitude of systems, for example diagnostic system, such as through historical record keeping, machine learning, and automation. Such a diagnostic system can include a system that forecasts computing system failures based on the application of knowledge to system vital signs such as thread or stack segment intensity and memory heap usage. These vital signs are facts that can be classified to produce information such as memory leaks, convoy effects, or other problems. Classification can involve the automatic generation of classes, states, observations, predictions, norms, objectives, and the processing of sample intervals having irregular durations.

Claims

1. A computer-implemented method comprising:

obtaining a series of thread dump samples by generating a separate thread dump during each particular time interval in a series of time interval;

based on the series of thread dump samples, automatically generating a set of stack segment classifications;

for each particular thread represented in the series of thread dump samples, automatically selecting, from the set of stack segment classifications, a particular stack segment classification for the particular thread based at least in part on a number of invocations of a code block in the particular stack segment multiple by an execution time of the code block;

determining a trend for each stack segment classification in the set of stack segment classifications, thereby generating a set of trends;

determining a set of anomalies based at least in part on the set of trends; and

generating output based at least in part on the set of anomalies;

wherein said set of anomalies includes at least one of;

a level spike, a level shift, a level drift, a variance change, a saturation, a stuck thread, a lingering thread, a deadlock condition, a congestion upstream, a congestion downstream, a congestion in resource pool, a convoy effect, an impedance mismatch, an outlying measurement, or a burst of outlying measurements.

2. The method of claim 1, wherein automatically generating the set of stack segment classifications comprises:

determining that intensities of stack frames within a stack segment represented in the series of thread dump samples diverge over time beyond a specified threshold; and

in response to determining that the intensities of the stack frames within the stack segment diverge over time beyond the specified threshold, automatically splitting the stack segment into multiple constituent stack segments; and

generating a new stack segment classification for each of the multiple constituent stack segments.

3. The method of claim 1, further comprising:

determining whether a thread name, an Execution Context ID (ECID), or an epoch counter of a thread represented in a first thread dump is the same as, respectively, a thread name, an Execution Context ID (ECID), or an epoch counter of a thread represented in a second thread dump;

in response to determining that the thread name, Execution Context ID (ECID), or epoch counter of the thread represented in the first thread dump is the same as, respectively, the thread name, Execution Context ID (ECID), or epoch counter of the thread represented in the second thread dump, preventing the thread represented in the second thread dump from being counted for intensity statistics.

4. The method of claim 3, further comprising:

determining that a CPU time of the thread represented in the first thread dump is same as a CPU time of a thread represented in the second thread dump; and

in response to determining that the thread name, Execution Context ID (ECID), or epoch counter of the thread represented in the first thread dump is the same as, respectively, the thread name, Execution Context ID (ECID), or epoch counter of the thread represented in the second thread dump and that the CPU time of the thread represented in the first thread dump is same as the CPU time of a thread represented in the second thread dump, identifying the same thread represented in the first and second thread dump as a stuck thread.

5. The method of claim 4, further comprising:

determining, based at least in part on a lack of change in a plurality of CPU times of a first thread within separate thread dump samples in the series of thread dump samples, that the first thread is stuck.

6. The method of claim 3, further comprising:

determining that a CPU time of a thread represented in the first thread dump is different from a CPU time of the thread represented in the second thread dump; and

in response to determining that the thread name, Execution Context ID (ECID), or epoch counter of the thread represented in the first thread dump is, respectively, the same as the thread name, Execution Context ID (ECID), or epoch counter of the thread represented in the second thread dump and that the CPU time of the thread represented in the first thread dump is different from the CPU time of a thread represented in the second thread dump, identifying the same thread represented in the first and second thread dump as a lingering thread.

7. The method of claim 6, further comprising:

determining, based at least in part on a lack of change in a plurality of the thread name, Execution Context ID (ECID), or epoch counter of a first thread within separate thread dump samples in the series of thread dump samples, that a first thread is lingering.

8. The method of claim 1, further comprising:

maintaining a first intensity statistics of a first sampling interval for thread dumps in the series of thread dumps;

adjusting a second intensity statistics of a second sampling interval for thread dumps in the series of thread dumps in response to a determination that a particular thread is determined to be stuck or lingering over a first and second sampling intervals, wherein a second sampling interval avoids duplicate counting of a stuck or lingering thread.

9. The method of claim 1, wherein selecting, from the set of stack segment classifications, a particular stack segment classification for the particular thread comprises:

determining an intensity value for a particular stack frame in a particular stack trace;

determining an intensity value for a predecessor stack frame that occurs immediately before the particular stack frame in the particular stack trace;

determining an intensity value for a successor stack frame that occurs immediately after the particular stack frame in the particular stack trace;

in response to determining that the intensity value for the particular stack frame is exactly or approximately the same as the intensity value for the predecessor stack frame, assigning a particular classification to both the particular stack frame and the predecessor stack frame; and

in response to determining that the intensity value for the particular stack frame is the same as the intensity value for the successor stack frame, assigning the particular classification to both the particular stack frame and the successor stack frame.

10. The method of claim 1, further comprising:

creating a new stack segment information item;

assigning the new stack segment information item to a value of a current stack segment information item;

generating a new seasonal trend information item in response to a classification of the new stack segment information item; and

setting a value of a trend attribute of the current stack segment information item to be the new seasonal trend information item.

11. The method of claim 1, further comprising:

splitting a coalesced stack segment into a first new stack segment and a second new stack segment;

creating a first clone of a seasonal trend information item of the coalesced stack segment;

creating a second clone of the seasonal trend information item of the coalesced stack segment;

setting a value of a trend attribute of the first new stack segment to be the first clone; and

setting a value of a trend attribute of the second new stack segment to be the second clone.

12. The method of claim 1, further comprising:

generating a set of stack segment classifications; and

generating a thread classification based at least in part on a chain of stack segment classifications.

13. The method of claim 12, further comprising:

generating a set of thread classifications;

generating a classification of a chain of threads based at least in part on one of an invocation chain, a dependency chain, and a client-server chain among threads.

14. The method of claim 1, further comprising:

organizing one or more stack segment classifications in the set of stack segment classifications by assigning the one or more stack segment classifications to a dimension;

setting a name of the particular stack segment classification; and

setting a name of a particular dimension.

15. The method of claim 14, further comprising:

drilling down into a class hierarchy of the particular stack segment classification by taking a cross product of the dimensions to which the particular stack segment classification is assigned.

16. The method of claim 14, further comprising:

assigning the one or more stack segment classifications to a dimension based at least in part on tacit knowledge, to at least one of the domain dimension, the server dimension, the channel dimension, the application dimension, the library dimension, and the operation dimension.

17. The method of claim 16, further comprising:

setting a name of the particular stack segment classification based at least in part on the tacit knowledge; and

setting a name of a particular dimension based at least in part on the tacit knowledge;

wherein assigning to a dimension based at least in part on tacit knowledge, to at least one of the domain dimension, the server dimension, the channel dimension, the application dimension, the library dimension, and the operation dimension.

18. The method of claim 1, further comprising:

classifying one or more stack segments based on human tacit knowledge;

assigning one or more classifications to one or more dimensions based at least in part on the human tacit knowledge.

19. The method of claim 1, further comprising:

based at least in part on an inductive learning process, assigning one or more of the stack segment classifications in the set of stack segment classifications to one or more dimensions that were defined prior to the generation of the one or more of the stack segment classifications.

20. The method of claim 1, further comprising:

storing a generalized dependency relation that includes multiple subclasses of dependency relations that each include a client part and a server part;

drilling down a generalized client part of a particular subclass dependency relation to a client part that is a higher intensity than a generalized server part of the particular subclass; and

drilling down a generalized server part of the particular subclass dependency relation to a server part that is a higher intensity than a generalized client part of the particular subclass.

21. The method of claim 1, further comprising:

generating a knowledge base of thread dependency relations based at least in part on tacit knowledge; and

based at least in part on the thread dependency relations, correlating at least one incidence across a pair of serving entities as a part of analyzing a root cause of a problem;

wherein the pair of serving entities are at least one of a pair of client and server, a pair of servers, or a pair of clients.

22. The method of claim 21, further comprising:

performing impedance matching based at least in part on channels between the pair of serving entities;

wherein the channels support a client-server dependency relation between a class of stack segments for a first serving entity in the pair and a class of stack segments for the second serving entity in the pair.

23. The method of claim 1, further comprising:

determining a stack segment intensity of the particular stack segment classification based on a composition of stack segment intensities of the sub-classifications of stack segments; and

segmenting an expected response time of a stack segment of the particular stack segment classification into response times of stack segments of the sub-classifications of stack segments;

wherein segmenting an expected response time is based at least in part to a composition of stack segment intensities.

24. A non-transitory computer-readable storage memory storing instructions when, when executed by one or more processors, cause the one or more processors to perform operations comprising:

obtaining a series of thread dump samples by generating a separate thread dump during each particular time interval in a series of time interval;

based on the series of thread dump samples, automatically generating a set of stack segment classifications;

for each particular thread represented in the series of thread dump samples, automatically selecting, from the set of stack segment classifications, a particular stack segment classification for the particular thread based at least in part on a number of invocations of a code block in the particular stack segment multiple by an execution time of the code block;

determining a trend for each stack segment classification in the set of stack segment classifications, thereby generating a set of trends;

determining a set of anomalies based at least in part on the set of trends; and

generating output based at least in part on the set of anomalies;

wherein said set of anomalies includes at least one of;

a level spike, a level shift, a level drift, a variance change, a saturation, a stuck thread, a lingering thread, a deadlock condition, a congestion upstream, a congestion downstream, a congestion in resource pool, a convoy effect, an impedance mismatch, an outlying measurement, or a burst of outlying measurements.