Confidence Intervals for Key Performance Indicators in Communication Networks

• US 20130166730A1
• Filed: 04/20/2012
• Published: 06/27/2013
• Est. Priority Date: 12/27/2011
• Status: Active Grant

• Current Assignee: NetScout Systems Texas, LLC (NetScout Systems Incorporated)
• Original Assignee: Tektronix Incorporated (Fortive Corp.)
• Inventors: Wilkinson, Robert Todd
• Granted Patent: US 9,130,825 B2
• Time in Patent Office: Days
• US Class Current: 709/224
• CPC Class Codes: H04L 41/142 using statistical or mathematical methods
• CPC Class Codes: H04L 41/5009 Determining service level performance parameters or violations of service level contracts, e.g. violations of agreed response time or mean time between failures [MTBF]
• CPC Class Codes: H04L 43/024 by adaptive sampling
• CPC Class Codes: H04L 43/0852 Delays
• CPC Class Codes: H04L 43/16 Threshold monitoring

Abstract

Systems and methods for calculating and presenting confidence interval(s) for key performance indicator(s) (KPIs) are described. For example, in some embodiments, a method may include identifying vectors representing network events observed by a network monitoring system, each vector including: a dimension, an indication of a sampling ratio with which a respective event was observed, and a value associated with the dimension. The method may also include calculating a KPI corresponding to the observed events for the dimension based, at least in part, upon the values. The method may further include calculating a confidence associated with the KPI, based, at least in part, upon the sampling ratios. In some cases, events may be observed with different sampling ratios. Additionally or alternatively, sampling ratios may include adaptive sampling ratios controlled by the network monitoring system in response to network or resource loading (e.g., subject varying over time), whitelist differentiated sampling ratios, etc.

Claims

1. A method, comprising:

performing, using one or more computer systems,

identifying vectors representing network events observed by a network monitoring system, each vector including;

a dimension, an indication of a sampling ratio with which a respective network event was observed, and a value associated with the dimension;

calculating a Key Performance Indicator (KPI) corresponding to the observed network events for the dimension based, at least in part, upon the values; and

calculating a confidence associated with the KPI, based, at least in part, upon the sampling ratios.

2. The method of claim 1, wherein the dimension includes at least one of:

International Mobile Equipment Identity (IMEI), International Mobile Subscriber Identity (IMSI), Mobile Station Integrated Services Digital Network (MSISDN), User Agent (UA) Profile, User Agent, Handset Make, Handset Model, Software Version, Uniform Resource Locator (URL), Service, Application, Location, Mobile Country Code (MCC), or Mobile Network Code (MNC).

3. The method of claim 1, wherein the dimension is observed with a first sampling ratio, wherein another dimension is observed with a second sampling ratio, and wherein the second sampling ratio is different from the first sampling ratio.

4. The method of claim 1, wherein the sampling ratio includes an adaptive sampling ratio controlled by the network monitoring system in response to network or resource loading.

5. The method of claim 1, wherein the sampling ratio varies over time.

6. The method of claim 1, wherein the sampling ratio includes a whitelist or blacklist -based differentiated sampling ratio.

7. The method of claim 1, wherein the value includes at least one of:

session length, uplink byte count, downlink byte count, number of attempts, number of failures, or latency.

8. The method of claim 1, wherein calculating the confidence further comprises:

performing, using the one or more computer systems,

estimating a number of network events (K_N) that would have been observed by the network monitoring system in the absence of sampling;

determining a number of observed network events (K_n); and

calculating a standard deviation (Kσ

) of the values.

9. The method of claim 8, wherein the confidence includes at least one of:

a confidence level or a confidence interval, and wherein the confidence level or the confidence interval is based, at least in part, upon K_N, K_n, and Kσ

.

10. The method of claim 9, wherein calculating the confidence further comprises:

performing, using the one or more computer systems,

identifying a desired confidence level; and

calculating a confidence interval based, at least in part, upon the desired confidence level.

11. The method of claim 9, wherein calculating the confidence further comprises:

performing, using the one or more computer systems,

identifying a desired confidence interval; and

calculating a confidence level based, at least in part, upon the desired confidence interval.

12. The method of claim 9, further comprising:

performing, using the one or more computer systems,

causing a modification of a sampling ratio with which network events are to be observed in the future based, at least in part, upon a user'"'"'s adjustment to the confidence level or to the confidence interval.

13. A network monitoring system, comprising:

a processor; and

a memory coupled to the processor, the memory configured to store program instructions executable by the processor to cause the network monitoring system to;

identify a plurality of vectors representing sampled network events, each vector including a plurality of dimensions, each of the plurality of dimensions associated with an indication of a sampling ratio with which the respective dimension was sampled, and a value associated with each respective dimension, the sampling ratios being subject to modification over time as a function of network or resource loading;

calculate a Key Performance Indicator (KPI) associated with a selected one of the plurality of dimensions based, at least in part, upon an operation performed with respect to respective values, the operation including at least one of a;

mean, average, minimum, or maximum;

estimate a number of network events (K_N) that would have been observed in the absence of sampling;

determine a number of sampled network events (K_n);

calculate a standard deviation (Kσ

) of the values; and

calculate a confidence value associated with the KPI, based, at least in part, upon K_N, K_n, and Kσ

.

14. The network monitoring system of claim 13, wherein the program instructions are further executable by the processor to cause the network monitoring system to:

display the KPI to a user; and

display the confidence value to the user in a graphical or textual format.

15. The network monitoring system of claim 13, wherein to calculate the confidence value, the program instructions are further executable by the processor to cause the network monitoring system to:

identify a desired confidence level; and

calculate a confidence interval based, at least in part, upon the desired confidence level.

16. The network monitoring system of claim 13, wherein to calculate the confidence value, the program instructions are further executable by the processor to cause the network monitoring system to:

identify a desired confidence interval; and

calculate a confidence level based, at least in part, upon the desired confidence interval.

17. The network monitoring system of claim 13, the program instructions further executable by the processor to cause the network monitoring system to automatically modify a sampling ratio with which the selected dimension is to be sampled in subsequent network events based, at least in part, upon an adjustment to the confidence value.

18. A tangible electronic storage medium having program instructions stored thereon that, upon execution by a processor within a computer system, cause the computer system to:

determine a Key Performance Indicator (KPI) based, at least in part, upon values associated with a dimension in a plurality of vectors representing detected network events, the detected network events having been adaptively sampled by a network monitoring system using a sampling ratio;

calculate a confidence value associated with the KPI, based, at least in part, upon the sampling ratio; and

calculate a second sampling ratio with which subsequent network events are detected based, at least in part, upon a user'"'"'s adjustment to the confidence value.

19. The tangible electronic storage medium of claim 18, wherein to the calculate the confidence value, the program instructions further cause the computer system to:

estimate a number of network events (K_N) that would have been detected in the absence of the variable sampling ratio;

determine a number of observed detected events (K_n); and

calculate a standard deviation (Kσ

) of the values.

20. The tangible electronic storage medium of claim 19, wherein the confidence value includes at least one of:

a confidence level or a confidence interval.