System and method for evaluating the performance of a computer application

• US 6,782,421 B1
• Filed: 07/09/2002
• Issued: 08/24/2004
• Est. Priority Date: 03/21/2001
• Status: Expired due to Term

• Current Assignee: Fintegraph, LLC (Ascend Innovation Management, LLC)
• Original Assignee: Bellsouth Intellectual Property Corporation (AT&T, Inc.)
• Inventors: Thompson, Marion B., Anth, Arthur J., Soles, William E., OʼShea, Edward T., Bowlin, Amy H., Garguilo, Frank E., Morgan, Joseph
• Primary Examiner(s): Barot, Bharat
• Application Number:
  US10/191,882
• Time in Patent Office: 777 Days
• Field of Search: 709/200-203, 709/217-219, 709/220-221, 709/223-229, 709/236, 709/250, 718/100, 718/104-106, 370/229-235, 370/238, 370/248, 370/252-254
• US Class Current: 709/223
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: H04L 63/1433 Vulnerability analysis

Abstract

A system for evaluating the performance of a computer-implemented application, wherein the performance is measured in terms of availability. The system includes a metrics data module comprising actual service level data, wherein the actual service level data can be used to determine an actual service level; a defined data module comprising defined service level data, wherein the defined service level data can be used to determine a defined service level; and a service level analysis module in communication with the metrics data module and the defined data module for calculating the actual service level and the defined service level and for comparing the actual service level to the defined service level using a grading scale defined as a function of the defined service level.

Claims

1. A system for evaluating the performance of a computer-implemented application, wherein the performance is measured in terms of availability, comprising:

a metrics data module including actual service level data, wherein the actual service level data can be used to determine an actual service level;

a defined data module including defined service level data, wherein the defined service level data can be used to determine a defined service level; and

a service level analysis module in communication with the metrics data module and the defined data module for calculating the actual service level and the defined service level and for comparing the actual service level to the defined service level using a grading scale defined as a function of the defined service level, wherein the grading scale comprises;

an availability spectrum having a top and a bottom, wherein the defined service level is between the top and the bottom, and wherein the availability spectrum is divided into a plurality of bands having boundaries defined by breakpoints, and wherein the bands have a width defined as a function of the defined service level;

a numerical scoring spectrum, wherein the scoring spectrum has a top and a bottom, wherein the top of the scoring spectrum corresponds to the top of the availability spectrum, and wherein the scoring spectrum is divided into a plurality of bands, and wherein the boundaries of the bands are defined by breakpoints, and wherein the breakpoints of the scoring spectrum correspond to the breakpoints of the availability spectrum; and

an interpolation module for translating an actual service level into a service level score, wherein the interpolation module interpolates between the availability spectrum and the scoring spectrum.

2. The system of claim 1, wherein the scoring spectrum is defined by a plurality of non-uniform bands.

3. The system of claim 1, wherein the scoring spectrum is defined by a plurality of uniform bands.

4. The system of claims 1, wherein the availability and scoring spectrums have four breakpoints defining five bands.

5. The system of claim 4, wherein two of the bands are situated above the defined service level and three bands are situation below the defined service level.

6. The system of claim 1, wherein the overall service level grade is the grade associated with the band that encompasses the average of the service level scores over an evaluation period.

7. A system for evaluating the performance of a computer-implemented application, wherein the performance is measured in terms of availability, comprising:

a metrics data module including actual service level data, wherein the actual service level data can be used to determine an actual service level;

a defined data module including defined service level data, wherein the defined service level data can be used to determine a defined service level;

a service level analysis module in communication with the metrics data module and the defined data module for calculating the actual service level and the defined service level and for comparing the actual service level to the defined service level using a grading scale defined as a function of the defined service level;

an architecture data module comprising architecture data, wherein the architecture data can be used to determine an architecture designed service level in terms of application availability; and

an architecture analysis module in communication with the architecture data module and the metrics data module for calculating the architecture designed service level and for calculating an overall architecture capabilities grade for the application by comparing the architecture designed service level to the defined service level.

8. The system of claim 7, wherein the architecture analysis module is also for developing an architecture capability chart, and wherein the architecture capability chart shows historical performance, an architecture capability, and a defined service level, all in terms of availability.

9. The system of claim 7, wherein the architecture data module contains a plurality of system characteristics arranged by category, wherein each category is defined by an availability spectrum band, and wherein each system characteristic is categorized according to the lowest availability level that the system characteristic can support.

10. The system of claim 9, wherein the architecture analysis module determines an overall architecture capability equal to the lowest availability band that all system characteristics can support.

11. The system of claim 9, wherein the overall architecture capability is equal to the lowest availability band that at least 80 percent of all system characteristics can support.

12. A system for evaluating the performance of a computer-implemented application, wherein the performance is measured in terms of availability, comprising:

a metrics data module including actual service level data, wherein the actual service level data can be used to determine an actual service level;

a defined data module including defined service level data, wherein the defined service level data can be used to determine a defined service level;

a service level analysis module in communication with the metrics data module and the defined data module for calculating the actual service level and the defined service level and for comparing the actual service level to the defined service level using a grading scale defined as a function of the defined service level;

a best practices data module defining a plurality of areas of best practice, wherein each area of best practice is designed to guard against a certain problem;

a vulnerability data module defining application vulnerability data; and

a vulnerability analysis module, in communication with the vulnerability data module and the best practices data module, for calculating an overall vulnerability grade for the application and an implementation of best practices grade.

13. The system of claim 12, wherein the vulnerability data module comprises:

degree of implementation data for the areas of best practice;

frequency of occurrence data defining the frequency to which best practices occur; and

business impact data defining the potential economic and productivity impact should the problem occur.

14. The system of claim 13, wherein the vulnerability analysis module includes:

a business risk module for determining business risk for each practice area, wherein the business risk is the product of the frequency of occurrence of the best practice area and the business impact; and

a vulnerability scoring module for determining an overall vulnerability score, wherein the vulnerability score is a function of the degree of implementation and the business risk for each area of best practice.

15. A method for evaluating the performance of a computer-implemented application, wherein the performance is measured in terms of availability, comprising:

collecting actual service level data, wherein the actual service level data can be used to determine an actual service level in terms of application availability;

collecting defined service level data, wherein the defined service level data can be used to determine a defined service level in terms of application availability;

calculating an actual service level in terms of application availability;

calculating a defined service level in terms of application availability; and

comparing the actual service level to the defined service level using a grading scale defined as a function of the defined service level, wherein the grading system comprises;

defining an availability spectrum having a top and a bottom, wherein the defined service level is between the top and the bottom, and wherein the availability spectrum is divided into a plurality of bands having boundaries defined by breakpoints, and wherein the bands have a width defined as a function of the defined service level;

defining a numerical scoring spectrum, wherein the scoring spectrum has a top and a bottom, wherein the top of the scoring spectrum corresponds to the top of the availability spectrum, and wherein the scoring spectrum is divided into a plurality of bands, and wherein the boundaries of the bands are defined by breakpoints, and wherein the breakpoints of the scoring spectrum correspond to the breakpoints of the availability spectrum; and

translating an actual service level into a service level score by interpolating between the availability spectrum and the scoring spectrum.

16. The method of claim 15, wherein the scoring spectrum is defined by a plurality of non-uniform bands.

17. The method of claim 15, wherein the scoring spectrum is defined by a plurality of uniform bands.

18. The method of claim 15, wherein the availability and scoring spectrums have four breakpoints defining five bands.

19. The method of claim 18, wherein two of the bands are situated above the defined service level and three bands are situation below the defined service level.

20. The method of claim 15, wherein the overall service level grade is the grade associated with the band that encompasses the average of the service level scores over an evaluation period.

21. A method for evaluating the performance of a computer-implemented application, wherein the performance is measured in terms of availability, comprising:

collecting actual service level data, wherein the actual service level data can be used to determine an actual service level in terms of application availability;

collecting defined service level data, wherein the defined service level data can be used to determine a defined service level in terms of application availability;

calculating an actual service level in terms of application availability;

calculating a defined service level in terms of application availability;

comparing the actual service level to the defined service level using a grading scale defined as a function of the defined service level;

collecting architecture data, wherein the architecture data can be used to determine an architecture designed service level in terms of application availability;

calculating the architecture designed service level; and

calculating an overall architecture capabilities grade for the application by comparing the architecture designed service level to the defined service level.

22. The method of claim 21, further comprising:

developing an architecture capability chart, wherein the architecture capability chart shows historical performance, an architecture capability, and a defined service level, all in terms of availability.

23. The method of claim 21, wherein collecting includes:

arranging a plurality of system characteristics by category, wherein each category is defined by an availability spectrum band, wherein each system characteristic is categorized according to the lowest availability level that the system characteristic can support.

24. The method of claim 23, further comprising

determining an overall architecture capability equal to the lowest availability band that all system characteristics can support.

25. The method of claim 23, wherein the overall architecture capability is equal to the lowest availability band that at least 80 percent of all system characteristics can support.

26. A method for evaluating the performance of a computer-implemented application, wherein the performance is measured in terms of availability, comprising:

collecting actual service level data, wherein the actual service level data can be used to determine an actual service level in terms of application availability;

collecting defined service level data, wherein the defined service level data can be used to determine a defined service level in terms of application availability;

calculating an actual service level in terms of application availability;

calculating a defined service level in terms of application availability;

comparing the actual service level to the defined service level using a grading scale defined as a function of the defined service level;

defining a plurality of areas of best practice, wherein each area of best practice is designed to guard against a certain problem;

defining application vulnerability data; and

calculating an overall vulnerability grade for the application and an implementation of best practices grade.

27. The method of claim 26, wherein defining application vulnerability data comprises:

collecting degree of implementation data for the areas of best practice;

collecting frequency of occurrence data defining the frequency to which best practices occur; and

collecting business impact data defining the potential economic and productivity impact should the problem occur.

28. The method of claim 27, wherein calculating an overall vulnerability grade includes:

determining business risk for each practice area, wherein the business risk is the product of the frequency of occurrence of the best practice area and the business impact;

determining an overall vulnerability score, wherein the vulnerability score is a function of the degree of implementation and the business risk for each area of best practice.

29. A computer readable medium having stored thereon instructions which, when executed by a processor, cause the processor to perform the steps of:

collecting actual service level data, wherein the actual service level data can be used to determine an actual service level in terms of application availability;

collecting defined service level data, wherein the defined service level data can be used to determine a defined service level in terms of application availability;

calculating an actual service level in terms of application availability;

calculating a defined service level in terms of application availability;

comparing the actual service level to the defined service level using a grading scale defined as a function of the defined service level, wherein the grading system comprises;

defining an availability spectrum having a top and a bottom, wherein the defined service level is between the top and the bottom, and wherein the availability spectrum is divided into a plurality of bands having boundaries defined by breakpoints, and wherein the bands have a width defined as a function of the defined service level;

defining a numerical scoring spectrum, wherein the scoring spectrum has a top and a bottom, wherein the top of the scoring spectrum corresponds to the top of the availability spectrum, and wherein the scoring spectrum is divided into a plurality of bands, and wherein the boundaries of the bands are defined by breakpoints, and wherein the breakpoints of the scoring spectrum correspond to the breakpoints of the availability spectrum; and

translating an actual service level into a service level score by interpolating between the availability spectrum and the scoring spectrum.

30. The medium of claim 29, wherein the scoring spectrum is defined by a plurality of non-uniform bands.

31. The medium of claim 29, wherein the scoring spectrum is defined by a plurality of uniform bands.

32. The medium of claim 29, wherein the availability and scoring spectrums have four breakpoints defining five bands.

33. The medium of claim 32, wherein two of the bands are situated above the defined service level and three bands are situation below the defined service level.

34. The medium of claim 29, wherein the overall service level grade is the grade associated with the band that encompasses the average of the service level scores over an evaluation period.

35. A computer readable medium having stored thereon instructions which, when executed by a processor, cause the processor to perform the steps of:

collecting actual service level data, wherein the actual service level data can be used to determine an actual service level in terms of application availability;

collecting defined service level data, wherein the defined service level data can be used to determine a defined service level in terms of application availability;

calculating an actual service level in terms of application availability;

calculating a defined service level in terms of application availability;

comparing the actual service level to the defined service level using a grading scale defined as a function of the defined service level;

collecting architecture data, wherein the architecture data can be used to determine an architecture designed service level in terms of application availability;

calculating the architecture designed service level; and

calculating an overall architecture capabilities grade for the application by comparing the architecture designed service level to the defined service level.

36. The medium of claim 35, further comprising:

developing an architecture capability chart, wherein the architecture capability chart shows historical performance, an architecture capability, and a defined service level, all in terms of availability.

37. The medium of claim 35, wherein collecting includes:

arranging a plurality of system characteristics by category, wherein each category is defined by an availability spectrum band, and wherein each system characteristic is categorized according to the lowest availability level that the system characteristic can support.

38. The medium of claim 37, further comprising

determining an overall architecture capability equal to the lowest availability band that all system characteristics can support.

39. The medium of claim 37, wherein the overall architecture capability is equal to the lowest availability band that at least 80 percent of all system characteristics can support.

40. A computer readable medium having stored thereon instructions which, when executed by a processor, cause the processor to perform the steps of:

collecting actual service level data, wherein the actual service level data can be used to determine an actual service level in terms of application availability;

collecting defined service level data, wherein the defined service level data can be used to determine a defined service level in terms of application availability;

calculating an actual service level in terms of application availability;

calculating a defined service level in terms of application availability;

comparing the actual service level to the defined service level using a grading scale defined as a function of the defined service level;

defining a plurality of areas of best practice, wherein each area of best practice is designed to guard against a certain problem;

defining application vulnerability data; and

calculating an overall vulnerability grade for the application and an implementation of best practices grade.

41. The medium of claim 40, wherein defining application vulnerability data comprises:

collecting degree of implementation data for the areas of best practice;

collecting frequency of occurrence data defining the frequency to which best practices occur; and

collecting business impact data defining the potential economic and productivity impact should the problem occur.

42. The medium of claim 41, wherein calculating an overall vulnerability grade includes:

determining business risk for each practice area, wherein the business risk is the product of the frequency of occurrence of the best practice area and the business impact;

determining an overall vulnerability score, wherein the vulnerability score is a function of the degree of implementation and the business risk for each area of best practice.

43. A system for evaluating the performance of a computer-implemented application, wherein the performance is measured in terms of availability, comprising:

a metrics data module comprising actual service level data, wherein the actual service level data can be used to determine an actual service level in terms of application availability;

a defined data module comprising defined service level data, wherein the defined service level data can be used to determine a defined service level in terms of application availability;

means for calculating the actual service level and the defined service level; and

means for comparing the actual service level to the defined service level using a grading scale defined as a function of the defined service level, wherein the grading system comprises;

an availability spectrum having a top and a bottom, wherein the defined service level is between the top and the bottom, and wherein the availability spectrum is divided into a plurality of bands having boundaries defined by breakpoints, and wherein the bands have a width defined as a function of the defined service level;

a numerical scoring spectrum, wherein the scoring spectrum has a top and a bottom, wherein the top of the scoring spectrum corresponds to the top of the availability spectrum, and wherein the scoring spectrum is divided into a plurality of bands, and wherein the boundaries of the bands are defined by breakpoints, and wherein the breakpoints of the scoring spectrum correspond to the breakpoints of the availability spectrum; and

means for translating an actual service level into a service level score.

44. The system of claim 43, wherein the grading system includes means for emphasizing lower scores.

45. A system for evaluating the performance of a computer-implemented application, wherein the performance is measured in terms of availability, comprising:

a metrics data module comprising actual service level data, wherein the actual service level data can be used to determine an actual service level in terms of application availability;

a defined data module comprising defined service level data, wherein the defined service level data can be used to determine a defined service level in terms of application availability;

means for calculating the actual service level and the defined service level;

means for comparing the actual service level to the defined service level using a grading scale defined as a function of the defined service level;

an architecture data module comprising architecture data, wherein the architecture data can be used to determine an architecture designed service level in terms of application availability; and

means for calculating the architecture designed service level and means for calculating an overall architecture capabilities grade for the application by comparing the architecture designed service level to the defined service level.

46. The system of claim 45, wherein the architecture data module includes means for assessing system characteristics according to the highest availability that the system characteristic can support.

47. A system for evaluating the performance of a computer-implemented application, wherein the performance is measured in terms of availability, comprising:

a metrics data module comprising actual service level data, wherein the actual service level data can be used to determine an actual service level in terms of application availability;

a defined data module comprising defined service level data, wherein the defined service level data can be used to determine a defined service level in terms of application availability;

means for calculating the actual service level and the defined service level;

means for comparing the actual service level to the defined service level using a grading scale defined as a function of the defined service level;

a best practices data module defining a plurality of areas of best practice, wherein each area of best practice is designed to guard against a certain problem;

a vulnerability data module defining application vulnerability data; and

means for calculating an overall vulnerability grade for the application.

48. The system of claim 47, wherein the vulnerability data module comprises:

means for defining the degree of implementation for the areas of best practice;

means for defining the frequency at which best practices occur; and

means for defining the potential economic and productivity impact should the problem occur.

49. The system of claim 48, wherein the vulnerability analysis module includes:

means for determining business risk for each practice area, wherein the business risk is the product of the frequency of occurrence of the best practice area and the business impact; and

means for determining an overall vulnerability score, wherein the vulnerability score is a function of the degree of implementation and the business risk for each area of best practice.