Method, system, and storage medium for using comparisons of empirical system data for testcase and workload profiling

• US 20060095312A1
• Filed: 10/28/2004
• Published: 05/04/2006
• Est. Priority Date: 10/28/2004
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Miller, Geoffrey E., Conti, Thomas W., Prewitt, Richard D., Menendez, Terri M.
• Application Number:
  US10/976,586
• Publication Number:
  US 20060095312A1
• Time in Patent Office: Days
• US Class Current: 717/124
• CPC Class Codes: G06F 11/3672 Test management
• CPC Class Codes: G06F 11/3692 for test results analysis

Abstract

Systems and methods for using comparisons of empirical system data, (e.g., performance, accounting, software module or function frequency), for testcase and workload profiling are provided. Instead of asking a customer what he does, simply asking for some set of empirical data that can be formatted, reduced, and analyzed. By gathering the same kind of data for the test systems that is used by the customer, testcases and workload profiling are improved by making comparisons between the customer data and the test data in an iterative process. The iterative processes change test data and compare not only customer data with test data but also compare data from prior iterations with current data. There is a feedback loop for providing a comparison of how close or distant the testcases and workload profiling are from customer-like data and workloads in a particular test.

Claims

1. A method of comparing empirical system data for testcase and workload profiling, comprising:

gathering customer data from a customer system;

gathering test data from a test system corresponding to the customer data, the test system including testcases and workloads;

improving at least one testcase or at least one workload in the test system by making comparisons between the customer data and the test data in an iterative process.

2. The method of claim 1, further comprising:

comparing data from a prior iteration with data in a current iteration.

3. The method of claim 1, further comprising:

providing a comparison of how close or distant the at least one testcase and the at least one workload are from the customer data.

4. The method of claim 1, further comprising:

formatting, reducing, and analyzing the customer data.

5. A method for using comparisons of empirical system data for testcase and workload profiling, comprising:

receiving customer data from a customer system, the customer data including a plurality of customer activities;

gathering test data from a test system, the test system including a plurality of testcases and a plurality of workloads, the test data including a plurality of test activities, each test activity in the test activities corresponding to a customer activity in the customer activities;

determining whether a select test activity meets or exceeds the corresponding customer activity; and

changing a workload in the test system and gathering new test data, when the select test activity does not meet or exceed the corresponding customer activity.

6. The method of claim 5, further comprising:

performing a statistical analysis of a comparison of the customer data and the test data.

7. The method of claim 6, further comprising:

calculating differences between the customer data and the test data based on the statistical analysis.

8. The method of claim 7, further comprising:

creating charts indicating the differences.

9. The method of claim 5, further comprising:

changing a testcase in the test system and gathering new test data, when the test activity does not meet or exceed the corresponding customer activity.

10. The method of claim 5, wherein changes are made in a plurality of workloads in the test system and new test data is gathered, until each test activity meets or exceed the corresponding customer activity.

11. The method of claim 10, further comprising:

performing a statistical analysis of a comparison of the customer data and the new test data;

calculating differences between the customer data and the new test data based on the statistical analysis; and

creating charts indicating the differences.

12. A system for using comparisons of empirical system data for testcase and workload profiling, comprising:

a customer system for providing customer data, the customer data including a plurality of customer activities;

a test system for providing test data, the test data including a plurality of testcases and a plurality of workloads, the test data including a plurality of test activities, each test activity in the test activities corresponding to a customer activity in the customer activities;

wherein the test system changes a workload in the test system and gathers new test data, when a select test activity does not meet or exceed a corresponding customer activity.

13. The system of claim 12, further comprising:

a data analysis system for performing a statistical analysis of a comparison of the customer data and the test data.

14. The system of claim 12, wherein the data analysis system calculates differences between the customer data and the test data based on the statistical analysis.

15. The system of claim 14, wherein the data analysis system creates charts indicating the differences.

16. The system of claim 12, wherein the test system changes a plurality of workloads and provides new test data, until each test activity meets or exceed the corresponding customer activity.

17. A storage medium having instructions stored thereon to perform a method of comparing empirical system data for testcase and workload profiling, the method comprising:

gathering customer data from a customer system;

gathering test data from a test system corresponding to the customer data, the test system including testcases and workloads;

improving at least one testcase or at least one workload in the test system by making comparisons between the customer data and the test data in an iterative process.

18. A storage medium having instructions stored thereon to perform a method for using comparisons of empirical system data for testcase and workload profiling, the method comprising:

receiving customer data from a customer system, the customer data including a plurality of customer activities;

gathering test data from a test system, the test system including a plurality of testcases and a plurality of workloads, the test data including a plurality of test activities, each test activity in the test activities corresponding to a customer activity in the customer activities;

determining whether a select test activity meets or exceeds the corresponding customer activity; and

changing a workload in the test system and gathering new test data, when the select test activity does not meet or exceed the corresponding customer activity.