System and method for analyzing alternatives in test plans

• US 8,689,188 B2
• Filed: 09/11/2009
• Issued: 04/01/2014
• Est. Priority Date: 09/11/2009
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Bassin, Kathryn A., Hess, Howard M., Kagan, Steven, Li, Shao C., Li, Zhong J., Liu, He H., Skrabanek, Susan E., Tan, Hua F., Zhu, Jun
• Primary Examiner(s): SNYDER, STEVEN G
• Application Number:
  US12/557,886
• Publication Number:
  US 20110066890A1
• Time in Patent Office: 1,663 Days
• Field of Search: None
• US Class Current: 717/124
• CPC Class Codes: G06F 11/3672 Test management

Abstract

A method includes creating an initial test plan including initial estimates of effort and defect distributions, creating an alternative test plan including alternative estimates of effort and defect distributions, and displaying at least one metric of the initial test plan and the alternative test plan side by side for comparison by a user.

Claims

1. A method implemented in a computer infrastructure having computer executable code tangibly embodied on a computer readable storage medium having programming instructions operable to:

create using a processor of the computer infrastructure an initial test plan including initial estimates of effort and defect distributions, the creating comprising;

receiving empirical data; and

determining test activities based on the empirical data,

wherein the initial estimates of the effort and the defect distribution comprises calculated values associated with each test activity,

create an alternative test plan including alternative estimates of effort and defect distributions, the creating comprising;

providing a questionnaire to a user for adjusting the test activities of the initial test plan;

generating suggestions for a set of alternative test activities based on answers provided by the user for the questionnaire, and

determining alternative test activities based on suggestions selected by the user from the set of alternative test activities,

wherein the alternative estimates of the effort and the defect distribution comprises calculated values associated with each alternative test activity; and

display at least one metric of the initial test plan and the alternative test plan side by side for comparison by the user.

2. The method of claim 1, wherein:

the first test plan is generated using initial input data, and

the alternative test plan is generated using alternative input data.

3. The method of claim 2, wherein the programming instructions are further operable to permit a user to define the alternative input data by changing at least one parameter of the initial input data.

4. The method of claim 3, wherein the at least one parameter comprises at least one of:

organization maturity level;

code size;

activities performed during testing;

estimated effort for a trigger and activity;

estimated defects for a trigger and activity;

total test effort for all activities; and

number of field defects.

5. The method of claim 3, wherein the alternative test plan is generated by recalculating the calculated values of the initial test plan based upon the changing the at least one parameter.

6. The method of claim 3, wherein the programming instructions are further operable to limit a changed value of the at least one parameter to a determined range.

7. The method of claim 3, wherein:

the creating the initial test plan includes generating an initial test schedule, and

the creating the alternative test plan includes generating an alternative test schedule.

8. The method of claim 7, wherein:

the programming instructions are further operable to receive from a user a definition of a test team and an assignment of the test team to a testing activity, and

at least one of the initial test schedule and the alternative test schedule is generated based on the definition and the assignment.

9. The method of claim 7, wherein at least one of the initial test schedule and the alternative test schedule includes:

a duration per test activity, and a total test duration of all activities.

10. The method of claim 7, wherein the displaying the at least one metric comprises displaying at least a portion of the initial test schedule and the alternative test schedule.

11. The method of claim 3, wherein:

the creating the initial test plan includes generating an initial cost estimate, and

the creating the alternative test plan includes generating an alternative cost estimate.

12. The method of claim 11, wherein:

the initial cost estimate includes an initial test cost, an initial defect-fix cost, and an initial business cost, and

the alternative cost estimate includes an alternative test cost, an alternative defect-fix cost, and an alternative business cost.

13. The method of claim 12, wherein the displaying the at least one metric comprises displaying at least one of:

the initial cost estimate and the alternative cost estimate;

the initial test cost and the alternative test cost;

the initial defect-fix cost and the alternative defect-fix cost;

the initial business cost and the alternative business cost.

14. The method of claim 1, wherein the initial estimates of effort and defect distributions and the alternative estimates of effort and defect distributions are defined in terms of orthogonal defect classification (ODC) triggers and the activities.

15. The method of claim 1, wherein a service provider at least one of creates, maintains, deploys and supports the computing infrastructure.

16. The method of claim 1, wherein the computing infrastructure is provided by a service provider under a subscription and/or fee agreement and/or based on advertising content to one or more third parties.

26. The method of claim 1, wherein:

the empirical data comprises a maturity level of an organization performing the activities and a size of code to be tested;

the determining the test activities comprises defining what activities are best suited for the test plan having the maturity level of the organization performing the activities and the size of code to be tested;

the calculated value for the initial estimate of the effort comprises an estimated amount of effort that will be required to complete each activity; and

the calculated value for the initial estimate of the defect distribution comprises an estimated amount of defects that will be identified in each activity.

27. The method of claim 26, wherein the creating the initial test plan further comprises:

receiving a constraint regarding a maximum total project effort or a maximum production defect percentage, wherein the initial test plan is created such that a sum of the estimated effort for all the activities does not exceed the maximum total project effort or the estimated defect distribution for all the activities does not exceed the maximum production defect percentage; and

generating a total cost for the test plan based on the estimates of the effort and the defect distributions, wherein the generating the total cost comprises using cost estimation logic that takes into account empirical cost data that defines a cost to fix a defect during each activity.

17. A system comprising:

a test planning optimization workbench including;

a processor operable to;

receive empirical data and determine test activities for an initial test plan based on the empirical data,

provide a questionnaire for adjusting the test activities of the initial test plan,

generate suggestions for a set of alternative test activities for an alternative test plan based on answers provided for the questionnaire, and

determine alternative test activities based on suggestions selected from the set of alternative test activities,

a memory,

a defect projection engine operable to estimate an effort distribution and a defect distribution for the initial test plan and the alternative test plan, wherein the estimates of the effort and the defect distribution for the initial test plan comprises calculated values associated with each test activity, and the estimates of the effort and the defect distribution for the alternative test plan comprises calculated values associated with each alternative test activity,

a schedule generation engine operable to generate a schedule for the initial test plan and the alternative test plan,

a cost calculation engine operable to determine a cost of the initial test plan and the alternative test plan, and

a dashboard operable to display at least one aspect of the initial test plan and the alternative test plan for comparison by a user.

18. The system of claim 17, wherein:

the initial test plan is based on initial input data, and

the alternative test plan is based on alternative input data that is different than the initial input data.

19. The system of claim 18, wherein the test planning optimization workbench further includes a changeable parameters configuration engine operable to determine changeable inputs and restrict a user to changing only the changeable inputs.

20. The system of claim 17, wherein the effort distribution and the defect distribution are defined in terms of orthogonal defect classification (ODC) triggers and the activities.

21. A computer program product comprising a memory having readable program code embodied in the memory, the program code when executed on a computing device causes the computing device to:

receive initial input data from a user;

determining test activities based on the initial input;

create an initial test plan including initial estimates of effort and defect distributions based on the initial input data, wherein the initial estimates of the effort and the defect distribution comprises calculated values associated with each test activity;

provide a questionnaire for adjusting the test activities based on the initial input;

receive answers to the questionnaire from a user;

generate suggestions for a set of alternative test activities for an alternative test plan based on the answers received for the questionnaire;

receive alternative input data from the user, the alternative input data comprising a selection of alternative test activities from the suggestions for the set of alternative test activities;

create an alternative test plan including alternative estimates of effort and defect distributions based on the alternative input data, wherein the alternative estimates of the effort and the defect distribution comprises calculated values associated with each alternative test activity; and

display at least one metric of the initial test plan and the alternative test plan side by side for comparison by a user.

22. The computer program product of claim 21, wherein the initial estimates of effort and defect distributions and the alternative estimates of effort and defect distributions are defined in terms of orthogonal defect classification (ODC) triggers and the activities.

23. The computer program product of claim 21, wherein:

the alternative input data comprises a change to the initial input data, and

the creating the alternative test plan includes propagating the change through the initial estimates of effort and defect distributions to calculate the alternative estimates of effort and defect distributions.

24. A computer system for providing an alternatives analysis for a test plan, the system comprising:

a processor and a memory;

first program instructions to receiving empirical data and determine test activities based on the empirical data;

second program instructions to estimate an effort distribution and a defect distribution for an initial test plan, wherein the estimates of the effort and the defect distribution comprises calculated values associated with each test activity;

third program instructions to generate a schedule for the initial test plan;

fourth program instructions to determine a cost of the initial test plan;

fifth program instructions to provide a questionnaire for adjusting the test activities;

sixth program instructions to generate suggestions for a set of alternative test activities for an alternative test plan based on answers provided for the questionnaire;

seventh program instructions to determine alternative test activities based on suggestions selected from the set of alternative test activities;

eighth program instructions to estimate an alternative effort distribution and a defect distribution for the alternative test plan, wherein the alternative estimates of the effort and the defect distribution comprises calculated values associated with each determined alternative test activity;

ninth program instructions to generate a schedule for the alternative test plan;

tenth program instructions to determine a cost of the alternative test plan; and

eleventh program instructions to display at least one aspect of the initial test plan and the alternative test plan for comparison by a user, wherein the first through eleventh program instructions are stored on the memory for execution by the processor.

25. The system of claim 24, further comprising twelfth program instructions to determine changeable inputs for generating the alternative test plan and restrict a user to changing only the changeable inputs, wherein the changeable inputs comprise the suggestions generated for the set of alternative test activities.