MEASURING AND OPTIMIZING TEST RESOURCES AND TEST COVERAGE EFFECTIVENESS THROUGH RUN TIME CUSTOMER PROFILING AND ANALYTICS

• US 20180067843A1
• Filed: 09/08/2016
• Published: 03/08/2018
• Est. Priority Date: 09/08/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Conti, Thomas W., Moser, Kyle R.
• Granted Patent: US 10,467,128 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 11/3672 Test management
• CPC Class Codes: G06F 11/3676 for coverage analysis
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites

Abstract

Aspects of the present invention include a method, system and computer program product. The method includes a processor performing an accounting of available test resources within one or more organizations; storing data relating to the accounting of available test resources; storing data relating to one or more test resource goals; determining to maintain the data relating to the accounting of available test resources; determining to analyze the data relating to the accounting of available test resources; analyzing the data relating to the accounting of available test resources with respect to the data relating to one or more test resource goals; and determining from the analyzing, by the processor, the data relating to the accounting of available test resources with respect to the data relating to one or more test resource goals that the available test resources within one or more organizations are either optimal or non-optimal.

Claims

1-7. -7. (canceled)

8. A system comprising:

a processor in communication with one or more types of memory, the processor configured to;

perform an accounting of available test resources within one or more organizations;

store data relating to the accounting of available test resources;

store data relating to one or more test resource goals;

determine to maintain the data relating to the accounting of available test resources;

determine to analyze the data relating to the accounting of available test resources;

analyze the data relating to the accounting of available test resources with respect to the data relating to one or more test resource goals; and

determine, from the processor configured to analyze the data relating to the accounting of available test resources with respect to the data relating to one or more test resource goals, that the available test resources within one or more organizations are either optimal or non-optimal.

9. The system of claim 8 wherein the available test resources within one or more organizations comprise available test resources across one or more of each of test organizations, test environments and an enterprise.

10. The system of claim 8 wherein the data relating to one or more test resource goals is selected from the group consisting of enterprise, organizations, test environments, test workloads, test cases, test scenarios, test skills, test planners, test architects, testers, business partners, OEMs, open source foundations, other related factors and metrics that are resources expenses, and limitations or enablement'"'"'s associated with business geography, cultural sensitivities and legal obligations, and time zone contradictions.

11. The system of claim 8 wherein the available test resources comprise human resources and physical resources.

12. The system of claim 8 wherein the processor configured to analyze the data relating to the accounting of available test resources with respect to the data relating to one or more test resource goals comprises the processor configured to determine via historical and run time customer profiling and analytics a score and ranking of use of resources using one or more factors or metrics.

13. The system of claim 12 wherein the one or more factors or metrics is selected from the group consisting of:

test problems found per resource cost using data from resources data stores along with test execution workload run facts data;

customer detected problems;

environment or configuration optimization;

workload optimization comprising functional coverage, stress and activity levels, and workload characteristics'"'"' attainment;

operational optimization;

duplication of test effort within and across test organizations;

test plan coverage efficiency;

problem and/or critical situation recreate capabilities; and

factors or metrics impacting test resource return on investment.

14. The system of claim 8 wherein the processor configured to determine that the available test resources within one or more organizations are non-optimal comprises the processor configured to change at least one of the one or more test resource goals to reallocate resources within the one or more organizations to improve productivity.

15. A computer program product comprising:

a non-transitory storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method comprising;

performing, by a processor, an accounting of available test resources within one or more organizations;

storing, by the processor, data relating to the accounting of available test resources;

storing, by the processor, data relating to one or more test resource goals;

determining, by the processor, to maintain the data relating to the accounting of available test resources;

determining, by the processor, to analyze the data relating to the accounting of available test resources;

analyzing, by the processor, the data relating to the accounting of available test resources with respect to the data relating to one or more test resource goals; and

determining, by the processor, from the analyzing, by the processor, the data relating to the accounting of available test resources with respect to the data relating to one or more test resource goals that the available test resources within one or more organizations are either optimal or non-optimal.

16. The computer program product of claim 15 wherein the available test resources within one or more organizations comprise available test resources across one or more of each of test organizations, test environments and an enterprise.

17. The computer program product of claim 15 wherein the data relating to one or more test resource goals is selected from the group consisting of enterprise, organizations, test environments, test workloads, test cases, test scenarios, test skills, test planners, test architects, testers, business partners, OEMs, open source foundations, other related factors and metrics that are resources expenses, and limitations or enablement'"'"'s associated with business geography, cultural sensitivities and legal obligations, and time zone contradictions.

18. The computer program product of claim 15 wherein the available test resources comprise human resources and physical resources.

19. The computer program product of claim 15 wherein analyzing, by the processor, the data relating to the accounting of available test resources with respect to the data relating to one or more test resource goals comprises determining via historical and run time customer profiling and analytics a score and ranking of use of resources using one or more factors or metrics.

20. The computer program product of claim 15 wherein determining, by the processor, that the available test resources within one or more organizations are non-optimal comprises changing, by the processor, at least one of the one or more test resource goals to reallocate resources within the one or more organizations to improve productivity.