RANKING AND OPTIMIZING AUTOMATED TEST SCRIPTS

• US 20090265681A1
• Filed: 04/21/2008
• Published: 10/22/2009
• Est. Priority Date: 04/21/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kurniawan, Ricky, Beto, Kevin Rocco, Wong, Che Yan
• Granted Patent: US 8,266,592 B2
• Time in Patent Office: Days
• US Class Current: 717/100
• CPC Class Codes: G06F 11/3672 Test management
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites

Abstract

Technologies are described herein for ranking and optimizing test scripts utilized in the automated testing of software products. A score is calculated for each test script from various metrics collected from executions of the test script. Metrics utilized in calculating the score for a test script may include those indicating the propensity of the script to result in “false failures” and/or those indicating the effectiveness of the script for finding product bugs. The test scripts are then ranked by their score, and this ranking is used in determining the frequency of execution of the test script in future testing.

Claims

1. A method for ranking and optimizing automated test scripts, the method comprising:

executing a plurality of test scripts, wherein the test scripts are members of a collection;

recording a test result and execution data in a repository for each of the plurality of test scripts executed;

calculating a score for each of the plurality of test scripts based upon the test results and execution data recorded in the repository for the test script;

ranking the plurality of test scripts according to the score calculated for each; and

determining the members of the collection based upon the rank of each of the plurality of test scripts.

2. The method of claim 1, wherein the method is repeated during one or more iterations and the test result and execution data for each of the plurality of test scripts is recorded in the repository for each iteration.

3. The method of claim 2, wherein the test result consists of one of success, failure due to product bug, failure not due to product bug, and failure requiring investigation.

4. The method of claim 3, wherein calculating a score for each of the plurality of test scripts comprises multiplying a number of executions of the test script within a first period of time by a sum of a plurality of terms comprising a ratio of the number of executions of the test script resulting in the test result of failure not due to product bug to the number of executions of the test script within the first period of time, a ratio of the number of executions of the test script resulting in the test result of failure not due to product bug to the number of executions of the test script within a second period of time, and the number of executions of the test script resulting in the test result of failure due to product bug or failure not due to product bug within the first period of time.

5. The method of claim 4, wherein each of the plurality of terms is multiplied by a weighting factor.

6. The method of claim 5, further comprising adjusting the first time period, the second time period, and the weighting factor of each of the plurality of terms between each iteration to further optimize the test scripts.

7. The method of claim 4, wherein determining the members of the collection comprises removing from the collection the test scripts having a score higher than a threshold score.

8. The method of claim 3, wherein the execution data comprises an execution time of the test script and a number of unique lines of code tested by the test script.

9. The method of claim 8, wherein calculating a score for each of the plurality of test scripts comprises adding a plurality of terms comprising an average of the execution time of the test script over a period of time, a number of executions of the test script resulting in the test result of failure due to product bug within the period of time, a ratio of the number of executions of the test script resulting in the test result of failure not due to product bug to the number of executions of the test script within the period of time, and the number of unique lines of code tested by the test script, and wherein each of the plurality of terms is multiplied by a weighting factor.

10. The method of claim 9, further comprising adjusting the period of time and the weighting factor of each of the plurality of terms between each iteration to further optimize the test scripts.

11. The method of claim 9, wherein determining the members of the collection comprises removing from the collection the test scripts having a score lower than a threshold score.

12. A method for ranking and optimizing automated test scripts, the method comprising:

collecting in a repository a test result and execution data for an execution of each of a plurality of test scripts, each of the plurality of test scripts belonging to a collection which determines a frequency of execution of the test script;

calculating a score for each of the plurality of test scripts based upon the test results and execution data;

ranking the plurality of test scripts according to the score; and

determining to which collection each of the plurality of test scripts belongs based upon the ranking of the test scripts.

13. The method of claim 12, wherein the repository comprises test results and execution data from of a plurality of executions of each of the plurality of test scripts.

14. The method of claim 12, wherein the score calculated for each of the plurality of test scripts indicates a propensity of the test script to result in a test result of failure not due to a product defect, and wherein ranking the plurality of test scripts comprises ranking the test scripts in descending order of propensity to result in a test result of failure not due to a product defect.

15. The method of claim 12, where the score calculated for each of the plurality of test scripts indicates an effectiveness of the test script in finding a product defect, and wherein ranking the plurality of test scripts comprises ranking the test scripts in ascending order of effectiveness in finding a product defect.

16. The method of claim 12, wherein the score calculated for each of the plurality of test scripts combines a propensity of the test script to result in a test result of failure not due to a product defect and an effectiveness of the test script in finding a product defect.

17. A computer storage medium having computer executable instructions stored thereon, which, when executed by a computer, cause the computer to:

collect in a repository a test result and execution data for a plurality of executions of each of a plurality of test scripts;

calculate a score for each of the plurality of test scripts based upon the test results and execution data;

rank the plurality of test scripts according to the score;

present to a test reviewer a list of the plurality test scripts sorted according to the rank; and

receive from the test reviewer a change in frequency of execution for a subset of test scripts in the list.

18. The computer storage medium of claim 17, wherein the score calculated for each of the plurality of test scripts indicates a propensity of the test script to result in a test result of failure not due to a product defect, and wherein ranking the plurality of test scripts comprises ranking the test scripts in descending order of propensity to result in a test result of failure not due to a product defect.

19. The computer storage medium of claim 17, where the score calculated for each of the plurality of test scripts indicates an effectiveness of the test script in finding a product defect, and wherein ranking the plurality of test scripts comprises ranking the test scripts in ascending order of effectiveness in finding a product defect.

20. The computer storage medium of claim 17, wherein the score calculated for each of the plurality of test scripts combines a propensity of the test script to result in a test result of failure not due to a product defect and an effectiveness of the test script in finding a product defect.