Test prioritization techniques

• US 9,032,259 B1
• Filed: 02/29/2012
• Issued: 05/12/2015
• Est. Priority Date: 02/29/2012
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Hood, James L.
• Primary Examiner(s): Leibovich, Yair
• Application Number:
  US13/408,673
• Time in Patent Office: 1,168 Days
• Field of Search: 714/32, 714/27, 714/48, 714/57
• US Class Current: 714/57
• CPC Class Codes: G06F 11/34 Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
• CPC Class Codes: G06F 11/3692 for test results analysis

Abstract

Systems and methods involve prioritizing information based at least in part on test results for tests. A computing device may administer one or more tests and/or may receive test results for one or more tests. Multiple executions of one or more tests may be administered over a period of time. A device administering a test may evaluate the functionality of at least a portion of an application programming interface (API) or at least a portion of a user interface. Test results may be analyzed to determine a failure pattern and/or pass rate for one or more tests. Test results may be analyzed to determine an error signature and/or error signature frequency for one or more test results. A report can be generated that prioritizes information based at least in part on the tests, test results, and/or any determined information.

Claims

1. A computer-implemented method for prioritizing tests based on test results, comprising:

under the control of one or more computer systems configured with executable instructions,

receiving a plurality of test results for each of a plurality of tests, each test result comprising an indicator indicating whether the test was successful or unsuccessful, each test result having an indicator that indicates that the test was unsuccessful further comprising an error message;

determining, for each test in the plurality of tests, a failure pattern selected from a plurality of predetermined failure pattern categories, wherein the failure pattern is selected based at least in part on a pattern of indicators for successive test results of the test, the indicators indicating whether the test was successful or unsuccessful;

determining, for each test in the plurality of tests, a pass rate based at least in part on a proportion of the test results having indicators indicating that the test was successful;

determining, for each test result having the indicator indicating that the test was unsuccessful, an error signature for the test result based at least in part on the error message;

determining, for each different error signature associated with each test, an error signature frequency based at least in part on a proportion of the test results for the test associated with the error signature; and

generating a report, the report prioritizing the tests based at least in part on the determined failure patterns, pass rates, and error signature frequencies, wherein the generated report is prioritized first by the determined failure patterns, then by the determined pass rates within each failure pattern, and then by the determined error signature frequencies within each pass rate.

2. The computer-implemented method of claim 1, wherein at least one of the tests in the plurality of tests is configured to evaluate functionality of at least part of an application programming interface (API) or at least part of a user interface.

3. The computer-implemented method of claim 1, wherein the plurality of predetermined failure pattern categories comprises at least one of always passes, always fails, appears broken, appears fixed, or undetermined.

4. The computer-implemented method of claim 1, wherein each of at least a subset of the plurality of tests is configured to evaluate functionality of a different service from a plurality of interdependent services.

5. The computer-implemented method of claim 1, wherein determining the error signature for the test result comprises hashing at least a portion of the error message for the test result.

6. The computer-implemented method of claim 1, wherein generating the report comprises:

ordering at least a portion of the information in the report based at least in part on the determined failure patterns, pass rates, or error signature frequencies.

7. The computer-implemented method of claim 6, further comprising:

determining, for each test in the plurality of tests, a maximum error signature frequency for the test, wherein ordering the portion of the information in the report is based at least in part on the determined maximum error signature frequencies.

8. A computer-implemented method for categorizing test results, comprising:

under the control of one or more computer systems configured with executable instructions,

receiving a plurality of test results for each of a plurality of tests, each test result comprising an indicator indicating whether the test was successful or unsuccessful;

determining, for each of at least a portion of the plurality of tests, a failure pattern being a sequence of indicators for successive test results for the test, a pass rate based at least in part on a proportion of the test results having indicators indicating that an execution of the test was successful, and an error frequency based at least in part on a frequency with which an error type occurs, the error type determined based at least in part on the failure pattern; and

generating a report, wherein the generated report is prioritized first by the determined failure patterns, then by the determined pass rates within each failure pattern, and then by the determined error frequencies within each pass rate.

9. The computer-implemented method of claim 8, further comprising:

determining, for each of at least a portion of the plurality of test results, an error signature, the error signature based at least in part on at least a portion of an error message associated with the test; and

wherein the generated report is based at least in part on the determined error signatures.

10. The computer-implemented method of claim 9, wherein:

the method further comprises determining, for each of at least a portion of the determined error signatures, an error signature frequency, wherein the error signature frequency based at least in part on a proportion of the error signatures; and

the generated report is based at least in part on the determined error signature frequencies.

11. The computer-implemented method of claim 9, wherein:

the method further comprises determining, for each of at least a subset of the plurality of tests, a maximum error signature frequency, wherein the maximum error signature frequency is based at least in part on the error signature frequency; and

the generated report is based at least in part on the determined maximum error signature frequencies.

12. The computer-implemented method of claim 8, wherein the generated report arranges the portion of the tests from a highest priority to a lowest priority.

13. The computer-implemented method of claim 8, further comprising:

sending the generated report to at least one electronic address.

14. A computer system for generating prioritized reports, comprising:

one or more processors; and

memory, including executable instructions that, when executed by the one or more processors, cause the computer system to at least;

receive a plurality of test results for each of a plurality of tests, each test result comprising an indicator indicating whether the test was successful or unsuccessful, each test result of at least a subset of test results having an indicator that indicates that the test was unsuccessful further comprising an error message;

determine, for each of at least a subset of the plurality of tests, a categorization based at least in part on a pattern of indicators for that test;

determine, for each of at least a portion of the test results having an indicator indicating that the test was unsuccessful, an error signature for the test result, the error signature based at least in part on the error message;

determine, for each of at least a portion of the determined error signatures, an error signature frequency based at least in part on a proportion of the determined error signatures having the same error signature; and

generate a report, the report providing an indication of a priority for at least a portion of the report, based at least in part on the categorization of each test, wherein the generated report is prioritized first by the determined categorization, then by a pass rate calculated as a proportion of successful indicators in each pattern of indicators, and then by the determined error signature frequency.

15. The computer system of claim 14, further including executable instructions that, when executed by the one or more processors, cause the computer system to at least:

determine, for each of at least a portion of the plurality of tests, a failure pattern based at least in part on indicators for successive test results for the test, the failure pattern selected from a plurality of predetermined failure pattern categories; and

wherein the generated report is based at least in part on the determined failure patterns.

16. The computer system of claim 14, further including executable instructions that, when executed by the one or more processors, cause the computer system to at least:

determine, for each of at least a portion of the plurality of tests, a failure rate based at least in part on the number of the plurality of test results associated with the test; and

wherein the generated report is based at least in part on the determined failure rates.

17. The computer system of claim 14 wherein determining the error signature comprises hashing at least a portion of the error message associated with the test result.

18. The computer system of claim 14, wherein generating the report comprises:

determining, for each of at least a portion of the test results, a priority based at least in part on determined error signatures;

determining an order for the portion of the test results, the order indicating the priority of the test results; and

generating the report based at least in part on the determined order for the portion of the test results.

19. The computer system of claim 14, wherein generating the report comprises:

determining at least a portion of the report having a high importance; and

visually indicating the high importance of the portion of the report.

20. One or more non-transitory computer-readable storage media having collectively stored thereon instructions that, when executed by one or more processors of a computer system, cause the computer system to at least:

receive a plurality of test results for each of a plurality of tests, each test result comprising an indicator indicating whether the test was successful or unsuccessful, each test result of at least a subset of test results having an indicator that indicates that the test was unsuccessful further comprising an error message;

determine, for each of at least a portion of the plurality of tests, a failure pattern, the failure pattern selected from a plurality of predetermined failure pattern categories by comparing a sequence of indicators for the test to the predetermined failure pattern categories;

determine, for each of at least a portion of the plurality of test results having an indicator indicating that the test was unsuccessful, an error signature for the test result based at least in part on the error message;

determine, for at least a portion of the determined error signatures, an error signature frequency based at least in part on a proportion of the test results for the test associated with the error signature; and

generate a report, wherein the generated report is prioritized first by the determined failure pattern, then by a pass rate calculated as a proportion of indicators indicating a successful test to total indicators for each test, and then by the determined error signature frequencies within each pass rate.

21. The one or more non-transitory computer-readable storage media of claim 20, further comprising instructions that, when executed by a computer system, cause the computer system to at least:

send at least a portion of the generated report to at least one electronic address.

22. The one or more non-transitory computer-readable storage media of claim 20, further comprising instructions that, when executed by a computer system, cause the computer system to at least:

obtain historical information associated with at least one error signature; and

based at least in part on the obtained historical information, determine a potential solution associated with the at least one error signature, wherein the generated report includes the potential solution.

23. The one or more non-transitory computer-readable storage media of claim 20 wherein the error signature is determined at least by hashing a portion of the error message associated with the test result.

24. The one or more non-transitory computer-readable storage media of claim 20, further comprising instructions that, when executed by a computer system, cause the computer system to at least:

receive a request for the report;

in response to receiving the request for the report, dynamically generate the report; and

display the report, wherein at least a portion of the report is ordered based at least in part on the determined error signatures.