Method and apparatus for ordering test cases for regression testing

• US 7,178,063 B1
• Filed: 07/22/2003
• Issued: 02/13/2007
• Est. Priority Date: 07/22/2003
• Status: Expired due to Fees

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Smith, Zachary S.
• Primary Examiner(s): LE, DIEU MINH T
• Application Number:
  US10/624,767
• Time in Patent Office: 1,302 Days
• Field of Search: 714/38, 714/37, 714/39, 714/25, 717/124
• US Class Current: 714/38.1
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites

Abstract

A method and apparatus for ordering test cases for regression testing in a regression test system. The regression testing may include executing ordered test cases identified and ordered according to preferences. Each prospective test case may be assigned a result record for retaining previous testing history of the specific test case as well as the design model tested. The test may be selected according to preferences that yield more timely discovery of design bugs. Once ordered, the ordered test cases may be tested against a current design model, which results in test results, at least a portion of which updates the corresponding result record.

Claims

1. A method of performing a regression test of ordered test cases, comprising:

assigning a result record to each of a plurality of candidate test cases, said result record including test results from testing said each of said plurality of candidate test cases against a design model;

selecting said ordered test cases from said plurality of candidate test cases according to said test results in said result record of said each of said plurality of candidate test cases,

wherein said selecting said ordered test cases comprises;

excluding noninsightful ones of said plurality of candidate test cases from said ordered test cases; and

preferentially ordering remaining ones of said plurality of candidate test cases to form ordered test cases comprises;

dividing said remaining ones of said plurality of candidate test cases into shorter execution ones and longer execution ones;

separately ordering said shorter execution ones into shorter-execution ordered test cases and said longer execution ones into longer-execution ordered test cases; and

interleaving said shorter-execution ordered test cases and said longer-execution ordered test cases to form ordered test cases;

testing said ordered test cases against a current design model; and

updating said result record of each of said ordered test cases in response to new test results from said testing.

2. The method of claim 1, wherein said excluding comprises excluding one or more of said plurality of candidate test cases when said result record specifies at least one of test case waived, test case currently executing, test case unable to complete previous execution attempt, test case previously executed on current design model, and previously identified bug detected in said test case not fixed in current design model.

3. The method of claim 1, wherein preferentially ordering includes preferring ones of said remaining ones of said plurality of candidate test cases having above-average coverage of said current design model.

4. The method of claim 1, wherein preferentially ordering further includes preferring ones of said remaining ones of said plurality of candidate test cases having shorter testing duration on said design model.

5. The method of claim 1, wherein preferentially ordering further includes preferring not previously executed ones of said remaining ones of said plurality of candidate test cases.

6. The method of claim 1, wherein preferentially ordering further includes preferring ones of said remaining ones of said plurality of candidate test cases having longer idle durations.

7. The method of claim 1, wherein preferentially ordering further includes preferring ones of said remaining ones of said plurality of candidate test cases having greater failing histories.

8. A computer-readable medium having computer-executable instructions for performing regression test of ordered test cases, said test comprising instructions for:

assigning a result record to each of a plurality of candidate test cases, said result record including test results from testing said each of said plurality of candidate test cases against a design model;

selecting said ordered test cases from said plurality of candidate test cases according to said test results in said result record of said each of said plurality of candidate test cases, comprises;

excluding noninsightful ones of said plurality of candidate test cases from said ordered test cases; and

preferentially ordering remaining ones of said plurality of test cases to form ordered test cases, wherein said preferentially ordering comprises;

dividing said remaining ones of said plurality of test cases into shorter execution ones and longer execution ones;

separately ordering said shorter execution ones into shorter-execution ordered test cases and said longer execution ones into longer-execution ordered test cases; and

interleaving said shorter-execution ordered test cases and said longer-execution test cases to form ordered test cases;

testing said ordered test cases against a current design model; and

updating said result record of said ordered test cases in response to new test results from said testing.

9. The computer-readable medium of claim 8, wherein said computer-executable instructions for excluding comprise computer-executable instructions for excluding one or more of said plurality of candidate test cases when said result record specifies at least one of test case waived, test case currently executing, test case unable to complete previous execution attempt, test case previously executed on current design model, and previously identified bug detected in said test case not fixed in current design model.

10. The computer-readable medium of claim 8, wherein said computer-executable instructions for preferentially ordering further comprise computer-executable instructions for preferring ones of said remaining ones of said plurality of test cases having above-average coverage of said current design model.

11. The computer-readable medium of claim 8, wherein said computer-executable instructions for preferentially ordering further comprise computer-executable instructions for preferring ones of said remaining ones of said plurality of test cases having shorter testing duration on said design model.

12. The computer-readable medium of claim 8, wherein said computer-executable instructions for preferentially ordering further comprise computer-executable instructions for preferring not previously executed ones of said remaining ones of said plurality of test cases.

13. The computer-readable medium of claim 8, wherein said computer-executable instructions for preferentially ordering further comprise computer-executable instructions for preferring ones of said remaining ones of said plurality of test cases having longer idle durations.

14. The computer-readable medium of claim 8, wherein said computer-executable instructions for preferentially ordering further comprise computer-executable instructions for preferring ones of said remaining ones of said plurality of test cases having greater failing histories.

15. An ordered regression test system, comprising:

a test case database including a plurality of candidate test cases, each one of said plurality of candidate test cases including a result record including test results from testing said one of said plurality of test cases against a design model;

a test case ordering process to select said ordered test cases from said plurality of candidate test cases according to said test results in said result record of said each of said plurality of candidate test cases comprises;

excluding noninsightful ones of said plurality of candidate test cases from said ordered test cases; and

preferentially ordering remaining ones of said plurality of candidate test cases to form ordered test cases comprises;

dividing said remaining ones of said plurality of candidate test cases into shorter execution ones and longer execution ones;

separately ordering said shorter execution ones into shorter-execution ordered test cases and said longer execution ones into longer-execution ordered test cases; and

interleaving said shorter-execution ordered test cases and said longer-execution ordered test cases to form ordered test cases;

a current design model to undergo regression testing; and

one or more testers for testing said ordered test cases against a current design model and generating new test results to update said result record.

16. The ordered regression test system of claim 15, further comprising a tester scheduling process operably coupled between said test case ordering process and said one or more testers, said tester scheduling process configured to distribute portions of said ordered test cases for execution among said one or more testers.

17. An ordered regression test system, comprising:

means for assigning a result record to each of a plurality of candidate test cases, said result record including test results from testing said each of said plurality of candidate test cases against a design model;

means for selecting said ordered test cases from said plurality of candidate test cases according to said test results in said result record of said each of said plurality of candidate test cases, wherein said means for selecting said ordered test cases comprises;

means for excluding noninsightful ones of said plurality of candidate test cases from said ordered test cases; and

means for preferentially ordering remaining ones of said plurality of candidate test cases to form ordered test cases comprises;

means for dividing said remaining ones of said plurality of candidate test cases into shorter execution ones and longer execution ones;

means for separately ordering said shorter execution ones into shorter-execution ordered test cases and said longer execution ones into longer-execution ordered test cases; and

means for interleaving said shorter-execution ordered test cases and said longer-execution ordered test cases to form ordered test cases;

means for testing said ordered test cases against a current design model; and

means for updating said result record of said ordered test cases in response to new test results from said testing.