Parameterized test driven development

• US 7,681,180 B2
• Filed: 06/06/2007
• Issued: 03/16/2010
• Est. Priority Date: 06/06/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Tillmann, Nikolai, de Halleux, Jonathan Paul, Schulte, Wolfram
• Primary Examiner(s): Yigdall; Michael J
• Application Number:
  US11/759,144
• Publication Number:
  US 20080307264A1
• Time in Patent Office: 1,014 Days
• Field of Search: None
• US Class Current: 717/124
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites

Abstract

In one embodiment a computer system automatically generates unit tests. The computer system accesses a parameterized unit test that provides a base outline from which one or more unit tests are automatically generated, generates input parameter values for a unit of software code, automatically generates a unit test configured to assess the functionality of the unit of software code, and receives test results from a software testing program and provides feedback to a user. In other embodiments, a computer system automatically maintains a unit test database. The computer system receives a unit test at a unit test database, assigns a test identity to the received unit test, determines that the test identity assigned to the received unit test is unique when compared to other unit tests, determines that the received unit test has different functionality coverage characteristics, and adds the received unit test to the unit test database.

Claims

1. At a computer system, a method for automatically generating unit tests by instantiating one or more generated input parameter values, the unit tests being derived from a corresponding parameterized unit test, the method comprising:

an act of accessing a parameterized unit test, the parameterized unit test configured to test any of a plurality of behaviors of the unit of software code by providing a base outline from which any of a plurality of corresponding other unit tests can be formed to test individual behaviors from among the plurality of behaviors;

an act of generating input parameter values for testing an individual behavior of the unit of software code;

an act of automatically generating a unit test from the parameterized unit test by applying the one or more generated input parameter values to the parameterized unit test so as to modify the base outline into a unit test specifically configured for testing the individual behavior of the unit of software code, the unit test configured to assess the functionality of the individual behavior of the unit of software code, the generated input parameter values corresponding to parameters in the unit of software code that are to be tested;

an act of submitting the unit of software code and the automatically generated unit test to a software testing program to test execution of the unit of software code based on the automatically generated unit test;

an act of receiving test results from the software testing program, the test results including a test results status generated from executing the unit of software code based on the automatically generated unit test; and

an act of providing feedback to a user, the feedback indicating at least the test results status.

2. The method of claim 1, further comprising generating the generated input parameter values for the parameterized unit test using symbolic execution.

3. The method of claim 1, further comprising generating the generated input parameter values for the parameterized unit test using pairwise combinatorial techniques.

4. The method of claim 1, wherein the feedback comprises code suggestions that indicate how to fix a detected error.

5. The method of claim 4, wherein the feedback is presented as an in-line hint in the software code.

6. The method of claim 4, further comprising:

an act of automatically applying the code suggestions to the software code unit; and

an act of automatically re-submitting the unit of software code and the automatically generated unit test to the software testing program subsequent to automatically applying the code suggestions so as to re-test execution of the unit of software code with the automatically applied code suggestions.

7. The method of claim 1, further comprising an act of searching for new input values for new tests subsequent to submitting the unit of software code and the automatically generated unit test and prior to receiving test results such that new input values are searched for while the software testing program tests execution of the unit of software code.

8. The method of claim 1, further comprising:

an act of determining that the received test results indicate that a desired level of test quality was reached; and

based on the determination, an act of generating a new unit test.

9. The method of claim 1, further comprising:

an act of determining that the received test results indicate that a desired level of test quality was not reached; and

based on the determination, an act of revising the software code unit such that a subsequent test of the software code unit yields the desired level of test quality.

10. The method of claim 1, wherein the base outline comprises one or more software functions that are to be tested.

11. A computer program product for use at a computer system, the computer program product for implementing a method for automatically generating unit tests by instantiating one or more generated input parameter values, the unit tests being derived from a corresponding parameterized unit test, the computer program product comprising one or more computer storage media having stored thereon computer-executable instructions that, when executed at a processor, cause the computer system to perform the method, including the following:

access a parameterized unit test, the parameterized unit test configured to test any of a plurality of behaviors of the unit of software code by providing a base outline from which any of a plurality of corresponding other unit tests can be formed to test individual behaviors from among the plurality of behaviors;

generate input parameter values for testing an individual behavior of the unit of software code;

generate a unit test from the parameterized unit test by applying the one or more generated input parameter values to the parameterized unit test so as to modify the base outline into a unit test specifically configured for testing the individual behavior of the unit of software code, the unit test configured to assess the functionality of the individual behavior of the unit of software code, the generated input parameter values corresponding to parameters in the unit of software code that are to be tested;

submit the unit of software code and the automatically generated unit test to a software testing program to test execution of the unit of software code based on the automatically generated unit test;

receive test results from the software testing program, the test results including a test results status generated from executing the unit of software code based on the automatically generated unit test; and

provide feedback to a user, the feedback indicating at least the test results status.

12. The computer program product of claim 11, further comprising computer-executable instructions that, when executed, cause the computer system to generate the generated input parameter values for the parameterized unit test using symbolic execution.

13. The computer program product of claim 11, further comprising computer-executable instructions that, when executed, cause the computer system to generate the generated input parameter values for the parameterized unit test using pairwise combinatorial techniques.

14. The computer program product of claim 11, wherein the feedback comprises code suggestions that indicate how to fix a detected error.

15. The computer program product of claim 14, wherein the feedback is presented as an in-line hint in the software code.

16. The computer program product of claim 14, further comprising computer-executable instructions that, when executed, cause the computer system to:

apply the code suggestions to the software code unit; and

re-submit the unit of software code and the automatically generated unit test to the software testing program subsequent to automatically applying the code suggestions so as to re-test execution of the unit of software code with the automatically applied code suggestions.

17. The computer program product of claim 11, further comprising computer-executable instructions that, when executed, cause the computer system to search for new input values for new tests subsequent to submitting the unit of software code and the automatically generated unit test and prior to receiving test results such that new input values are searched for while the software testing program tests execution of the unit of software code.

18. The computer program product of claim 11, further comprising computer-executable instructions that, when executed, cause the computer system to:

determine that the received test results indicate that a desired level of test quality was reached; and

based on the determination, generate a new unit test.

19. The computer program product of claim 11, further comprising computer-executable instructions that, when executed, cause the computer system to:

determine that the received test results indicate that a desired level of test quality was not reached; and

based on the determination, revise the software code unit such that a subsequent test of the software code unit yields the desired level of test quality.

20. The computer program product of claim 11, wherein the base outline comprises one or more software functions that are to be tested.