Automatically generating observations of program behavior for code testing purposes

• US 20050278577A1
• Filed: 06/01/2004
• Published: 12/15/2005
• Est. Priority Date: 06/01/2004
• Status: Abandoned Application

• Current Assignee: Agitar Software Incorporated (Oridium LLC)
• Original Assignee: Agitar Software Incorporated (Oridium LLC)
• Inventors: Doong, Roongko, Pio di Savoia, Luigi Alberto
• Application Number:
  US10/859,265
• Publication Number:
  US 20050278577A1
• Time in Patent Office: Days
• US Class Current: 714/38
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites

Abstract

One embodiment of the present invention provides a system that automatically generates observations of program behavior for code testing purposes. During operation, the system analyzes the code-under-test to determine a set of test inputs. Next, the system exercises the code-under-test on the set of test inputs to produce a set of test results. Finally, the system analyzes the set of test results to automatically generate observations, wherein the observations are boolean-valued expressions containing variables and/or constants which are consistent with the set of test inputs and the set of test results.

Claims

1. A method for automatically generating observations of program behavior for code testing purposes, comprising:

analyzing code-under-test to determine a set of test inputs;

exercising the code-under-test on the set of test inputs to produce a set of test results; and

analyzing the set of test results to automatically generate observations, wherein the observations are boolean-valued expressions containing variables and/or constants which are consistent with the set of test inputs and the set of test results.

2. The method of claim 1, wherein analyzing the code-under-test to determine the set of test inputs involves:

analyzing the code-under-test to determine test data for the code-under-test, and to determine a number of test executions; and

producing the set of test inputs by creating various combinations of the test data to exercise code-under-test.

3. The method of claim 1, wherein the method further comprises promoting observations to become assertions by:

presenting the observations, which were automatically generated, to a user;

allowing the user to select observations that reflect intended behavior of the code-under-test; and

promoting the selected observations to become assertions, which will be verified when a subsequent version of the code-under-test is exercised during subsequent testing.

4. The method of claim 3, wherein the method additionally involves allowing the user to:

manually enter assertions; and

to

modify observations (or assertions) to produce assertions.

5. The method of claim 3, wherein presenting the observations to the user involves filtering and/or ranking the observations based on a relevance score before presenting the observations to the user.

6. The method of claim 3, wherein the method further comprises verifying assertions by:

exercising the subsequent version of the code-under-test on a subsequent set of test inputs to produce a subsequent set of test results;

verifying that the assertions hold for the subsequent set of test inputs and the subsequent set of test results; and

reporting pass/fail results for the assertions to the user, thereby allowing the user to fix any problems indicated by the pass/fail results.

7. The method of claim 6, wherein prior to exercising the subsequent version of the code-under-test, the method further comprises analyzing the subsequent version of the code-under-test to determine the subsequent set of test inputs to be used while exercising the subsequent version of the code-under-test.

8. The method of claim 1, wherein automatically generating the observations involves generalizing the observations, whenever possible, by using variables instead of constants in the corresponding boolean-valued expressions.

9. The method of claim 1, wherein automatically generating the observations can involve:

partitioning the test results based on one or more outcome conditions specified in the set of test results; and

generating observations for separately for each partition.

10. The method of claim 1, wherein automatically generating the observations involves:

analyzing the code-under-test to produce a set of candidate boolean-valued expressions;

eliminating any candidate boolean-valued expressions which are not consistent with the set of test inputs and the set of test results; and

promoting remaining candidate expressions, which were not eliminated, to become observations.

11. The method of claim 1, wherein the boolean-valued expressions that comprise the observations can include:

inputs to the code-under-test;

results produced by the code-under test;

variables within the code-under-test, which are visible/accessible outside of a method (or function) body;

observations of the state of the system obtained through programmatic interface; and

properties of objects.

12. The method of claim 1, wherein the boolean-valued expressions that comprise the observations can include:

boolean operators or functions;

relational operators or functions;

arithmetic operators or functions;

operators or functions on objects; and

operators or functions on types.

13. The method of claim 1, wherein exercising the code-under-test involves:

compiling the code-under-test to produce executable code; and

executing the executable code using the set of test inputs to produce the set of test results.

14. A computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for automatically generating observations of program behavior for code testing purposes, the method comprising:

analyzing code-under-test to determine a set of test inputs;

exercising the code-under-test on the set of test inputs to produce a set of test results; and

analyzing the set of test results to automatically generate observations, wherein the observations are boolean-valued expressions containing variables and/or constants which are consistent with the set of test inputs and the set of test results.

15. The computer-readable storage medium of claim 14, wherein analyzing the code-under-test to determine the set of test inputs involves:

analyzing the code-under-test to determine test data for the code-under-test, and to determine a number of test executions; and

producing the set of test inputs by creating various combinations of the test data to exercise code-under-test.

16. The computer-readable storage medium of claim 14, wherein the method further comprises promoting observations to become assertions by:

presenting the observations, which were automatically generated, to a user;

allowing the user to select observations that reflect intended behavior of the code-under-test; and

promoting the selected observations to become assertions, which will be verified when a subsequent version of the code-under-test is exercised during subsequent testing.

17. The computer-readable storage medium of claim 16, wherein the method additionally involves allowing the user to:

manually enter assertions; and

to modify observations (or assertions) to produce assertions.

18. The computer-readable storage medium of claim 16, wherein presenting the observations to the user involves filtering and/or ranking the observations based on a relevance score before presenting the observations to the user.

19. The computer-readable storage medium of claim 16, wherein the method further comprises verifying assertions by:

exercising the subsequent version of the code-under-test on a subsequent set of test inputs to produce a subsequent set of test results;

verifying that the assertions hold for the subsequent set of test inputs and the subsequent set of test results; and

reporting pass/fail results for the assertions to the user, thereby allowing the user to fix any problems indicated by the pass/fail results.

20. The computer-readable storage medium of claim 19, wherein prior to exercising the subsequent version of the code-under-test, the method further comprises analyzing the subsequent version of the code-under-test to determine the subsequent set of test inputs to be used while exercising the subsequent version of the code-under-test.

21. The computer-readable storage medium of claim 14, wherein automatically generating the observations involves generalizing the observations, whenever possible, by using variables instead of constants in the corresponding boolean-valued expressions.

22. The computer-readable storage medium of claim 14, wherein automatically generating the observations can involve:

partitioning the test results based on one or more outcome conditions specified in the set of test results; and

generating observations for separately for each partition.

23. The computer-readable storage medium of claim 14, wherein automatically generating the observations involves:

analyzing the code-under-test to produce a set of candidate boolean-valued expressions;

eliminating any candidate boolean-valued expressions which are not consistent with the set of test inputs and the set of test results; and

promoting remaining candidate expressions, which were not eliminated, to become observations.

24. The computer-readable storage medium of claim 14, wherein the boolean-valued expressions that comprise the observations can include:

inputs to the code-under-test;

results produced by the code-under test;

variables within the code-under-test, which are visible/accessible outside of a method (or function) body;

observations of the state of the system obtained through programmatic interface; and

properties of objects.

25. The computer-readable storage medium of claim 14, wherein the boolean-valued expressions that comprise the observations can include:

boolean operators or functions;

relational operators or functions;

arithmetic operators or functions;

operators or functions on objects; and

operators or functions on types.

26. The computer-readable storage medium of claim 14, wherein exercising the code-under-test involves:

compiling the code-under-test to produce executable code; and

executing the executable code using the set of test inputs to produce the set of test results.

27. An apparatus that automatically generates observations of program behavior for code testing purposes, comprising:

a test input generation mechanism configured to analyze code-under-test to determine a set of test inputs;

an execution mechanism configured to exercise the code-under-test on the set of test inputs to produce a set of test results; and

an observation generation mechanism configured to analyze the set of test results to automatically generate observations, wherein the observations are boolean-valued expressions containing variables and/or constants which are consistent with the set of test inputs and the set of test results.

28. The apparatus of claim 27, wherein the test input generation mechanism is configured to:

analyze the code-under-test to determine test data for the code-under-test, and to determine a number of test executions; and

to

produce the set of test inputs by creating various combinations of the test data to exercise code-under-test.

29. The apparatus of claim 27, wherein the apparatus further comprises an assertion generation mechanism, which is configured to:

present the observations, which were automatically generated, to a user;

allow the user to select observations that reflect intended behavior of the code-under-test; and

to

promote the selected observations to become assertions, which will be verified when a subsequent version of the code-under-test is exercised during subsequent testing.

30. The apparatus of claim 29, wherein the assertion generation mechanism is additionally configured to allow the user to:

manually enter assertions; and

to

modify observations (or assertions) to produce assertions.

31. The apparatus of claim 29, wherein while presenting the observations to the user, the assertion generation mechanism is configured to filter and/or rank the observations based on a relevance score before presenting the observations to the user.

32. The apparatus of claim 29, wherein the apparatus further comprises an assertion verification mechanism, which is configured to:

exercise the subsequent version of the code-under-test on a subsequent set of test inputs to produce a subsequent set of test results;

verify that the assertions hold for the subsequent set of test inputs and the subsequent set of test results; and

to

report pass/fail results for the assertions to the user, thereby allowing the user to fix any problems indicated by the pass/fail results.

33. The apparatus of claim 32, wherein prior to exercising the subsequent version of the code-under-test, the test input generation mechanism is configured to analyze the subsequent version of the code-under-test to determine the subsequent set of test inputs to be used while exercising the subsequent version of the code-under-test.

34. The apparatus of claim 27, wherein while automatically generating the observations, the observation generation mechanism is configured to generalize the observations, whenever possible, by using variables instead of constants in the corresponding boolean-valued expressions.

35. The apparatus of claim 27, wherein while automatically generating the observations, the observation generation mechanism is configured to:

partition the test results based on one or more outcome conditions specified in the set of test results; and

to

generate observations for separately for each partition.

36. The apparatus of claim 27, wherein while automatically generating the observations, the observation generation mechanism is configured to:

analyze the code-under-test to produce a set of candidate boolean-valued expressions;

eliminate any candidate boolean-valued expressions which are not consistent with the set of test inputs and the set of test results; and

to

promote remaining candidate expressions, which were not eliminated, to become observations.

37. The apparatus of claim 27, wherein the boolean-valued expressions that comprise the observations can include:

inputs to the code-under-test;

results produced by the code-under test;

variables within the code-under-test, which are visible/accessible outside of a method (or function) body;

observations of the state of the system obtained through programmatic interface; and

properties of objects.

38. The apparatus of claim 27, wherein the boolean-valued expressions that comprise the observations can include:

boolean operators or functions;

relational operators or functions;

arithmetic operators or functions;

operators or functions on objects; and

operators or functions on types.

39. The apparatus of claim 27, wherein while exercising the code-under-test the execution mechanism is configured to:

compile the code-under-test to produce executable code; and

to

execute the executable code using the set of test inputs to produce the set of test results.

40. A means for automatically generating observations of program behavior for code testing purposes, comprising:

a test input generation means for analyzing code-under-test to determine a set of test inputs;

an execution means for exercising the code-under-test on the set of test inputs to produce a set of test results; and

an observation generation means for analyzing the set of test results to automatically generate observations, wherein the observations are boolean-valued expressions containing variables and/or constants which are consistent with the set of test inputs and the set of test results.