Configurable testing of computer programs

• US 9,274,926 B2
• Filed: 01/28/2014
• Issued: 03/01/2016
• Est. Priority Date: 01/03/2013
• Status: Active Grant

• Current Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Original Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Inventors: Larson, Brond, Offner, Carl, Wholey, Joseph Skeffington III
• Primary Examiner(s): Das, Chameli
• Application Number:
  US14/165,698
• Publication Number:
  US 20140189653A1
• Time in Patent Office: 763 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3604 Software analysis for verifying properties of programs byte-code verification G06F9/44589
• CPC Class Codes: G06F 11/368 for test version control, e.g. updating test cases to a new software version
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites

Abstract

Configurable testing of a computer program includes: storing a set of one or more testing specifications, and attribute information defining one or more attributes of a recognizable portion of the computer program; and processing, using at least one processor, the computer program according to at least a first testing specification associated with the computer program. The processing includes: traversing a representation of the computer program that includes elements that represent recognizable portions of the computer program, and while traversing the representation, recognizing recognizable portions of the computer program, and storing values of attributes, defined by the attribute information, of the recognized portions of the computer program.

Claims

1. A computing system for configurable testing of a computer program, the computing system including:

a storage medium storing a set of one or more testing specifications, and attribute information defining one or more attributes of a recognizable portion of the computer program; and

a configurable testing module, including at least one processor, configured to process the computer program according to at least a first testing specification associated with the computer program, the first testing specification including at least a first test that applies to multiple portions of the computer program, the processing including;

traversing a representation of the computer program that includes elements that represent recognizable portions of the computer program,

while traversing the representation, recognizing recognizable portions of the computer program,

while traversing the representation, for at least a first recognized portion of the computer program, storing one or more values of one or more attributes of the first recognized portion of the computer program, wherein each attribute whose value is stored while traversing the representation is defined by the attribute information as being an attribute of one or more of the recognizable portions of the computer program,

performing any aggregation functions specified in the first testing specification based at least in part on stored values of attributes of one or more recognized portions of the computer program, and

performing the first test specified in the first testing specification based at least in part on stored results of any aggregation functions performed while traversing the representation.

2. The system of claim 1, wherein the processing further includes:

performing at least one test specified in at least the first testing specification based at least in part on stored values of attributes of one or more recognized portions of the computer program.

3. The system of claim 2, wherein the first testing specification includes a test that tests at least one characteristic of the computer program that does not represent an error that would prevent the computer program from executing.

4. The system of claim 2, wherein the first testing specification includes a test that tests at least one characteristic of the computer program associated with code reuse efficiency.

5. The system of claim 2, wherein the first testing specification includes a test that tests at least one characteristic of the computer program associated with memory use constraints.

6. The system of claim 5, wherein the characteristic of the computer program associated with memory use constraints includes a maximum amount of main memory that a particular recognized portion of the computer program is allowed to use when the computer program executes.

7. The system of claim 2, wherein the first testing specification includes a test that tests a number of times at least one particular operation is used in the computer program.

8. The system of claim 2, wherein the first testing specification includes a test that references a value of a name attribute of a recognizable portion of the computer program to determine whether to continue applying the test.

9. The system of claim 2, the processing further including presenting in a user interface a result of at least one test, the result being visually associated with a displayed portion of the computer program on which the test was performed.

10. The system of claim 1, wherein the storage medium stores activation information indicating which of the testing specifications are active.

11. The system of claim 10, wherein at least one of the testing specifications includes at least one local test that applies to a particular portion of the computer program, and at least one of the testing specifications includes at least one global test that applies to multiple portions of the computer program.

12. The system of claim 11, wherein the processing further includes:

accessing the activation information to determine one or more testing specifications that are indicated as active.

13. The system of claim 12, wherein the set of one or more testing specifications is a hierarchical set that relates at least some of the testing specifications to each other according to a hierarchy.

14. The system of claim 13, wherein accessing the activation information to determine one or more testing specifications that are indicated as active includes identifying active testing specifications along a path up the hierarchy between the first testing specification and a root of the hierarchy.

15. The system of claim 12, wherein the processing further includes:

while traversing the representation, performing any local tests specified in any active testing specifications based at least in part on stored values of attributes of one or more recognized portions of the computer program.

16. The system of claim 12, wherein the processing further includes:

while traversing the representation, performing any aggregation functions specified in any active testing specifications based at least in part on stored values of attributes of one or more recognized portions of the computer program.

17. The system of claim 16, wherein the processing further includes:

after traversing the representation, performing any global tests specified in any active testing specifications based at least in part on stored results of any aggregation functions performed while traversing the representation.

18. The system of claim 1, wherein the representation of the computer program is a graph based-representation and the elements include:

nodes that represent at least some of the recognizable portions, and links that represent relationships between the recognizable portions represented by the nodes.

19. The system of claim 18, wherein the graph-based representation includes a dataflow graph that includes:

processing components represented by the nodes, and flows of data between processing components represented by the links.

20. The system of claim 19, wherein at least some of the recognizable portions include a flow of data between processing components.

21. The system of claim 19, wherein at least some of the recognizable portions include an input or output port of a processing component.

22. The system of claim 19, wherein at least some of the recognizable portions include a parameter associated with a processing component or a flow of data between processing components.

23. The system of claim 18, wherein the graph-based representation includes a control flow graph that includes:

basic blocks of code represented by the nodes, and changes of control between basic blocks represented by the links.

24. The system of claim 1, the testing further including receiving user input for specifying at least one testing specification of the set of one or more testing specifications.

25. The system of claim 24, wherein at least some of the attribute information is predetermined before at least some of the user input is received.

26. A computing system for configurable testing of a computer program, the computing system including:

means for storing a set of one or more testing specifications, and attribute information defining one or more attributes of a recognizable portion of the computer program; and

means for processing the computer program according to at least a first testing specification associated with the computer program, the first testing specification including at least a first test that applies to multiple portions of the computer program, the processing including;

traversing a representation of the computer program that includes elements that represent recognizable portions of the computer program,

while traversing the representation, recognizing recognizable portions of the computer program,

while traversing the representation, for at least a first recognized portion of the computer program, storing one or more values of one or more attributes of the first recognized portion of the computer program, wherein each attribute whose value is stored while traversing the representation is defined by the attribute information as being an attribute of one or more of the recognizable portions of the computer program,

performing any aggregation functions specified in the first testing specification based at least in part on stored values of attributes of one or more recognized portions of the computer program, and

performing the first test specified in the first testing specification based at least in part on stored results of any aggregation functions performed while traversing the representation.

27. A method for configurable testing of a computer program, the method including:

storing a set of one or more testing specifications, and attribute information defining one or more attributes of a recognizable portion of the computer program; and

processing, using at least one processor, the computer program according to at least a first testing specification associated with the computer program, the first testing specification including at least a first test that applies to multiple portions of the computer program, the processing including;

traversing a representation of the computer program that includes elements that represent recognizable portions of the computer program,

while traversing the representation, recognizing recognizable portions of the computer program,

while traversing the representation, for at least a first recognized portion of the computer program, storing one or more values of one or more attributes of the first recognized portion of the computer program, wherein each attribute whose value is stored while traversing the representation is defined by the attribute information as being an attribute of one or more of the recognizable portions of the computer program,

performing any aggregation functions specified in the first testing specification based at least in part on stored values of attributes of one or more recognized portions of the computer program, and

performing the first test specified in the first testing specification based at least in part on stored results of any aggregation functions performed while traversing the representation.

28. The method of claim 27, wherein the processing further includes:

performing at least one test specified in at least the first testing specification based at least in part on stored values of attributes of one or more recognized portions of the computer program.

29. The method of claim 28, wherein the first testing specification includes a test that tests at least one characteristic of the computer program that does not represent an error that would prevent the computer program from executing.

30. The method of claim 28, wherein the first testing specification includes a test that tests at least one characteristic of the computer program associated with code reuse efficiency.

31. The method of claim 28, wherein the first testing specification includes a test that tests at least one characteristic of the computer program associated with memory use constraints.

32. The method of claim 31, wherein the characteristic of the computer program associated with memory use constraints includes a maximum amount of main memory that a particular recognized portion of the computer program is allowed to use when the computer program executes.

33. The method of claim 28, wherein the first testing specification includes a test that tests a number of times at least one particular operation is used in the computer program.

34. The method of claim 28, wherein the first testing specification includes a test that references a value of a name attribute of a recognizable portion of the computer program to determine whether to continue applying the test.

35. The method of claim 28, the processing further including presenting in a user interface a result of at least one test, the result being visually associated with a displayed portion of the computer program on which the test was performed.

36. The method of claim 27, wherein the storage medium stores activation information indicating which of the testing specifications are active.

37. The method of claim 36, wherein at least one of the testing specifications includes at least one local test that applies to a particular portion of the computer program, and at least one of the testing specifications includes at least one global test that applies to multiple portions of the computer program.

38. The method of claim 37, wherein the processing further includes:

accessing the activation information to determine one or more testing specifications that are indicated as active.

39. The method of claim 38, wherein the set of one or more testing specifications is a hierarchical set that relates at least some of the testing specifications to each other according to a hierarchy.

40. The method of claim 39, wherein accessing the activation information to determine one or more testing specifications that are indicated as active includes identifying active testing specifications along a path up the hierarchy between the first testing specification and a root of the hierarchy.

41. The method of claim 38, wherein the processing further includes:

while traversing the representation, performing any local tests specified in any active testing specifications based at least in part on stored values of attributes of one or more recognized portions of the computer program.

42. The method of claim 38, wherein the processing further includes:

while traversing the representation, performing any aggregation functions specified in any active testing specifications based at least in part on stored values of attributes of one or more recognized portions of the computer program.

43. The method of claim 42, wherein the processing further includes:

after traversing the representation, performing any global tests specified in any active testing specifications based at least in part on stored results of any aggregation functions performed while traversing the representation.

44. The method of claim 27, wherein the representation of the computer program is a graph based-representation and the elements include:

nodes that represent at least some of the recognizable portions, and links that represent relationships between the recognizable portions represented by the nodes.

45. The method of claim 44, wherein the graph-based representation includes a dataflow graph that includes:

processing components represented by the nodes, and flows of data between processing components represented by the links.

46. The method of claim 45, wherein at least some of the recognizable portions include a flow of data between processing components.

47. The method of claim 45, wherein at least some of the recognizable portions include an input or output port of a processing component.

48. The method of claim 45, wherein at least some of the recognizable portions include a parameter associated with a processing component or a flow of data between processing components.

49. The method of claim 44, wherein the graph-based representation includes a control flow graph that includes:

basic blocks of code represented by the nodes, and changes of control between basic blocks represented by the links.

50. The method of claim 27, the testing further including receiving user input for specifying at least one testing specification of the set of one or more testing specifications.

51. The method of claim 50, wherein at least some of the attribute information is predetermined before at least some of the user input is received.

52. Software stored on a non-transitory medium, for configurable testing of a computer program, the software including instructions for causing a computing system to:

store a set of one or more testing specifications, and attribute information defining one or more attributes of a recognizable portion of the computer program; and

process the computer program according to at least a first testing specification associated with the computer program, the first testing specification including at least a first test that applies to multiple portions of the computer program, the processing including;

traversing a representation of the computer program that includes elements that represent recognizable portions of the computer program,

while traversing the representation, recognizing recognizable portions of the computer program,

while traversing the representation, for at least a first recognized portion of the computer program, storing one or more values of one or more attributes of the first recognized portion of the computer program, wherein each attribute whose value is stored while traversing the representation is defined by the attribute information as being an attribute of one or more of the recognizable portions of the computer program,

performing any aggregation functions specified in the first testing specification based at least in part on stored values of attributes of one or more recognized portions of the computer program, and

performing the first test specified in the first testing specification based at least in part on stored results of any aggregation functions performed while traversing the representation.

53. The software of claim 52, wherein the processing further includes:

performing at least one test specified in at least the first testing specification based at least in part on stored values of attributes of one or more recognized portions of the computer program.

54. The software of claim 53, wherein the first testing specification includes a test that tests at least one characteristic of the computer program that does not represent an error that would prevent the computer program from executing.

55. The software of claim 53, wherein the first testing specification includes a test that tests at least one characteristic of the computer program associated with code reuse efficiency.

56. The software of claim 53, wherein the first testing specification includes a test that tests at least one characteristic of the computer program associated with memory use constraints.

57. The software of claim 56, wherein the characteristic of the computer program associated with memory use constraints includes a maximum amount of main memory that a particular recognized portion of the computer program is allowed to use when the computer program executes.

58. The software of claim 53, wherein the first testing specification includes a test that tests a number of times at least one particular operation is used in the computer program.

59. The software of claim 53, wherein the first testing specification includes a test that references a value of a name attribute of a recognizable portion of the computer program to determine whether to continue applying the test.

60. The software of claim 53, the processing further including presenting in a user interface a result of at least one test, the result being visually associated with a displayed portion of the computer program on which the test was performed.

61. The software of claim 52, wherein the storage medium stores activation information indicating which of the testing specifications are active.

62. The software of claim 61, wherein at least one of the testing specifications includes at least one local test that applies to a particular portion of the computer program, and at least one of the testing specifications includes at least one global test that applies to multiple portions of the computer program.

63. The software of claim 62, wherein the processing further includes:

accessing the activation information to determine one or more testing specifications that are indicated as active.

64. The software of claim 63, wherein the set of one or more testing specifications is a hierarchical set that relates at least some of the testing specifications to each other according to a hierarchy.

65. The software of claim 64, wherein accessing the activation information to determine one or more testing specifications that are indicated as active includes identifying active testing specifications along a path up the hierarchy between the first testing specification and a root of the hierarchy.

66. The software of claim 63, wherein the processing further includes:

while traversing the representation, performing any local tests specified in any active testing specifications based at least in part on stored values of attributes of one or more recognized portions of the computer program.

67. The software of claim 63, wherein the processing further includes:

while traversing the representation, performing any aggregation functions specified in any active testing specifications based at least in part on stored values of attributes of one or more recognized portions of the computer program.

68. The software of claim 67, wherein the processing further includes:

after traversing the representation, performing any global tests specified in any active testing specifications based at least in part on stored results of any aggregation functions performed while traversing the representation.

69. The software of claim 52, wherein the representation of the computer program is a graph based-representation and the elements include:

nodes that represent at least some of the recognizable portions, and links that represent relationships between the recognizable portions represented by the nodes.

70. The software of claim 69, wherein the graph-based representation includes a dataflow graph that includes:

processing components represented by the nodes, and flows of data between processing components represented by the links.

71. The software of claim 70, wherein at least some of the recognizable portions include a flow of data between processing components.

72. The software of claim 70, wherein at least some of the recognizable portions include an input or output port of a processing component.

73. The software of claim 70, wherein at least some of the recognizable portions include a parameter associated with a processing component or a flow of data between processing components.

74. The software of claim 69, wherein the graph-based representation includes a control flow graph that includes:

basic blocks of code represented by the nodes, and changes of control between basic blocks represented by the links.

75. The software of claim 52, the testing further including receiving user input for specifying at least one testing specification of the set of one or more testing specifications.

76. The software of claim 75, wherein at least some of the attribute information is predetermined before at least some of the user input is received.