System and method for testing software using data-driven test variations
First Claim
1. A system for testing a software application comprising:
- a processor;
a testing runtime executing on the processor, the testing runtime directing the software application to perform operations based on a test definition and storing results related to performance of operations of the software application;
at least one variation data provider identified by the test definition and executing on the processor, each variation data provider generating a plurality of variations for a predetermined scope from a template and a plurality of values associated with a parameter, the template being a set of operations including the parameter, wherein each variation uses a different one of the plurality of values for the parameter, and wherein the variation data provider supplies the plurality of variations to the testing runtime and the scope of each of the plurality of variations is identified, the scope being used by the system to determine a sequence to run the variations in to ensure that all possible combinations are automatically evaluated.
2 Assignments
0 Petitions
Accused Products
Abstract
The integrated test framework of the present invention includes a test framework runtime that executes test scripts and that also allows a script to identify a variation tool called a “data provider” that is responsible for providing data to the test framework runtime and controlling the iteration of the runtime through the variations. The script also identifies the type of data set from which the data provider should obtain the data from which the variations are derived and the location of the data set. Multiple variation data providers may be used in conjunction with an adapter layer that coordinates the iteration of each variation data provider and creates an aggregated variation over multiple varying parameters of different scope.
-
Citations
20 Claims
-
1. A system for testing a software application comprising:
-
a processor; a testing runtime executing on the processor, the testing runtime directing the software application to perform operations based on a test definition and storing results related to performance of operations of the software application; at least one variation data provider identified by the test definition and executing on the processor, each variation data provider generating a plurality of variations for a predetermined scope from a template and a plurality of values associated with a parameter, the template being a set of operations including the parameter, wherein each variation uses a different one of the plurality of values for the parameter, and wherein the variation data provider supplies the plurality of variations to the testing runtime and the scope of each of the plurality of variations is identified, the scope being used by the system to determine a sequence to run the variations in to ensure that all possible combinations are automatically evaluated. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8)
-
-
9. A computer storage medium having computer-executable instructions for performing a method comprising:
-
a) initializing software on a computer system; b) initializing a variation data provider; c) requesting from the variation data provider, a variation comprising at least one operation to be performed by the software, wherein the operation belongs to a predetermined scope of operation, the scope being used in determining the sequence of variations; d) receiving a variation from the variation data provider, the variation including an initial configuration for the software and a set of one or more operations; e) configuring the software to the initial configuration; f) directing the software to perform the set of one or more operations; g) recording information describing the performance of the software of the set of one or more operations; h) repeating c) through g) wherein a different variation is received from the variation data provider each time until the variation data provider returns a message indicating there are no further variations. - View Dependent Claims (10, 11, 12, 13, 14)
-
-
15. A computer storage medium having computer-executable instructions for performing a method comprising:
-
a) initializing software on a computer system; b) initializing a first variation data provider and a second variation data provider and an adapter layer; c) identifying the scope of each variation data provider to the adapter layer; d) requesting from the adapter layer, a variation comprising at least one operation to be performed by the software; e) receiving a variation from the adapter layer, the variation including an initial configuration for the software and a set of one or more operations, wherein a first value in the set of one or more operations is provided by the first variation data provider and a second value in the set of one or more operations is provided by the second variation data provider; f) configuring the software to the initial configuration; g) directing the software to perform the set of one or more operations; h) recording information describing the performance of the software of the set of one or more operations; i) repeating d) through h) wherein a different variation is received from the adapter layer each time until the variation data provider returns a message indicating there are no further variations. - View Dependent Claims (16, 17)
-
-
18. A computer storage medium having computer-executable instructions for performing a method comprising:
-
receiving a request for a variation from a testing runtime executing a test script directing the testing of a software application; delivering to the testing runtime, in response to the request, a revert command, directing the testing runtime to configure the software application to the initial configuration; storing a template set of operations, the template set of operations including one or more operations to be performed by the software application, at least one operation having a parameter requiring a value; generating one or more variations based on the template set of operations by obtaining a next value from a variation data set and using the next value as the parameter, each variation relates to a predetermined scope wherein at least two or more scopes are a part of a hierarchical organization, wherein the predetermined scope comprises one of a Global scope, a Suite scope, a Class scope, and a Method scope; delivering to the testing runtime, in response to the request, the variation to the testing runtime; and executing each variation, wherein for each iteration of a variation related to an outer scope all inner scope variations are executed. - View Dependent Claims (19, 20)
-
Specification