Test manager for integrated test environments

• US 7,610,578 B1
• Filed: 08/24/2004
• Issued: 10/27/2009
• Est. Priority Date: 08/24/2004
• Status: Active Grant

• Current Assignee: The MathWorks, Inc.
• Original Assignee: The MathWorks, Inc.
• Inventors: Taillefer, Greg, Critz, David Karl
• Primary Examiner(s): DENG, ANNA CHEN
• Application Number:
  US10/925,413
• Time in Patent Office: 1,890 Days
• Field of Search: 717124-126
• US Class Current: 717/124
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 8/30 Creation or generation of source code

Abstract

Methods are provided to automatically generate, distribute and execute code locally and remotely. The test may therefore be performed using the automatically distributed code. Automated parameter mapping may be provided to couple parameters applicable to a test step with the test step. A test step in the test may therefore automatically include the coupled parameters. Data storage may be performed using a multidimensional parametric space. Each point in space may contain arbitrary nonscalar data which define an individual data workspace. These points can also be plotted against each other and overlayed to aid in visualization of multi-dimensional data.

Claims

1. In a computational device providing a test environment, a method for conducting a test for testing units under test, the method comprising:

enabling users to edit the test to include a sequence of one or more test steps for testing the units under test, each of the one or more test steps being of a type selected from a group consisting of a test step for testing a textual program, a test step for testing a graphical program, a test step for testing a function provided in a software tool and a test step for testing a hardware device;

enabling the users to edit one or more independent parameters, an independent parameter defining a dimension of a parametric space, the parametric space including a plurality of points;

generating code for the test; and

automatically executing the code using defined values of the one or more independent parameters to produce output data from a number of executions of the test, elements of the output data corresponding to the plurality of points of the parametric space of the one or more independent parameters, wherein automatically executing comprises;

using at least two independent parameters to execute the code, wherein at least one of the at least two used independent parameters has a plurality of defined values,

determining the number of executions of the test by multiplying the defined values in each of the at least two used independent parameters;

initializing a multidimensional parametric space for storing the output data generated from the number of executions of the test, dimensions of the multidimensional parametric space representing and corresponding to a product of the defined values in each of the at least two used independent parameters, and

adding the output data to the multidimensional parametric space.

2. The method of claim 1, wherein the enabling the users to edit the test further comprises:

determining test steps installed in the computational device; and

enabling the users to select the one or more test steps for the test from the test steps installed in the computational device.

3. The method of claim 1, wherein the enabling the users to edit the one or more independent parameters further comprises:

determining variables installed in the computational device; and

enabling the users to select one or more variables for the test from the variables installed in the computational device so that the users are able to specify variables and parameters of the test.

4. The method of claim 1, wherein the enabling the users to edit the test further comprises:

enabling the users to add new software tools to the test environment.

5. The method of claim 1, wherein the executing further comprises:

distributing at least a portion of the code for the test to one or more remote computational devices for a distributed execution of the code.

6. The method of claim 5, wherein the distributing comprises:

distributing all portions of the test at a time to one or more remote computational devices for a distributed execution of the test; and

enabling the all portions of the test to be run in parallel in the one or more remote computational devices.

7. The method of claim 5, wherein the distributing comprises:

(a) distributing a portion of the test at a time to one or more remote computational devices;

(b) enabling the distributed portion of the test to be run in parallel; and

(c) repeating (a) and (b) steps until all portions of the test are distributed and run in parallel.

8. The method of claim 1, wherein the executing comprises:

obtaining parameters for a current execution of the test.

9. The method of claim 8, wherein the executing comprises:

determining whether there is a remaining execution of the test; and

obtaining parameters for a next execution of the test.

10. The method of claim 1, further comprising:

storing the output data of the number of executions of the test in the multidimensional parametric space.

11. The method of claim 10, wherein the multidimensional parametric space stores metadata of the output data of the number of executions of the test.

12. The method of claim 10, wherein the output data of the number of executions of the test in the multidimensional parametric space is stored in memory in a binary or textual hierarchical data structure.

13. The method of claim 10, wherein each point in the multidimensional parametric space comprises a reference to one or more information storage and retrieval systems that contain data, the one or more information storage and retrieval systems being loaded one at a time as the data in the one or more information storage and retrieval systems is requested.

14. The method of claim 13, wherein the one or more information storage and retrieval systems comprises a multilevel hierarchical storage.

15. The method of claim 10, wherein the multidimensional parametric space is integrated in the testing environment.

16. The method of claim 10, further comprising:

providing a graphical display of the output data stored in the multidimensional parametric space.

17. The method of claim 16, wherein the graphical display is automatically generated based on data generated by the testing environment.

18. The method of claim 16, wherein the graphical display displays multi-dimensional data.

19. The method of claim 1, wherein the units under test comprise at least one of a graphical program, a textual program, a function provided in a software tool, and a hardware device.

20. A computer-implemented system for conducting a test for testing units under test, the system comprising:

a display; and

a processor configured to

provide a user interface on the display for users to edit the test to include a plurality of test steps for testing the units under test, the test including a sequence of at least a first test step and a second test step, the second step being of a different type than the first test step, the first test step and the second test step being of types selected from a group consisting of a test step for testing a textual program, a test step for testing a graphical program, a test step for testing a function provided in a software tool and a test step for testing a hardware device;

enable the user to edit one or more independent parameters, an independent parameter defining a dimension of a parametric space, the parametric space including a plurality of points;

generate code for the test; and

execute the code using defined values of the one or more independent parameters to generate output data from a number of executions of test, elements of the output data corresponding to the plurality of points of the parametric space of the one or more independent parameters, wherein execute the code comprising;

use at least two independent parameters to execute the code, wherein at least one of the at least two used independent parameters has a plurality of defined values,

determine the number of executions of the test by multiplying the defined values in each of the at least two used independent parameters;

initialize a multidimensional parametric space for storing the output data generated from the number of executions of the test, dimensions of the multidimensional parametric space representing and corresponding to a product of the defined values in each of the at least two used independent parameters, and

add the output data to the multidimensional parametric space.

21. The system of claim 20, wherein the user interface enables the users to select the plurality of test steps for the test so that the users are able to edit the sequence of the test steps in the test.

22. The system of claim 20, wherein the user interface enables the users to select one or more variables for the test so that the users are able to specify variables and parameters of the test.

23. The system of claim 20, wherein the processor is further configured to enable the users to add new software tools to the test environment.

24. The system of claim 20, wherein the code for the test is executed locally with the user interface.

25. The system of claim 20, wherein the code for the test is executed remotely from the user interface.

26. The system of claim 20, wherein the processor is further configured to store the output data of the test as a multidimensional dataset in the test environment.

27. The system of claim 26, wherein the multidimensional dataset comprises metadata of the output data of the test.

28. The system of claim 26, wherein the multidimensional dataset is stored in memory in a binary or textual hierarchical data structure.

29. The system of claim 26, wherein each point in the multidimensional dataset comprises a reference to one or more information storage and retrieval systems that contain data, the one or more information storage and retrieval systems being loaded one at a time as the data in the one or more information storage and retrieval systems is requested.

30. The system of claim 29, wherein the one or more information storage and retrieval systems comprises a multilevel hierarchical storage.

31. The system of claim 26, wherein a dimension of the multidimensional dataset is defined by a number of elements of an independent parameter contained by the test.

32. The system of claim 26, further comprising:

a viewer for displaying the output data stored in the multidimensional dataset.

33. The system of claim 20, wherein the units under test comprise at least one of a graphical program, a textual program, a function call provided in a software tool, and a hardware device.

34. A system for providing a test environment in a computational device, the system comprising:

storage; and

a processor configured to;

provide a first user interface for users to edit a test to include a sequence of one or more test steps for testing units under test, each of the one or more test steps being of a type selected from a group consisting of a test step for testing a textual program, a test step for testing a graphical program, a test step for testing a function provided in a software tool and a test step for testing a hardware device;

provide a second user interface for the users to edit one or more independent parameters, an independent parameter defining a dimension of a parametric space, the parametric space including a plurality of points;

generate code for the test;

automatically execute the code using defined values of the one or more independent parameters to produce output data from a number of executions of the test, elements of the output data corresponding to the plurality of points of the parametric space of the one or more independent parameters, wherein automatically execute comprises;

use at least two independent parameters to execute the code, wherein at least one of the at least two used independent parameters has a plurality of defined values,

determine the number of executions of the test by multiplying the defined values in each of the at least two used independent parameters;

initialize a multidimensional parametric space for storing the output data generated from the number of executions of the test, dimensions of the multidimensional parametric space representing and corresponding to a product of the defined values in each of the at least two used independent parameters,

add the output data to the multidimensional parametric space, and

store the output data as a multidimensional dataset in the storage, elements of the multidimensional dataset corresponding to the plurality of points of the parametric space in the at least two used independent parameters.

35. The system of claim 34, wherein the multidimensional dataset comprises metadata of the output data of the test.

36. The system of claim 34, wherein the multidimensional dataset is stored in memory in a binary or textual hierarchical data structure.

37. The system of claim 34, wherein each point in the multidimensional dataset comprises a reference to one or more information storage and retrieval systems that contain data, the one or more information storage and retrieval systems being loaded one at a time when the data in the one or more information storage and retrieval systems is requested.

38. The system of claim 37, wherein the one or more information storage and retrieval systems comprises multilevel hierarchical storage.

39. The system of claim 34, wherein the first user interface enables the users to select the plurality of test steps for the test from test steps installed in the computational device so that the users are able to edit a sequence of the test steps in the test.

40. The system of claim 34, wherein the second user interface enables the users to select one or more variables for the test from variables installed in the computational device so that the users are able to specify variables and parameters of the test.

41. The system of claim 34, further comprising:

an application program interface for enabling the users to add new software tools to the test environment.

42. The system of claim 34, further comprising:

a viewer for displaying the output data stored in the multidimensional dataset.

43. The system of claim 34, wherein the units under test comprise at least one of a graphical program, a textual program, a function call provided in a software tool, and a hardware device.

44. A computer-readable storage medium holding computational device executable instructions for conducting a test for testing units under test, the medium comprising:

instructions for enabling users to edit the test to include a sequence of one or more test steps for testing the units under test, each of the one or more test steps being of a type selected from a group consisting of a test step for testing a textual program, a test step for testing a graphical program, a test step for testing a function provided in a software tool and a test step for testing a hardware device;

instructions for enabling the users to edit one or more independent parameters, an independent parameter defining a dimension of a parametric space, the parametric space including a plurality of points;

instructions for generating code for the test; and

instructions for automatically executing the code using defined values of the one or more independent parameters to generate output data from a number of executions of the test, elements of the output data corresponding to the plurality of points of the parametric space of the one or more independent parameters, wherein the instructions for automatically executing comprises;

instructions for using at least two independent parameters to execute the code, wherein at least one of the at least two used independent parameters has a plurality of defined values,

instructions for determining the number of executions of the test by multiplying the defined values in each of the at least two used independent parameters;

instructions for initializing a multidimensional parametric space for storing the output data generated from the number of executions of the test, dimensions of the multidimensional parametric space representing and corresponding to a product of the defined values in each of the at least two used independents, and

instruction for adding the output data to the multidimensional parametric space.

45. The medium of claim 44, wherein the instructions for enabling the users to edit the test further comprises:

instructions for determining test steps installed in the computational device; and

instructions for enabling the users to select the one or more test steps for the test from the test steps installed in the computational device.

46. The medium of claim 44, wherein the instructions for enabling the users to edit the one or more independent parameters further comprises:

instructions for determining variables installed in the computational device; and

instructions for enabling the users to select one or more variables for the test from the variables installed in the computational device so that the users are able to specify variables and parameters of the test.

47. The medium of claim 44, wherein the instructions for enabling the users to edit the test further comprises:

instructions for determining software tools installed in the computational device; and

instructions for enabling the users to add new software tools to the test environment.

48. The medium of claim 44, wherein the instructions for executing further comprises:

instructions for distributing at least a portion of the code for the test to one or more remote computational devices for a distributed execution of the code.

49. The medium of claim 48, wherein the instructions for distributing comprises:

instructions for distributing all portions of the test at a time to one or more remote computational devices for a distributed execution of the test; and

instructions for enabling the all portions of the test to be run in parallel in the one or more remote computational devices.

50. The medium of claim 48, wherein the instructions for distributing comprises:

instructions for (a1) distributing a portion of the test at a time to one or more remote computational devices;

instructions for (b1) enabling the distributed portion of the test to be run in parallel; and

instructions for repeating (a1) and (b1) until all portions of the test are distributed and run in parallel.

51. The medium of claim 44, wherein the instructions for executing comprises:

instructions for obtaining parameters for a current execution of the test.

52. The medium of claim 51, wherein the instructions for executing comprises:

instructions for determining whether there is a remaining execution of the test; and

instructions for obtaining parameters for a next execution of the test.

53. The medium of claim 44, further comprising:

instructions for storing the output data of the number of executions of the test in the multidimensional parametric space.

54. The medium of claim 53, wherein the multidimensional parametric space stores metadata of the output data of the number of executions of the test.

55. The medium of claim 53, wherein the multidimensional parametric space is stored in memory in a binary or textual hierarchical data structure.

56. The medium of claim 53, wherein each point in the multidimensional parametric space comprises a reference to one or more information storage and retrieval systems that contain data, the one or more information storage and retrieval systems being loaded one at a time as the data in the one or more information storage and retrieval systems is requested.

57. The medium of claim 56, wherein the one or more information storage and retrieval systems comprises a multilevel hierarchical storage.

58. The medium of claim 53, wherein the multidimensional parametric space is integrated with the testing environment.

59. The medium of claim 53, further comprising:

instructions for providing a graphical display of the output data stored in the multidimensional parametric space.

60. The medium of claim 59, wherein the graphical display is automatically generated based on data generated by the testing environment.

61. The medium of claim 59, wherein the graphical display displays multi-dimensional data.

62. The medium of claim 44, wherein the units under test comprise at least one of a graphical program, a textual program, a function provided in a software tool, and a hardware device.