Test executive system and method including process models for improved configurability

• US 6,577,981 B1
• Filed: 02/26/1999
• Issued: 06/10/2003
• Est. Priority Date: 08/21/1998
• Status: Expired due to Term

• Current Assignee: National Instruments Corporation (Emerson Electric Company)
• Original Assignee: National Instruments Corporation (Emerson Electric Company)
• Inventors: Crank, Erik, Grey, James
• Primary Examiner(s): Hoff, Marc S.
• Assistant Examiner(s): Barbee, Manuel L.
• Application Number:
  US09/259,161
• Time in Patent Office: 1,565 Days
• Field of Search: 702/118, 702/119, 702/120, 702/121, 702/108, 702/117, 702/123, 702/183, 702/184, 714/733
• US Class Current: 702/119
• CPC Class Codes: G06F 11/263 Generation of test inputs, e.g. test vectors, patterns or sequences ; with adaptation of the tested hardware for testability with external testers
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases

Abstract

A test executive program which provides greatly improved configurability and modularity, thus simplifying the creation, modification and execution of test sequences. The test executive program includes process models for improved flexibility, modularity and configurability. Process models provide a modular and configurable entity for encapsulating operations and functionality associated with a class of test sequences. The process model thus encapsulates a “testing process” for a plurality of test sequences. The process model enables the user to write different test sequences without repeating standard testing operations in each sequence. The test executive program also includes step types for improved configurability. A step type is a modular, identifiable unit configured by the user which defines common properties and/or operations associated with a plurality of steps. Instances of the step type incorporate this common functionality and/or properties from the step type, and thus the user is not required to hard code that functionality and/or properties in each instance or step. The test executive program also provides distributed type storage and conflict resolution. When the user loads a file which includes a stored type, the method performs conflict resolution, including determining if the loaded type conflicts with one or more previously loaded/registered types. If the types do conflict, then the test executive or user selects a type, preferably using the version data associated with each type. The test executive system also performs automatic result collection to automatically collect the results of each step. The user can enable or disable result collection for a particular sequence or for the entire test station.

Claims

1. A method for creating a test system for testing one or more units under test (UUTs), the method comprising:

configuring a process model in response to user input, wherein the process model includes functionality which is common to a plurality of different test sequences, wherein the process model includes functionality for testing one or more UUTs, wherein the process model is user configurable, wherein said configuring the process model includes one or more of;

configuring steps or configuring code modules, wherein said configuring the process model includes selecting the process model;

creating a test sequence file for testing the unit under test; and

executing the process model and the test sequence file to test the unit under test.

2. The method of claim 1, wherein the process model is modularly separate from the test sequence file.

3. The method of claim 1, wherein the method operates in a system comprising a test executive engine, one or more operator interfaces, one or more test sequences, and test code;

wherein the process model is modularly separate from the test executive engine, the one or more operator interfaces, the one or more test sequences, and the test code.

4. The method of claim 1, wherein the process model includes one or more of pre-test operations and post-test operations.

5. The method of claim 1, wherein the process model includes a plurality of different entry points which are selectable by the user;

wherein said configuring the process model includes selecting an entry point within the process model in response to user input, wherein the process model begins execution at the selected entry point.

6. The method of claim 5, wherein the user selects a first entry point in the process model for testing a plurality of UUTs and wherein the user selects a second different entry point for a single pass test of a single UUT.

7. The method of claim 1, wherein the process model includes calls to callback functions, wherein the callback functions execute during execution of the process model;

wherein the calls are user definable in number and order of execution.

8. The method of claim 1, wherein the process model includes calls to callback functions, wherein the callback functions execute during execution of the process model;

wherein the callback functions comprise sequences.

9. The method of claim 8, wherein the test sequence file includes the callback functions;

wherein the callback functions are user configurable.

10. The method of claim 1, wherein the process model is configurable to include calls to callback functions, wherein the callback functions execute during execution of the process model;

wherein said configuring the process model includes creating at least one new call to a new callback function in the process model, wherein the at least one new call is comprised in the process model;

wherein said executing the process model and the test sequence file includes executing the new call in the process model to invoke the new callback function.

11. The method of claim 1, wherein the process model includes an existing call to a callback function, wherein the callback function executes during execution of the process model;

wherein said configuring the process model includes creating a new callback function which is called by said existing call in the process model;

wherein said executing the process model and the test sequence file includes executing the existing call in the process model to invoke the new callback function.

12. The method of claim 11,

wherein said creating the new callback function comprises including the new callback function in the test sequence file.

13. The method of claim 11, wherein the new callback function overrides one of:

a default callback function that exists in the process model or an empty callback function that exists in the process model.

14. The method of claim 1,

wherein said configuring the process model includes one or more of:

adding new entry points, changing the order of calls to callback functions, adding/deleting calls to callback functions, adding new callback functions in response to new calls to callback functions, overriding callback functions, or modifying existing callback functions.

15. The method of claim 1,

wherein said configuring the process model includes one or more of:

configuring variables, configuring parameters, configuring data types, or configuring step types.

16. The method of claim 1, wherein the process model comprises a sequence file, wherein the process model is user editable as a sequence file.

17. The method of claim 16, wherein the process model comprises a plurality of steps;

wherein the steps call code modules in one of a plurality of different languages.

18. The method of claim 1, wherein the process model includes one or more variables, parameters, data types, step types, and code modules.

19. The method of claim 1, wherein the process model is independent of an operator interface program;

wherein the process model is independent of language and construction of the operator interface program.

20. The method of claim 19, wherein said configuring the process model includes defining one or more entry points in the process model in response to user input;

wherein the one or more entry points appear in the operator interface program.

21. The method of claim 20, further comprising:

the operator interface program querying a test executive engine to determine the one or more entry points defined in the process model;

the operator interface program displaying the one or more entry points in response to said querying.

22. The method of claim 19,

wherein the operator interface program presents an operator interface;

wherein changes to the process model which affect the appearance of the operator interface do not require changes to the operator interface.

23. The method of claim 1, wherein the method of creating the test program further includes configuring the process model in response to user input to provide one or more of:

customized report generation, database logging, UUT identification, and UUT status notification.

24. The method of claim 1, wherein said executing the process model and the test sequence file comprises executing the process model, wherein during execution of the process model the process model operates to call the test sequence file;

wherein the test sequence file executes to test the unit under test.

25. A memory medium comprising program instructions for creating a test system for testing one or more units under test (UUTs), wherein the program instructions are executable to implement:

selecting a process model;

configuring the process model in response to user input, wherein the process model includes functionality which is common to a plurality of different test sequences, wherein the process model includes functionality for testing one or more UUTs, wherein the process model is user configurable, wherein said configuring the process model includes one or more of;

configuring steps or configuring code modules;

creating a test sequence file for testing the unit under test; and

executing the process model and the test sequence file to test the unit under test.

26. The memory medium of claim 25, wherein the process model is modularly separate from the test sequence file.

27. The memory medium of claim 25, wherein the memory medium operates in a system comprising a test executive engine, one or more operator interfaces, one or more test sequences, and test code;

wherein the process model is modularly separate from the test executive engine, the one or more operator interfaces, the one or more test sequences, and the test code.

28. The memory medium of claim 25, wherein the process model includes one or more of pre-test operations and post-test operations.

29. The memory medium of claim 25, wherein the process model includes a plurality of different entry points which are selectable by the user;

wherein said configuring the process model includes selecting an entry point within the process model in response to user input, wherein the process model begins execution at the selected entry point.

30. The memory medium of claim 25, wherein the process model includes calls to callback functions, wherein the callback functions execute during execution of the process model;

wherein the calls are user definable in number and order of execution.

31. The memory medium of claim 25, wherein the process model includes calls to callback functions, wherein the callback functions execute during execution of the process model;

wherein the callback functions comprise user-configurable sequences.

32. The memory medium of claim 25, wherein the process model is configurable to include calls to callback functions, wherein the callback functions execute during execution of the process model;

wherein said configuring the process model includes creating at least one new call to a new callback function in the process model, wherein the at least one new call is comprised in the process model;

wherein said executing the process model and the test sequence file includes executing the new call in the process model to invoke the new callback function.

33. The memory medium of claim 25, wherein the process model includes an existing call to a callback function, wherein the callback function executes during execution of the process model;

wherein said configuring the process model includes creating a new callback function which is called by said existing call in the process model;

wherein said executing the process model and the test sequence file includes executing the existing call in the process model to invoke the new callback function.

34. The memory medium of claim 33,

wherein said creating the new callback function comprises including the new callback function in the test sequence file.

35. The memory medium of claim 33, wherein the new callback function overrides one of:

a default callback function that exists in the process model or an empty callback function that exists in the process model.

36. The memory medium of claim 25,

wherein said configuring the process model includes one or more of:

adding new entry points, changing the order of calls to callback functions, adding/deleting calls to callback functions, adding new callback functions in response to new calls to callback functions, overriding callback functions, or modifying existing callback functions.

37. The memory medium of claim 25,

wherein said configuring the process model includes one or more of:

configuring variables, configuring parameters, configuring data types, or configuring step types.

38. The memory medium of claim 25, wherein the process model comprises a sequence file, wherein the process model is user editable as a sequence file.

39. The memory medium of claim 38, wherein the process model comprises a plurality of steps;

wherein the steps call code modules in one of a plurality of different languages.

40. The memory medium of claim 25, wherein the process model includes one or more variables, parameters, data types, step types, and code modules.

41. The memory medium of claim 25, wherein the process model is independent of an operator interface program;

wherein the process model is independent of language and construction of the operator interface program.

42. The memory medium of claim 41, wherein said configuring the process model includes defining one or more entry points in the process model in response to user input;

wherein the one or more entry points appear in the operator interface program.

43. The memory medium of claim 42, further comprising:

one or the operator interface program querying a test executive engine to determine the one or more entry points defined in the process model;

the operator interface program displaying the one or more entry points in response to said querying.

44. The memory medium of claim 41,

wherein the operator interface program presents an operator interface;

wherein changes to the process model which affect the appearance of the operator interface do not require changes to the operator interface.

45. The memory medium of claim 25, wherein said configuring the process model is operable to provide one or more of:

customized report generation, database logging, UUT identification, and UUT status notification.

46. The memory medium of claim 25, wherein said executing the process model and the test sequence file comprises executing the process model, wherein during execution of the process model the process model operates to call the test sequence file;

wherein the test sequence file executes to test the unit under test.

47. A memory medium comprising program instructions for creating a test system for testing one or more units under test (UUTs), wherein the program instructions comprise:

a process model which includes functionality which is common to a plurality of different test sequences, wherein the process model includes functionality for testing one or more UUTs, wherein the process model is user configurable, wherein the process model includes one or more of configured steps or configured code modules;

a test sequence file for testing the unit under test;

wherein the process model and the test sequence file are executable to test the unit under test;

wherein the process model includes a plurality of different entry points which are selectable by the user;

wherein the process model begins execution at a user-selected entry point.

48. The memory medium of claim 47, wherein the process model is modularly separate from the test sequence file.

49. The memory medium of claim 47, wherein the memory medium operates in a system comprising a test executive engine, one or more operator interfaces, one or more test sequences, and test code;

wherein the process model is modularly separate from the test executive engine, the one or more operator interfaces, the one or more test sequences, and the test code.

50. The memory medium of claim 47, wherein the process model includes one or more of pre-test operations and post-test operations.

51. The memory medium of claim 47, wherein the process model includes calls to callback functions, wherein the callback functions execute during execution of the process model;

wherein the calls are user definable in number and order of execution;

wherein the callback functions are user configurable.

52. The memory medium of claim 47,

wherein said configuring the process model includes one or more of:

adding new entry points, changing the order of calls to callback functions, adding/deleting calls to callback functions, adding new callback functions in response to new calls to callback functions, overriding callback functions, or modifying existing callback functions.

53. A method for creating a test system for testing one or more units under test (UUTs), the method comprising:

selecting a process model from a plurality of different process models, wherein each of the plurality of different process models includes functionality which is common to a plurality of different test sequences;

configuring the process model in response to user input, wherein the process model includes functionality which is common to a plurality of different test sequences, wherein the process model includes functionality for testing one or more UUTs, wherein the process model is user configurable;

creating a test sequence file for testing the unit under test, wherein the process model is modularly separate from the test sequence file; and

executing the process model and the test sequence file to test the unit under test.

54. The method of claim 53, wherein the process model includes one or more of pre-test operations and post-test operations.

55. The method of claim 53,

wherein said configuring the process model includes one or more of: configuring steps or configuring code modules.

56. The method of claim 53,

wherein said configuring the process model includes one or more of:

configuring variables, configuring parameters, configuring data types, or configuring step types.

57. The method of claim 53, wherein the process model comprises a sequence file.

58. The method of claim 53, wherein the test sequence file comprises a test sequence, wherein the test sequence comprises a plurality of steps.

59. The method of claim 53, wherein said configuring the process model includes configuring the process model to provide one or more of:

customized report generation, database logging, UUT identification, and UUT status notification.

60. The method of claim 53, wherein the process model includes a plurality of steps.

61. A method for creating a test system for testing one or more units under test (UUTs), the method comprising:

configuring a process model in response to user input, wherein the process model includes functionality which is common to a plurality of different test sequences, wherein the process model includes functionality for testing one or more UUTs, wherein the process model is user configurable;

creating a test sequence file for testing the unit under test;

executing the process model and the test sequence file to test the unit under test;

wherein the process model includes a plurality of different entry points which are selectable by the user; and

wherein said configuring the process model includes selecting an entry point within the process model in response to user input, wherein the process model begins execution at the selected entry point.

62. The method of claim 61, wherein the user selects a first entry point in the process model for testing a plurality of UUTs and wherein the user selects a second different entry point for a single pass test of a single UUT.

63. A method for creating a test system for testing one or more units under test (UUTs), the method comprising:

configuring a process model in response to user input, wherein the process model includes functionality which is common to a plurality of different test sequences, wherein the process model includes functionality for testing one or more UUTs, wherein the process model is user configurable, wherein said configuring the process model includes defining one or more entry points in the process model in response to user input, wherein the one or more entry points appear in an operator interface program;

creating a test sequence file for testing the unit under test; and

executing the process model and the test sequence file to test the unit under test.

64. The method of claim 63, further comprising:

the operator interface program querying a test executive engine to determine the one or more entry points defined in the process model; and

the operator interface program displaying the one or more entry points in response to said querying.

65. A method for creating a test system for testing one or more units under test (UUTs), the method comprising:

configuring a process model in response to user input, wherein the process model includes functionality which is common to a plurality of different test sequences, wherein the process model includes functionality for testing one or more UUTs, wherein the process model is user configurable, wherein said configuring the process model includes selecting the process model;

creating a test sequence file for testing the unit under test; and

executing the process model and the test sequence file to test the unit under test.

66. A memory medium comprising program instructions for creating a test system for testing one or more units under test (UUTs), wherein the program instructions are executable to implement:

selecting a process model;

configuring the process model in response to user input, wherein the process model includes functionality which is common to a plurality of different test sequences, wherein the process model includes functionality for testing one or more UUTs, wherein the process model is user configurable;

creating a test sequence file for testing the unit under test, wherein the process model is modularly separate from the test sequence file; and

executing the process model and the test sequence file to test the unit under test.

67. The memory medium of claim 66, wherein the process model includes one or more of pre-test operations and post-test operations.

68. The memory medium of claim 66,

wherein said configuring the process model includes one or more of: configuring steps or configuring code modules.

69. The memory medium of claim 66,

wherein said configuring the process model includes one or more of:

configuring variables, configuring parameters, configuring data types, or configuring step types.

70. The memory medium of claim 66, wherein the process model comprises a sequence file.

71. The memory medium of claim 66, wherein said configuring the process model includes configuring the process model to provide one or more of:

customized report generation, database logging, UUT identification, and UUT status notification.

72. The memory medium of claim 66, wherein the process model includes a plurality of steps.

73. A memory medium comprising program instructions for creating a test system for testing one or more units under test (UUTs), wherein the program instructions comprise:

a process model which includes functionality which is common to a plurality of different test sequences, wherein the process model includes functionality for testing one or more UUTs, wherein the process model comprises a plurality of test steps, wherein the process model is selected from a plurality of possible process models;

a test sequence for testing the unit under test; and

wherein the process model and the test sequence are executable to test the unit under test.

74. A method for creating a test system for testing one or more units under test (UUTs), the method comprising:

configuring a first test sequence in response to user input, wherein the first test sequence includes first functionality which is common to a plurality of different test sequences, wherein the first functionality is useable for testing the one or more UUTs, wherein said configuring the first test sequence includes selecting the first test sequence from a plurality of possible first test sequences;

creating a second test sequence for testing the one or more UUTs;

executing the first test sequence and the second test sequence to test the one or more UUTs.

75. The method of claim 74, wherein the first test sequence encapsulates functionality associated with the plurality of different test sequences.

76. The method of claim 74, wherein the first test sequence includes a first one or more common operations performed before each of the plurality of different test sequences.

77. The method of claim 74, wherein the first test sequence includes a first one or more common operations performed after each of the plurality of different test sequences.

78. The method of claim 74,

wherein said creating the second test sequence comprises creating the second test sequence to include second functionality, wherein the second functionality does not include the first functionality of the first test sequence;

wherein the second test sequence is designed to execute with the first test sequence.

79. The method of claim 74, wherein the first test sequence operates as a process model for each of the plurality of different test sequences.

80. The method of claim 74, further comprising:

creating a third test sequence for testing a different one or more UUTs;

executing the first test sequence and the third test sequence to test the different one or more UUTs.

81. The method of claim 74,

wherein the first test sequence is modularly separate from the second test sequence.

82. The method of claim 74,

wherein the first test sequence is modularly separate from each of the plurality of different test sequences.

83. The method of claim 74, wherein the method operates in a system comprising a test executive engine, one or more operator interfaces, one or more test sequences, and test code;

wherein the first test sequence is modularly separate from the test executive engine, the one or more operator interfaces, the one or more test sequences, and the test code.

84. The method of claim 74, wherein the first test sequence includes a plurality of different entry points which are selectable by the user;

wherein said configuring the first test sequence includes selecting an entry point within the first test sequence in response to user input, wherein the first test sequence begins execution at the selected entry point.

85. The method of claim 84, wherein the user selects a first entry point in the first test sequence for testing a plurality of UUTs and wherein the user selects a second different entry point in the first test sequence for a single pass test of a single UUT.

86. The method of claim 74, wherein the method of creating the test program further includes configuring the first test sequence in response to user input to provide one or more of:

customized report generation, database logging, UUT identification, and UUT status notification.

87. The method of claim 74, wherein said executing the first test sequence and the second test sequence comprises the first test sequence calling the second test sequence;

wherein the second test sequence executes to test the unit under test.

88. The method of claim 74, wherein said configuring the first test sequence includes selecting the first test sequence from a plurality of possible process models.

89. A memory medium comprising program instructions for creating a test system for testing one or more units under test (UUTs), wherein the program instructions are executable to implement:

configuring a first test sequence in response to user input, wherein the first test sequence includes first functionality which is common to a plurality of different test sequences, wherein the first functionality is useable for testing the one or more UUTs, wherein said configuring the first test sequence includes selecting the first test sequence from a plurality of possible first test sequences;

creating a second test sequence for testing the one or more UUTs;

executing the first test sequence and the second test sequence to test the one or more UUTs.

90. A method for creating a test system for testing one or more units under test (UUTs), the method comprising:

configuring a first test sequence in response to user input, wherein the first test sequence includes first functionality which is common to a plurality of different test sequences, wherein the first functionality is useable for testing the one or more UUTs;

wherein the first test sequence includes a plurality of different entry points which are selectable by the user;

wherein said configuring the first test sequence includes selecting an entry point within the first test sequence in response to user input, wherein the first test sequence begins execution at the selected entry point;

creating a second test sequence for testing the one or more UUTs;

executing the first test sequence and the second test sequence to test the one or more UUTs.

91. The method of claim 90, wherein the user selects a first entry point in the first test sequence for testing a plurality of UUTs and wherein the user selects a second different entry point in the first test sequence for a single pass test of a single UUT.

92. The method of claim 90, wherein the first test sequence encapsulates functionality associated with the plurality of different test sequences.

93. The method of claim 90, wherein the first test sequence includes a first one or more common operations performed before each of the plurality of different test sequences.

94. The method of claim 90, wherein the first test sequence includes a first one or more common operations performed after each of the plurality of different test sequences.

95. The method of claim 90,

wherein said creating the second test sequence comprises creating the second test sequence to include second functionality, wherein the second functionality does not include the first functionality of the first test sequence;

wherein the second test sequence is designed to execute with the first test sequence.

96. The method of claim 90, wherein the first test sequence operates as a process model for each of the plurality of different test sequences.

97. The method of claim 90, further comprising:

creating a third test sequence for testing a different one or more UUTs;

executing the first test sequence and the third test sequence to test the different one or more UUTs.

98. The method of claim 90,

wherein the first test sequence is modularly separate from the second test sequence.

99. The method of claim 90,

wherein the first test sequence is modularly separate from each of the plurality of different test sequences.

100. The method of claim 90, wherein the method operates in a system comprising a test executive engine, one or more operator interfaces, one or more test sequences, and test code;

wherein the first test sequence is modularly separate from the test executive engine, the one or more operator interfaces, the one or more test sequences, and the test code.

101. The method of claim 90, wherein the method of creating the test program further includes configuring the first test sequence in response to user input to provide one or more of:

customized report generation, database logging, UUT identification, and UUT status notification.

102. The method of claim 90, wherein said executing the first test sequence and the second test sequence comprises the first test sequence calling the second test sequence;

wherein the second test sequence executes to test the unit under test.

103. The method of claim 90, wherein said configuring the first test sequence includes selecting the first test sequence from a plurality of possible first test sequences.