Test engine and method for verifying conformance for server applications

• US 6,260,065 B1
• Filed: 01/13/1999
• Issued: 07/10/2001
• Est. Priority Date: 01/13/1999
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Leiba, Barry, Romano, Anthony, Von Känel, Jürg
• Primary Examiner(s): VU, VIET DUY
• Application Number:
  US09/229,852
• Time in Patent Office: 909 Days
• Field of Search: 709/223, 709/224, 709/230, 709/237, 709/220, 709/221, 709/313, 714/25, 714/37, 714/39, 714/47
• US Class Current: 709/224
• CPC Class Codes: G06F 11/2294 by remote test
• CPC Class Codes: H04L 41/0869 Validating the configuration within one network element
• CPC Class Codes: H04L 43/50 Testing arrangements

Abstract

A method to verify standards conformance for a server application, in accordance with the present invention includes the steps of providing configuration information and a sequence of test commands to the server application with response requirements for expected responses associated with each of the commands, issuing at least one of the commands from the sequence to the server application, receiving at least one response from the server application and comparing the at least one response with expected responses for performing compliance testing based on the response requirements. Systems for conformance testing are also included.

Claims

1. A method to verify standards conformance for a server application, comprising the steps of:

providing configuration information and a sequence of test commands to the server application with response requirements for expected responses associated with each of the commands;

issuing at least one of the commands from the sequence to the server application;

receiving at least one response from the server application;

permitting non-deterministic ordering of responses when a plurality of responses are received; and

comparing the at least one response with expected responses for performing compliance testing based on the response requirements.

2. The method as recited in claim 1, wherein the at least one response includes a plurality of responses, each response including at least one component and the method further comprising the step of:

permitting ordering of components within each response.

3. The method as recited in claim 1, further comprising the step of performing a stress test with stopping criteria to test durability of the server application.

4. The method as recited in claim 1, further comprising the step of specifying responses associated with the at least one command to be saved and used for subsequent commands and comparisons to the expected responses.

5. The method as recited in claim 1, further comprising the step of grouping commands such that a failure of compliance of any command in the group results in exiting the group and continuing in subsequent commands outside the group.

6. The method as recited in claim 5, wherein the grouping is hierarchical.

7. The method as recited claim 1, further comprising the step of generating a test sequence from a state machine.

8. The method as recited in claim 7, wherein the test sequence is based on probabilistic transitions.

9. The method as recited in claim 1, wherein the expected responses are marked as one of a mandatory response, an optional response, a forbidden response, several mandatory responses and several optional responses for permitting testing of types of responses, types of test data and implementations of the server application.

10. The method as recited in claim 1, wherein the expected responses include pattern symbols.

11. The method as recited in claim 1, further comprising the step of performing a scalability test.

12. A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for verifying standards conformance for a server application, the method steps comprising:

providing configuration information and a sequence of test commands from the server application with response requirements for expected responses associated with each of the commands;

issuing at least one of the commands from the sequence to the server application;

receiving at least one response from the server application;

permitting non-detenninistic ordering of responses when a plurality of responses are received; and

comparing the at least one response with expected responses for performing compliance testing based on the response requirements.

13. The program storage device as recited in claim 12, wherein the at least one response includes a plurality of responses, each response including at least one component and the method further comprising the step of:

permitting ordering of components within each response.

14. The program storage device as recited in claim 12, further comprising the step of performing a stress test with stopping criteria to test durability of the server application.

15. The program storage device as recited in claim 12, further comprising the step of specifying responses associated with the at least one command to be saved and used for subsequent commands and comparisons to the expected responses.

16. The program storage device as recited in claim 12, further comprising the step of grouping commands such that a failure of compliance of any command in the group results in exiting the group and continuing in subsequent commands outside the group.

17. The program storage device as recited in claim 16, wherein the grouping is hierarchical.

18. The program storage device as recited claim 12, further comprising the step of generating a test sequence from a state machine.

19. The program storage device as recited in claim 18, wherein the test sequence is based on probabilistic transitions.

20. The program storage device as recited in claim 12, wherein the expected responses are marked as one of a mandatory response, an optional response, a forbidden response, several mandatory responses and several optional responses for permitting testing of types of responses, types of test data and implementations of the server application.

21. The program storage device as recited in claim 12, wherein the expected responses include pattern symbols.

22. The program storage device as recited in claim 12, further comprising the step of performing a scalability test.

23. A test engine for verifying standard conformance for server applications comprising:

a test processor and a protocol component for communicating with a server having the server application;

the test processor including;

a parser which parses and checks test instructions input to the test processor, the parser converting parsed test instructions into objects; and

an execution engine which executes the objects output from the parser and analyzes the results of the execution of the instructions such that responses of the server application upon execution of the instructions are compared to expected responses included in the test instructions to determine conformance;

the execution engine including a response analyzer which analyzes the responses and provides for non-deterministically ordering of the responses.

24. The system as recited in claim 23, wherein the response analyzer provides ordering of components within the responses.

25. The system as recited in claim 23, wherein the response analyzer provides a mechanism for testing in a multi-way coordination among client sessions.

26. The system as recited in claim 23, wherein the response analyzer includes a test generator component for generating multiple request sessions to the server application according to an input specification.

27. The system as recited in claim 26, wherein the test generator provides a stress test with stopping criteria for testing durability of the server application.

28. The system as recited in claim 26, wherein the test generator includes a test sequence derived from a state machine.

29. The system as recited in claim 28, wherein the test sequence is derived from the state machine based on probabilistic transitions.

30. The system as recited in claim 23, wherein the response analyzer includes means for specifying if a response from a command is saved and used in subsequent commands.

31. The system as recited in claim 23, wherein the response analyzer includes a grouping mechanism for grouping commands wherein a failure of any command in the group results in exiting the group and continuing in subsequent commands outside the group.

32. The system as recited in claim 31, wherein the grouping mechanism is hierarchical.

33. The system of claim 23, wherein the expected responses are marked as one of a mandatory response, an optional response, a forbidden response, several mandatory responses and several optional responses for permitting testing of types of responses, types of test data and implementations of the server application.

34. The system of claim 23, wherein the expected responses are marked as pattern symbols.

35. A method to verify standards conformance for a server application, comprising the steps of:

providing configuration information and a sequence of test commands to the server application with response requirements for expected responses associated with each of the commands;

issuing at least one of the commands from the sequence to the server application;

receiving at least one response from the server application;

comparing the at least one response with expected responses for performing compliance testing based on the response requirements; and

grouping commands such that a failure of compliance of any command in the group results in exiting the group and continuing in subsequent commands outside the group.

36. A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform the method steps as recited in claim 35.