Integrated testing approach for publish/subscribe network systems

• US 7,523,198 B2
• Filed: 07/14/2006
• Issued: 04/21/2009
• Est. Priority Date: 01/25/2002
• Status: Expired due to Term

• Current Assignee: Architecture Technology Corporation
• Original Assignee: Architecture Technology Corporation
• Inventors: Wollnik, Matthias H., Wu, John, Burnett, Benjamin L.
• Primary Examiner(s): Tran, Philip B
• Application Number:
  US11/486,575
• Publication Number:
  US 20060259542A1
• Time in Patent Office: 1,012 Days
• Field of Search: 709/224, 709/223, 709201-202, 709/234, 714/25, 714/47, 714/48
• US Class Current: 709/224
• CPC Class Codes: H04L 43/0852 Delays
• CPC Class Codes: H04L 43/12 Network monitoring probes
• CPC Class Codes: H04L 67/125 involving control of end-device applications over a network
• CPC Class Codes: H04L 69/02 Protocol performance

Abstract

The invention is directed to techniques for monitoring and testing publish/subscribe network systems. For example, an enterprise network that has a number of geographically dispersed network devices and interconnected sub-networks may be configured to operate as a publish/subscribe network system in which each of the dispersed network devices sends and receives messages for specific groups of the network devices. The techniques described herein enable data object capture and analysis in the application layer of a publish/subscribe network system using shim adapters. The techniques also enable network packet capture and analysis in the network layer of a publish/subscribe network system. Furthermore, the techniques include integrated analysis of testing results from the application layer and the network layer to determine performance characteristics, such as latency, across both of the layers. In this way, the techniques provide more accurate measurements of performance characteristics in publish/subscribe network systems.

Claims

1. A method implemented at least in part by a network analyzer for monitoring and testing a publish/subscribe network system, comprising:

generating a test scenario for the publish/subscribe network system;

issuing commands to a plurality of agents positioned at different locations within the publish/subscribe network system to initiate the test scenario;

capturing data objects propagating between publish/subscribe clients and the publish/subscribe network system according to the test scenario with the plurality of agents;

communicating results of the test scenario including the captured data objects to an analyzer coupled to the publish/subscribe network system;

analyzing the results of the test scenario to determine performance characteristics of the publish/subscribe network system;

selecting portions of the data objects captured during the test scenario for use as replay data; and

simulating the publish/subscribe clients coupled to the publish/subscribe system to replicate portions of the test scenario on the publish/subscribe network system.

2. The method of claim 1, further comprising:

generating a plurality of test scenarios for the publish/subscribe network system; and

storing the plurality of test scenarios in a test scenario database.

3. The method of claim 2, further comprising:

selecting a test scenario from the plurality of test scenarios stored in the test scenario database; and

communicating the selected test scenario to the plurality of agents coupled to the publish/subscribe network system.

4. The method of claim 1, further comprising recording timestamps on the data objects propagating between the publish/subscribe clients and the publish/subscribe network system according to the test scenario with the plurality of agents.

5. The method of claim 1, further comprising storing the results of the test scenario in a local results database within each of the plurality of agents.

6. The method of claim 1, further comprising receiving a request from the analyzer for the results of the test scenario after the test scenario is complete, and communicating the results of the test scenario from the agents to the analyzer in response to the request.

7. The method of claim 1, wherein each of the plurality of agents includes shim adapters inserted within the application layer of the publish/subscribe clients coupled to the publish/subscribe network system.

8. The method of claim 7, wherein capturing data objects comprises capturing data objects propagating through the shim adapters within the application layer of the publish/subscribe clients.

9. The method of claim 7, further comprising recording a first timestamp on a data object when the data object enters one of the shim adapters, and recording a second timestamp on the data object when the data object exits the one of the shim adapters.

10. The method of claim 7, further comprising simulating the publish/subscribe clients coupled to the publish/subscribe system with the shim adapters during the test scenario when the publish/subscribe system is under development.

11. The method of claim 7, wherein communicating results of the test scenario comprises communicating shim results from the network layer to the analyzer, and wherein analyzing the results of the test scenario comprise analyzing the shim results to determine performance characteristics of the publish/subscribe network system over the application layer.

12. The method of claim 1, wherein capturing data objects comprises capturing data objects propagating within the network layer of the publish/subscribe clients.

13. The method of claim 12, further comprising recording a timestamp on a data object when the data object passes through the network layer of one of the publish/subscribe clients.

14. The method of claim 12, wherein communicating results of the test scenario comprises communicating network results from the network layer to the analyzer, and wherein analyzing the results of the test scenario comprise aggregating captured network packets within the network results, decoding the aggregated network packets, and analyzing the decoded network results to determine performance characteristics of the publish/subscribe network system over the network layer.

15. The method of claim 14, wherein the captured network packets conform to the remote method invocation (RMI) message protocol, and wherein decoding the captured network packets comprises performing a RMI decode.

16. The method of claim 1, wherein capturing data objects comprises capturing data objects propagating within the application layer of the publish/subscribe clients with shim adapters, and capturing data objects propagating within the network layer of the publish/subscribe clients with network monitors.

17. The method of claim 16, wherein communicating results of the test scenario comprises communicating shim results from the application layer and network results from the network layer to an integrated analyzer.

18. The method of claim 17, wherein analyzing the results of the test scenario comprise:

decoding the network results;

matching data objects within the decoded network results to data objects within the shim results; and

analyzing the matched data objects to determine performance characteristics of the publish/subscribe network system over both the application layer and the network layer.

19. The method of claim 1, wherein the publish/subscribe system comprises one of a deployed publish/subscribe system or a developing publish/subscribe system.

20. A computer-readable medium comprising instructions that cause a programmable processor to:

generate a test scenario for a publish/subscribe network system;

issue commands to a plurality of agents positioned at different locations within the publish/subscribe network system to initiate the test scenario;

capture data objects propagating between publish/subscribe clients and the publish/subscribe network system according to the test scenario with the plurality of agents;

communicate results of the test scenario including the captured data objects to an analyzer coupled to the publish/subscribe network system;

analyze the results of the test scenario to determine performance characteristics of the publish/subscribe network system;

select portions of the data objects captured during the test scenario for use as replay data; and

simulate the publish/subscribe clients coupled to the publish/subscribe system to replicate portions of the test scenario on the publish/subscribe network system.

21. The computer-readable medium of claim 20, further comprising instructions that cause the programmable processor to record timestamps on the data objects propagating between the publish/subscribe clients and the publish/subscribe network system according to the test scenario with the plurality of agents.

22. The computer-readable medium of claim 20, wherein each of the plurality of agents includes shim adapters inserted within the application layer of the publish/subscribe clients coupled to the publish/subscribe network system, and wherein the instructions cause the programmable processor to capture data objects propagating through the shim adapters within the application layer of the publish/subscribe clients.

23. The computer-readable medium of claim 22, further comprising instructions that cause the programmable processor to record a first timestamp on a data object when the data object enters one of the shim adapters, and record a second timestamp on the data object when the data object exits the one of the shim adapters.

24. The computer-readable medium of claim 22, further comprising instructions that cause the programmable processor to simulate the publish/subscribe clients coupled to the publish/subscribe system with the shim adapters during the test scenario when the publish/subscribe system is under development.

25. The computer-readable medium of claim 20, wherein the instructions cause the programmable processor to capture data objects propagating within the network layer of the publish/subscribe clients.

26. The computer-readable medium of claim 20, further comprising instructions that cause the programmable processor to record a timestamp on a data object when the data object passes through the network layer of one of the publish/subscribe clients.

27. The computer-readable medium of claim 20, wherein the instructions cause the programmable processor to capture data objects propagating within the application layer of the publish/subscribe clients with shim adapters, and capture data objects propagating within the network layer of the publish/subscribe clients with network monitors.

28. The computer-readable medium of claim 27, wherein the instructions cause the programmable processor to communicate shim results from the application layer and network results from the network layer to an integrated analyzer.

29. The computer-readable medium of claim 28, wherein the instructions cause the programmable processor to:

decode the network results;

match data objects within the decoded network results to data objects within the shim results; and

analyze the matched data objects to determine performance characteristics of the publish/subscribe network system over both the application layer and the network layer.

30. A system comprising:

a publish/subscribe network system;

publish/subscribe clients coupled to the publish/subscribe network system;

a plurality of agents positioned at different locations within the publish/subscribe network system; and

an analyzer coupled to the publish/subscribe network including a controller that generates a test scenario for the publish/subscribe network system and issues commands to the plurality of agents to initiate the test scenario,

wherein the plurality of agents capture data objects propagating between the publish/subscribe clients and the publish/subscribe network system according to the test scenario, and communicate results of the test scenario including the captured data objects to the analyzer, wherein the analyzer analyzes the results of the test scenario to determine performance characteristics of the publish/subscribe network system, and

wherein each of the plurality of agents includes a shim replay module that selects portions of the data objects captured during the test scenario for use as replay data, and simulates the publish/subscribe clients coupled to the publish/subscribe system to replicate portions of the test scenario on the publish/subscribe network system.

31. The system of claim 30, wherein the controller includes a scenario generation module that generates a plurality of test scenarios for the publish/subscribe network system, and test scenario database that stores the plurality of test scenarios.

32. The system of claim 30, wherein the controller includes a scenario controller that selects a test scenario from the plurality of test scenarios stored in the test scenario database, and communicates the selected test scenario to the plurality of agents coupled to the publish/subscribe network system.

33. The system of claim 30, wherein the plurality of agents record timestamps on the data objects propagating between the publish/subscribe clients and the publish/subscribe network system according to the test scenario.

34. The system of claim 30, wherein each of the plurality of agents includes a local results database that stores the results of the test scenario.

35. The system of claim 30, wherein each of the plurality of agents receives a request from the analyzer for the results of the test scenario after the test scenario is complete, and communicates the results of the test scenario to the analyzer in response to the request.

36. The system of claim 30, wherein each of the plurality of agents includes shim adapters inserted within the application layer of the publish/subscribe clients coupled to the publish/subscribe network system.

37. The system of claim 36, wherein the shim adapters capture data objects propagating through the shim adapters within the application layer of the publish/subscribe clients.

38. The system of claim 36, wherein one of the shim adapters records a first timestamp on a data object when the data object enters the one of the shim adapters, and records a second timestamp on the data object when the data object exits the one of the shim adapters.

39. The system of claim 36, wherein the shim adapters simulate the publish/subscribe clients coupled to the publish/subscribe system during the test scenario when the publish/subscribe system is under development.

40. The system of claim 36, wherein the plurality of agents communicate shim results from the network layer to the analyzer, and wherein the analyzer analyzes the shim results to determine performance characteristics of the publish/subscribe network system over the application layer.

41. The system of claim 30, wherein the plurality of agents include network monitors that capture data objects propagating within the network layer of the publish/subscribe clients.

42. The system of claim 41, wherein one of the network monitors records a timestamp on a data object when the data object passes through the network layer of one of the publish/subscribe clients.

43. The system of claim 41, wherein the plurality of agents communicate network results from the network layer to the analyzer, and wherein the analyzer aggregates captured network packets within the network results, decodes the aggregated network packets, and analyzes the decoded network results to determine performance characteristics of the publish/subscribe network system over the network layer.

44. The system of claim 43, wherein the captured network packets conform to the remote method invocation (RMI) message protocol, and wherein the controller includes a decode module that performs a RMI decode.

45. The system of claim 30, wherein each of the plurality of agents includes shim adapters that capture data objects propagating within the application layer of the publish/subscribe clients, and network monitors that capture data objects propagating within the network layer of the publish/subscribe clients.

46. The system of claim 45, wherein the plurality of agents communicate shim results from the application layer and network results from the network layer to an integrated analyzer included in the controller.

47. The system of claim 46, wherein a decode module included in the controller decodes the network results, and wherein the integrated analyzer matches data objects within the decoded network results to data objects within the shim results, and analyzes the matched data objects to determine performance characteristics of the publish/subscribe network system over both the application layer and the network layer.

48. The system of claim 30, wherein the publish/subscribe system comprises one of a deployed publish/subscribe system or a developing publish/subscribe system.