System and method of testing software and hardware in a reconfigurable instrumented network

• US 7,099,797 B1
• Filed: 07/08/2004
• Issued: 08/29/2006
• Est. Priority Date: 07/08/2003
• Status: Expired due to Fees

• Current Assignee: Avanza Technologies, Inc.
• Original Assignee: Avanza Technologies, Inc.
• Inventors: Richard, Philippe
• Primary Examiner(s): Hoff, Marc S.
• Assistant Examiner(s): Suglo, Janet L
• Application Number:
  US10/888,791
• Time in Patent Office: 782 Days
• Field of Search: 702/182, 702/183, 702/185, 702/186, 703/21, 709/224, 714/47
• US Class Current: 702/182
• CPC Class Codes: G06F 11/261 by simulating additional hardware, e.g. fault simulation
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3414 Workload generation, e.g. scripts, playback
• CPC Class Codes: G06F 11/3433 for load management allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 2201/88 Monitoring involving counting

Abstract

A method of testing a computer system in a testing environment formed of a network of routers, servers, and firewalls. Performance of the computer system is monitored. A log is made of the monitored performance of the computer system. The computer system is subjected to hostile conditions until it no longer functions. The state of the computer system at failure point is recorded. The performance monitoring is done with substantially no interference with the testing environment. The performance monitoring includes monitoring, over a sampling period, of packet flow, hardware resource utilization, memory utilization, data access time, or thread count. A business method entails providing a testing environment formed of a network of network devices including routers, servers, and firewalls, while selling test time to a customer on one or more of the network devices during purchased tests that test the security of the customer'"'"'s computer system. The purchased tests are conducted simultaneously with other tests for other customers within the testing environment. Customer security performance data based on the purchased tests is provided without loss of privacy by taking security measures to ensure that none of the other customers can access the security performance data. The tests may also be directed to scalability or reliability of the customer'"'"'s computer system. Data about a device under test is gathered using a managed information kernel that is loaded into the devices operating memory before its operating system. The gathered data is prepared as managed information items.

Claims

1. A system for testing a network architecture comprising:

a test network;

an inventory of network devices connected to the test network;

a closed test environment comprising selected network devices selected from the inventory of network devices, wherein the selected network devices are adapted for duplicating a configuration of the network architecture and wherein a selected network device comprises a resident operating system, a resident processor, and a resident memory;

a test protocol server connected to the test network and adapted for operating a test protocol to simulate a loading of the closed test environment, wherein the loading of the closed test environment comprises increasing a number of simulated users over time;

a managed information kernel adapted for;

loading into the resident operating memory of the selected network device,

wherein the managed information kernel operates independently of the resident

operating system and resident processor;

acquiring performance metrics from the resident processor indicative of the performance of the selected network device; and

transmitting the performance metrics of the selected network device to a managed information processor associated with the selected network device, wherein the managed information processor is independent of the resident processor.

2. The system of claim 1, wherein the selected network device is selected from the group consisting of a server, a router, a hub, a printer, a firewall, a wireless access point, and a storage device.

3. The system of claim 1, wherein the performance metrics are selected from the group consisting of packet flow, hardware resource utilization, memory utilization, data access time, and thread count.

4. The system of claim 1 further comprising a data store, and wherein the data store is adapted for:

receiving and storing the performance metrics from the managed information processor associated with the selected network device; and

receiving and storing other performance metrics from other managed information processors associated with other selected network devices of the closed test environment.

5. The system of claim 4, wherein the test protocol server is further adapted to:

retrieving the performance metrics and the other performance metrics from the data store; and

analyzing the performance metrics and the other performance metrics to determine a measure of performance of the network architecture.

6. The system of claim 5, wherein the measure of performance of the network architecture is selected from the group consisting of network response time, network capacity, network stress, network vulnerability, network scalability, and network stability.

7. The system of claim 1, wherein the test protocol server is further adapted for ceasing operation of the test protocol when a test objective is achieved.

8. The system of claim 7, wherein the test objective is selected from the group consisting of a predetermined unacceptable level of performance, a predetermined capacity minimum, a predetermined maximum response time, a predetermined error rate, and a failure of the test environment to function.

9. The system of claim 1, wherein the network architecture comprises software components and hardware components.

16. The method of testing a network architecture of claim 1 further comprising ceasing operation of the test protocol when a test objective is achieved.

17. The method of testing a network architecture of claim 16, wherein the test objective is selected from the group consisting of a predetermined unacceptable level of performance, a predetermined capacity minimum, a predetermined maximum response time, a predetermined error rate, and a failure of the test environment to function.

10. A method of testing a network architecture comprising:

defining a configuration of the network architecture;

creating a closed test environment from network devices selected from an inventory of network devices, wherein a selected network device comprises a resident operating system, a resident processor, and resident memory and wherein the selected network devices are selected to duplicate the configuration of the network architecture;

loading a managed information kernel into the resident memory of the selected network device, wherein the managed information kernel operates independently of the resident operating system and the resident processor;

operating a test protocol adapted for simulating loading of the closed test environment, wherein the loading of the closed test environment comprises increasing a number of simulated users over time;

acquiring performance metrics from the resident processor indicative of the performance of the selected network device during the operation of the test protocol; and

transmitting the performance metrics to a managed information processor associated with the selected network device independent of the resident processor.

11. The method of testing a network architecture of claim 10, wherein the selected network device is selected from the group consisting of a server, a router, a hub, a printer, a firewall, a wireless access point, and a storage device.

12. The method of testing a network architecture of claim 10, wherein the performance metrics are selected from the group consisting of packet flow, hardware resource utilization, memory utilization, data access time, and thread count.

13. The method of testing a network architecture of claim 10 further comprising:

storing the performance metrics from the managed information processor associated with the selected network device in a data store; and

storing other performance metrics from other managed information processors associated with other selected network devices of the closed test environment in the data store.

14. The method of testing a network architecture of claim 13 further comprising:

retrieving the performance metrics and the other performance metrics from the data store; and

analyzing the performance metrics and the other performance metrics to determine a measure of performance of the network architecture.

15. The method of testing a network architecture of claim 14, wherein the measure of performance of the network architecture is selected from the group consisting of network response time, network capacity, network stress, network vulnerability, network scalability, and network stability.

18. The method of testing a network architecture of claim 11, wherein the network architecture comprises software and hardware components.