System and method for dynamically testing networked target systems

• US 10,277,475 B2
• Filed: 08/03/2016
• Issued: 04/30/2019
• Est. Priority Date: 10/01/2013
• Status: Active Grant

• Current Assignee: Perforce Software Incorporated
• Original Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Inventors: Girmonsky, Alon
• Primary Examiner(s): Bonzo, Bryce P
• Assistant Examiner(s): Arcos, Jeison C
• Application Number:
  US15/227,560
• Publication Number:
  US 20160342507A1
• Time in Patent Office: 1,000 Days
• CPC Class Codes: G06F 11/368 for test version control, e.g. updating test cases to a new software version
• CPC Class Codes: G06F 11/3692 for test results analysis
• CPC Class Codes: G06F 11/3696 Methods or tools to render software testable
• CPC Class Codes: H04L 41/5009 Determining service level performance parameters or violations of service level contracts, e.g. violations of agreed response time or mean time between failures [MTBF]
• CPC Class Codes: H04L 43/062 related to network traffic
• CPC Class Codes: H04L 43/08 Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 43/50 Testing arrangements
• CPC Class Codes: H04L 43/55 Testing of service level quality, e.g. simulating service usage

Abstract

A computerized method and system for conducting performance testing of a networked target system (NTS). The method comprises receiving at least one instruction and at least one parameter generated respective of a plurality of actions provided in a received test specification; testing the performance of the NTS using the at least one instruction and the at least one parameter; and gathering information respective of the performance testing of the NTS.

Claims

1. A computerized method for conducting performance testing of a networked target system (NTS), comprising:

receiving, with one or more processors, at least one instruction and at least one parameter corresponding to a plurality of different actions provided in a received test specification, wherein the test specification indicates a plurality of different geographic locations form which load is to be applied to an NTS;

testing with one or more processors, the performance of the NTS using the at least one instruction and the at least one parameter, wherein the testing comprises applying load to the NTS with request to the NTS from the plurality of different geographic locations; and

gathering, with one or more processor, information respective of the performance testing of the NTS.

2. The computerized method of claim 1, further comprising:

dynamically changing the at least one instruction and the at least one parameter based on the received test specification or a modified test specification.

3. The computerized method of claim 1, wherein the NTS is a web resource.

4. The computerized method of claim 1, wherein the NTS is partially or completely in a cloud-computing platform, wherein the cloud-computing platform is at least one of:

a public cloud system, a private cloud system, and a hybrid cloud system.

5. The computerized method of claim 1, wherein the ARD is deployed in a cloud-computing platform, wherein the cloud-computing platform is at least one of:

a public cloud system, a private cloud system, and a hybrid cloud system.

6. The computerized method of claim 1, wherein the information respective of the performance testing includes at least one of:

an estimated maximum capacity;

if at least one error was received during testing, the at least one error received during testing;

a throughput of the NTS; and

a response time of the NTS.

14. The method of claim 1, wherein the test specification indicates testing for accessing the NTS from a given type of user device, the testing comprises:

applying a load to the NTS that simulates the given type of user device performing the plurality of actions on the NTS, wherein types of user devices include a smartphone and a desktop computer.

15. The method of claim 1, wherein the test specification includes:

an amount of simulated user devices to be used in testing;

respective geographical locations from the plurality of different geolocations for the simulated user devices;

a plurality of types of the simulated user devices; and

a type of test.

16. The method of claim 1, wherein the operations comprise:

identifying the test specification by receiving a selection of a predetermined test specification from a user terminal.

17. The method of claim 1, wherein the operations comprise:

at least partially defining at least some of the plurality of actions by receiving captured actions performed by a user terminal with respect to the NTS.

18. The method of claim 1, wherein:

the method comprises dynamically changing the at least one instruction or the at least one parameter based on the received test specification;

the plurality of actions include selecting items on a website with a plurality of simulated user devices in the plurality of different geolocations simulating a plurality of different types of user devices;

gathering information comprises determining elapsed times until hypertext transport protocol responses are received from the NTS to the requests from the simulated user devices;

the at least one instruction is a configuration instruction specifying at least part of an application-layer request;

the at least one parameter identifies a network address and port of the NTS;

testing comprises simulating load on the NTS from more than 10,000 users; and

the method comprises instantiating a first number of simulated user devices testing the NTS and then, after some testing, dynamically changing the number of simulated user devices to a second number of simulated user devices testing the NTS, the first number being different from the second number.

19. The method of claim 1, comprising:

steps for dynamic and real-time performance testing of networked systems.

7. One or more non-transitory computer readable media having stored thereon instructions configured to cause one or more processors to effectuate operations, comprising:

receiving, with one or more processors, at least one instructions and at least one parameter corresponding to a plurality of different actions provided in a received test specification;

testing, with one or more processors, the performance of an NTS using the at least one instruction and the at least one parameter; and

gathering, with one or more processors, information respective of the performance testing of the NTS.

8. A system configured to dynamically conduct performance testing of a networked target system (NTS), comprising:

a processor; and

a memory, the memory containing instructions that, when executed by the processor, configure the system to;

receive, by an action repeater device, at least one instruction and at least one parameter corresponding to a plurality of different actions provided in a received test specification, wherein the test specification indicates a plurality of different geographic locations from which load is to be applied to an NTS;

test the performance of the NTS using the at least one instruction and the at least one parameter, wherein the testing comprises applying load to the NTS with requests to the NTS from the plurality of different geographic locations; and

gather information respective of the performance testing of the NTS.

9. The system of claim 8, wherein the system is further configured to:

dynamically change the at least one instruction and the at least one parameter based on the received test specification or a modified test specification.

10. The system of claim 8, wherein the NTS is a web resource.

11. The system of claim 8, wherein the NTS is part of a hybrid cloud system.

12. The system of claim 8, wherein the ARD is deployed in a cloud-computing platform, wherein the cloud-computing platform is a hybrid cloud system.

13. The system of claim 8, wherein the testing report performance comprises:

an estimated maximum capacity;

at least one error received during testing;

a throughput of the NTS; and

a response time of the NTS.

20. The system of claim 8, wherein the test specification indicates testing for accessing the NTS from a given type of user device, the testing comprises:

applying load to the NTS that simulates the given type of user device performing the plurality of actions on the NTS, wherein types of user devices include a smartphone and a desktop computer.

21. The system of claim 8, wherein the test specification includes:

an amount of simulated user devices to be used in testing;

respective geographical locations from the plurality of different geolocations for the simulated user devices;

a plurality of types of the simulated user devices; and

a type of test.

22. The system of claim 8, wherein:

the system is configured to dynamically change the at least one instruction or the at least one parameter based on the received test specification;

the plurality of actions include selecting items on a website with a plurality of simulated user devices in the plurality of different geolocations simulating a plurality of different types of user devices;

gathering information comprises determining elapsed times until hypertext transport protocol responses are received from the NTS to the requests from the simulated user devices;

the at least one instruction is a configuration instruction specifying at least part of an application-layer request;

the at least one parameter identifies a network address and port of the NTS;

testing comprises simulating load on the NTS from more than 10,000 users; and

the method comprises instantiating a first number of simulated user devices testing the NTS and then, after some testing, dynamically changing the number of simulated user devices to a second number of simulated user devices testing the NTS, the first number being different from the second number.

24. The method of claim 20, wherein:

the test specification indicates the plurality of different geographic locations from which the load is to be applied to the NTS, and

wherein the testing comprises applying load to the NTS with the requests to the NTS from the plurality of different geographic locations, the requests corresponding to the selections of the items on the website.

25. The method of claim 20, wherein the test specification indicates:

an amount of simulated user devices to be used in testing;

respective geographical locations from the plurality of different geolocations for the simulated user devices;

the plurality of types of the simulated user devices; and

a type of test.

23. A computerized method for conducting performance testing of a networked target system (NTS), comprising:

receiving, with one or more processors, at least one instruction and at least one parameter corresponding to a plurality of different actions provided in a received test specification;

testing, with one or more processors, the performance of an NTS using the at least one instruction and the at least one parameter, wherein testing comprises;

dynamically changing the at least one instruction or the at least one parameter based on the received test specification, and wherein;

the plurality of actions include selecting items on a website with a plurality of simulated user devices in a plurality of different geolocations simulating a plurality of different types of user devices;

the at least one instruction is a configuration instruction specifying at least part of an application-layer request;

the at least one parameter identifies a network address and port of the NTS; and

testing comprises simulating load on the NTS from more than 10,000 users; and

instantiating, with one or more processors, a first number of simulated user devices testing the NTS and then, after some testing, dynamically changing the number of simulated user devices to a second number of simulated user devices testing the NTS, the first number being different from the second number; and

gathering, with one or more processors, information respective of the performance testing of the NTS, wherein gathering information comprises determining elapsed times until hypertext transport protocol responses are received from the NTS to requests from the simulated user devices.