Systems And Methods For Performance Testing Cloud Applications From Multiple Different Geographic Locations

• US 20170315902A1
• Filed: 07/17/2017
• Published: 11/02/2017
• Est. Priority Date: 08/26/2014
• Status: Active Grant

• Current Assignee: Cloudy Days Inc.
• Original Assignee: Cloudy Days Inc.
• Inventors: Moretto, Paola, Rossaro, Paola, MacArthur, Shawn
• Granted Patent: US 10,515,000 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/34 Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 11/3672 Test management
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 11/3692 for test results analysis

Abstract

A method and system for testing the end-to-end performance of cloud based applications. Real workload is created for the cloud based applications using synthetic users. The load and length of demand may be adjusted to address different models allowing the measurement and analysis of user performance metrics under desired conditions. Web applications and other cloud services may be tested from multiple cloud-based concurrent geographic locations in the world. The method may include generating controlled and complex web load from multiple concurrent geographic locations. The method may also include splitting traffic among multiple geographic locations. The method may also include processing test results and generating performance metrics per location as well as comparing locations. The method may also be applicable to cloud services such as mobile apps and API endpoints.

Claims

1. A cloud based performance testing system comprising:

a test creation and management interface in a cloud;

an execution mechanism on an application server which receives requests from the test creation and management interface in the cloud, processes the requests and generates load concurrently from multiple geographic locations in cooperation with an elastic load balancer based on parameters specified in the test creation and management interface and directed against a service under test in the cloud;

a results mechanism hardware server that collects, processes and stores performance measurements of the service under test from the multiple geographic locations in response to the load generated by the execution mechanism; and

a display mechanism that presents on a screen the performance measurements collected by the results mechanism.

2. The cloud based performance testing system of claim 1, wherein the service under test comprises a web application, a mobile application, or an API endpoint.

3. The cloud based performance testing system of claim 1, wherein the display mechanism compares performance measurements from multiple geographic locations.

4. The cloud based performance testing system of claim 1, wherein the display mechanism filters out specific geographic locations.

5. The cloud based performance testing system of claim 1, wherein the test creation and management interface allows a user to assign traffic percentages to different geographic locations.

6. The cloud based performance testing system of claim 1, wherein the test creation and management interface allows a user to set different performance goals for different geographic locations.

7. The cloud based performance testing system of claim 1, wherein the execution mechanism keeps track of available geographic regions, determines which regions are involved in a test, determines the number of workers needed for the test in each region, dynamically instantiates any workers in each region if they are not already available, and sends requests for test execution to the workers located in the regions involved in the test.

8. A cloud based performance testing system comprising:

a plurality of remotely distributed subscriber computers connected to an internet;

a network interface to the plurality of remotely distributed subscriber computers;

a plurality of application servers which receive requests sent by the remotely distributed subscriber computers to test a cloud service;

a plurality of traffic generator computers located in multiple geographic regions which receive instructions from the plurality of application servers to generate synthetic user load concurrently in the multiple geographic regions;

a service under test residing on one or more first computers connected to the internet against which the synthetic user load is directed;

a master database residing on a second computer which receives, records and processes measurements of a performance of the service under test in response to the synthetic user load; and

a user interface which displays data of the performance of the service under test.

9. The cloud based performance testing system of claim 8, wherein the plurality of application servers determine a percentage of traffic allocated to each of the geographic regions and calculates a number of workers needed to execute the traffic on the traffic generator computers located in the multiple geographic regions.

10. The cloud based performance testing system of claim 9, wherein for each of the regions, the plurality of application servers reserves workers that are already present in the region and instantiates additional workers in the region if more are needed.

11. The cloud based performance testing system of claim 8, wherein the plurality of application servers provision multiple instances of workers with the substantially the same image so they are aligned and synchronized.

12. The cloud based performance testing system of claim 8, wherein the plurality of application servers dispatch requests and maintain status information about the workers available in each of the geographic regions.

13. The cloud based performance testing system of claim 8, wherein the plurality of application servers emulate mobile network traffic.

14. The cloud based performance testing system of claim 8, wherein the mobile network traffic is emulated by any combination of uplink speed, downlink speed, and latency.

15. A method for testing cloud based applications comprising:

inputting of test parameters by a user, wherein the test parameters include multiple geographic regions;

creating a test based on the test parameters inputted by the user;

bringing on-line cloud based hardware and software resources in the multiple geographic regions required to execute the test;

generating a plurality of synthetic users in the multiple geographic regions required by the test parameters;

executing the test;

collecting data from a service under test challenged by the test in the multiple geographic regions; and

processing and displaying performance metrics from the test.

16. The method of claim 15, wherein the test parameters further include a total amount of traffic for the test.

17. The method of claim 16, wherein the test parameters further include a percentage of the total traffic allocated to each geographic region.

18. The method of claim 15, wherein the multiple geographic regions may be selected from any of a map, a menu, or a specific address.

19. The method of claim 15, wherein:

the synthetic uses may emulate mobile network traffic; and

different emulation parameters may be used for different geographic regions.

20. The method of claim 15, wherein:

the synthetic uses may emulate mobile network traffic; and

different emulation parameters may be used within the same geographic region.