Conducting performance snapshots during test and using feedback to control test based on customer experience parameters

• US 9,727,449 B2
• Filed: 12/31/2014
• Issued: 08/08/2017
• Est. Priority Date: 12/30/2014
• Status: Active Grant

• Current Assignee: Spirent Communications Incorporated
• Original Assignee: Spirent Communications Incorporated
• Inventors: Hermeto, Guilherme, Buege, Brian
• Primary Examiner(s): KO, CHAE M
• Application Number:
  US14/587,997
• Publication Number:
  US 20160188445A1
• Time in Patent Office: 951 Days
• Field of Search: 714 472
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 11/3692 for test results analysis
• CPC Class Codes: H04L 41/5067 Customer-centric QoS measurements
• CPC Class Codes: H04L 43/045 for graphical visualisation of monitoring data
• CPC Class Codes: H04L 43/06 Generation of reports
• CPC Class Codes: H04L 43/062 related to network traffic
• CPC Class Codes: H04L 43/067 using time frame reporting
• CPC Class Codes: H04L 43/50 Testing arrangements
• CPC Class Codes: H04L 43/55 Testing of service level quality, e.g. simulating service usage
• CPC Class Codes: H04L 47/19 at layers above the network layer network arrangements for networked applications for scheduling or organising the servicing of application requests H04L67/60

Abstract

The technology disclosed enables understanding the user experience of accessing a web page under high loads. A testing system generates a simulated load by retrieving and loading a single web object. A performance snapshot is taken of accessing an entire web page from the server under load. The performance snapshot may be performed by emulating a browser accessing a web page'"'"'s URL, the web page comprising multiple objects that are independently retrieved and loaded. The simulated load is configured with a number of users per region of the world where the user load will originate, and a single object from the web page to retrieve. Performance data such as response time for the single object retrieved, number of hits per second, number of timeouts per sec, and errors per second may be recorded and reported. An optimal number of users may be determined to achieve a target user experience goal.

Claims

1. A method of understanding user experience under high loads and diverse regional traffic mix, the method including:

initiating a test using a hybrid test system including first simulated load generators and second web browser emulators that send a multitude of object requests related to a web page to a system under test from one or more user selected regions;

wherein the first simulated load generators;

generate object requests for at least one object on the web page, thereby simulating an object request load resulting from at least dozens of client-requestors; and

report at least object response time and error rates responsive to the object requests; and

wherein the second web browser emulators;

emulate a sequence of requests for multiple separately loadable object components of the web page from a plurality of the second web browser emulators; and

report at least initial response time, object load time, and returned errors for the separately loadable components;

receiving reports of test result statistics from the first simulated load generators and the second web browser emulators; and

reporting at least some of the test result statistics for display to a user.

2. The method of claim 1, wherein the reporting includes reporting a plurality of graphs depicting at least a number of simulated users, response times from the second web browser emulators, hit rates from the first simulated load generators, and error rates from the first simulated load generators.

3. The method of claim 1, further including reporting test result statistics from at least one of the second web browser emulators organized by requested object component and formatted to include a cascaded bar chart of initial response times and object load times for at least some of the separately loadable object components.

4. The method of claim 1, further including reporting test result statistics averaged from a plurality of the second web browser emulators operating on hardware in one of the user selected regions, organized by requested object component, and formatted to include a cascaded bar chart of object response times for at least some of the separately loadable object components.

5. The method of claim 1, further including reporting test result statistics from a plurality of the second web browser emulators operating on hardware in one of the user selected regions, organized by requested object component, and formatted to include a cascaded box-and-whisker chart of object response time distributions for at least some of the separately loadable object components.

6. The method of claim 1, further including reporting in multiple cascaded bar charts timing derived from reports of a plurality of the second web browsers under different system loads, wherein each cascaded bar chart illustrates the response times for at least some of the separately loadable object components.

7. The method of claim 3, further including:

receiving data indicating user selection of a bar in the cascaded bar chart; and

responsive to the user selection, transmitting data representing an approximate visual appearance to a user of the web page components at a time after start of loading the web page corresponding to the user selected bar.

8. The method of claim 3, wherein the cascade bar chart further includes milestone indicators of events that change a visual appearance with a browser of the web page.

9. The method of claim 1, further including:

receiving data that indicates a browser to emulate from among a plurality of browsers that have different operating characteristics; and

using a web browser emulator corresponding to the indicated browser to emulate.

10. The method of claim 2, further including in the hit rates and the error rates results from both the first simulated load generators and the second web browser emulators.

11. The method of claim 2, further including filtering the graphs by user selected region.

12. The method of claim 2, further including:

using the hybrid test system from multiple user selected regions with both the first simulated load generators and the second web browser emulators operating on hardware in the multiple user selected regions; and

varying a mix of traffic from the multiple regions during the test.

13. A computer-implemented system that implements a method of understanding user experience under high loads and diverse regional traffic mix, the implementation including:

a processor and a memory coupled to the processor, the memory storing computer instructions that implement a method, including;

initiating a test using a hybrid test system including first simulated load generators and second web browser emulators that send a multitude of object requests related to a web page to a system under test from one or more user selected regions;

wherein the first simulated load generators;

generate object requests for at least one object on the web page, thereby simulating an object request load resulting from at least dozens of client-requestors; and

report at least object response time and error rates responsive to the object requests; and

wherein the second web browser emulators;

emulate a sequence of requests for multiple separately loadable object components of the web page from a plurality of the second web browser emulators; and

report at least initial response time, object load time, and returned errors for the separately loadable components;

receiving reports of test result statistics from the first simulated load generators and the second web browser emulators; and

reporting at least some of the test result statistics for display to a user.

14. The computer-implemented system of claim 13, wherein the reporting further includes reporting in a plurality of graphs at least a number of simulated users, response times from the second web browser emulators, hit rates from the first simulated load generators, and error rates from the first simulated load generators.

15. The computer-implemented system of claim 13, wherein the implementation further includes reporting test result statistics averaged from a plurality of the second web browser emulators operating on hardware in one of the user selected regions, organized by requested object component, and formatted to include a cascaded bar chart of object response times for at least some of the separately loadable object components.

16. The computer-implemented system of claim 13, wherein the implementation further includes reporting test result statistics from a plurality of the second web browser emulators operating on hardware in one of the user selected regions, organized by requested object component, and formatted to include a cascaded box-and-whisker chart of object response time distributions for at least some of the separately loadable object components.

17. The computer-implemented system of claim 13, wherein the implementation further includes reporting in multiple cascaded bar charts timing derived from reports of a plurality of the second web browsers under different system loads, wherein each cascaded bar chart illustrates the response times for at least some of the separately loadable object components.

18. The computer-implemented system of claim 13, wherein the implementation further includes:

using the hybrid test system from multiple user selected regions with both the first simulated load generators and the second web browser emulators operating on hardware in the multiple user selected regions; and

varying a mix of traffic from the multiple regions during the test.

19. A non-transitory computer readable medium that stores computer instructions that, when run on hardware, implement understanding user experience under high loads and diverse regional traffic mix, the implementation including:

initiating a test using a hybrid test system including first simulated load generators and second web browser emulators that send a multitude of object requests related to a web page to a system under test from one or more user selected regions;

wherein the first simulated load generators;

generate object requests for at least one object on the web page, thereby simulating an object request load resulting from at least dozens of client-requestors; and

report at least object response time and error rates responsive to the object requests; and

wherein the second web browser emulators;

emulate a sequence of requests for multiple separately loadable object components of the web page from a plurality of the second web browser emulators; and

report at least initial response time, object load time, and returned errors for the separately loadable components;

receiving reports of test result statistics from the first simulated load generators and the second web browser emulators; and

reporting at least some of the test result statistics for display to a user.

20. The non-transitory computer readable medium of claim 19, the implementation further including reporting in a plurality of graphs at least a number of simulated users, response times from the second web browser emulators, hit rates from the first simulated load generators, and error rates from the first simulated load generators.

21. The non-transitory computer readable medium of claim 19, the implementation further including reporting test result statistics from at least one of the second web browser emulators organized by requested object component and formatted to include a cascaded bar chart of object response times for at least some of the separately loadable object components.

22. The non-transitory computer readable medium of claim 19, the implementation further including:

receiving data indicating user selection of a bar in the cascaded bar chart; and

responsive to the user selection, transmitting data representing an approximate visual appearance to a user of the web page components at a time after start of loading the web page corresponding to the user selected bar.

23. The non-transitory computer readable medium of claim 19, wherein the cascade bar chart further includes milestone indicators of events that change a visual appearance with a browser of the web page.

24. The non-transitory computer readable medium of claim 19, the implementation further including:

receiving data that indicates a browser to emulate from among a plurality of browsers that have different operating characteristics; and

using a web browser emulator corresponding to the indicated browser to emulate.

25. The non-transitory computer readable medium of claim 21, further including in the hit rates and the error rates results from both the first simulated load generators and the second web browser emulators.