Systems and methods for automated web performance testing for cloud apps in use-case scenarios

• US 10,372,600 B2
• Filed: 03/01/2017
• Issued: 08/06/2019
• Est. Priority Date: 03/01/2017
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Mathur, Ashraya R.
• Primary Examiner(s): Nguyen, Duy Khuong T
• Assistant Examiner(s): Hebert, Theodore E
• Application Number:
  US15/446,152
• Publication Number:
  US 20180253373A1
• Time in Patent Office: 888 Days
• Field of Search: 717124
• CPC Class Codes: G06F 11/32 with visual or acoustical indication of the functioning of the machine
• CPC Class Codes: G06F 11/34 Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
• CPC Class Codes: G06F 11/3404 for parallel or distributed programming
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 11/3692 for test results analysis
• CPC Class Codes: G06F 8/60 Software deployment

Abstract

Systems and methods for measuring performance metrics of apps where a controller schedules performance testing of a plurality of apps to generate a set of performance metrics from a client, server and device relating to performance of each app wherein the generated set of performance metrics comprises processing times and requests of the app. The scheduled performance testing is executed by a combination of the client, server, and device includes different networks, operating systems, and browsers. A performance engine captures the set of performance metrics of each app from the different client, server and device, and organizes the app metrics into categories based on an instrumentation and profile of each app. The categories include clusters comprising performance metrics of the client, server, and device. A user interface renders the set of performance metrics to facilitate comparisons between each cluster and category of the set of performance metrics.

Claims

1. An automated testing system for measuring performance metrics of apps, comprising:

at least one processor deployed on a server being programmed to implement the measuring of metrics for the performance testing by the server of a plurality of app coupled to the server;

the at least one processor scheduling performance testing of a plurality of apps to generate a set of performance metrics from a client, server and device relating to performance of each app wherein the performance metrics comprises processing times and requests associated with the app, wherein the scheduling performance testing of each app is executed by a combination of the client, server, and device comprising different networks, operating systems, and browsers;

the at least one processor having a performance engine to capture the set of performance metrics of each app from the client, server and device, and further for organizing the set of performance metrics into categories based on an instrumentation and profile of each app wherein the categories comprise clusters of the performance metrics;

the at least one processor having a graphic user interface for rendering the set of performance metrics in a manner to facilitate comparisons between each cluster and category of the set of performance metrics; and

the at least one processor having an input to receive data from processors of the client, server, and network to present in a first instance in a waterfall graph of event logging and processing time for events across the client, server and network to;

visually monitor processing at different stages of the instrumentation, determine lengths of processing time at each of the different stages and determine whether an event is processed faster at the server than at the network; and

in a second instance to present in a flame graph for discrete events of creation time for each component to;

inspect component creation time on a page, and inspect the component creation time for a performance hot spot whereby a bottleneck in an event and component processing is identified visually at an event level across the client, server and network and at a component level across components of the page related to the client, server and network by a particular display of the waterfall and flame graphs.

2. The system of claim 1, wherein the generated set of performance metrics from the networks comprise performance metrics related to a plurality of different network protocols.

3. The system of claim 1, wherein the controller is configured to implement a set of instructions to download the plurality of apps, load pages and capture performance metrics for each page of the app across a plurality of scenarios from the server to each client using different combinations of the network, operating system and browser.

4. The system of claim 1, wherein the performance testing comprises testing on different sets of at least two browsers.

5. The system of claim 1, wherein the performance testing comprises testing on sets of at least two operating systems.

6. The system of claim 1, further comprising:

a database which stores the set of performance metrics which is captured and enables the controller to retrieve the set of performance metrics for automated analysis and trending by the performance engine to determine an improvement or regression of a feature of an app through an app development cycle.

7. The system of claim 1, further comprising:

a plurality of types of performance metrics provided of the client, server and network at least comprising;

client metrics of page loading and rendering time, component loading and rendering time, and client processor profiles;

server metrics of server processing time, request time of each layer of a server, number of calls and latency time for app services to a server, application program interfaces (APIs) executions at a server, and server database structured query language (SQL) statements; and

network metrics of network requests, time in a network layer, and network navigation timing API metrics.

8. A method for measuring performance metrics of apps during an app development cycle, the method comprising:

implementing performance testing by a server of a plurality of apps coupled to the server by programming a processor of the server to schedule performance testing of a plurality of apps to generate a set of performance metrics from a client, server and network relating to performance of each app wherein the set of performance metrics comprises processing and request times associated with the app, wherein the scheduling performance testing of each app is executed from a processor of the server for a combination of clients, servers, and networks comprising different devices, operating systems, and browsers;

capturing by a performance engine the set of performance metrics of each app from the clients, servers and networks, and further organizing the set of performance metrics into categories based on an instrumentation and profile of each app wherein the categories comprise clusters of the performance metrics;

rendering by a user interface the set of performance metrics in a manner to facilitate comparisons between each cluster and category of the set of performance metrics for automated analysis and trending by the performance engine to determine an improvement or regression of a feature of an app through the app development cycle; and

receiving data from the processor of the client, server, and network to present in a first instance in a waterfall graph of event logging and processing time for events across the client, server and network to;

visually monitor processing at different stages of the instrumentation, determine lengths of processing time at each of the different stages and determine whether an event is processed faster at the server than at the network; and

in a second instance to present in a flame graph for discrete events of creation time for each component to;

inspect component creation time on a page, and inspect the component creation time for a performance hot spot whereby a bottleneck in an event and component processing is identified visually at an event level across the client, server and network and at a component level across components of the page related to the client, server and network by a particular display of the waterfall and flame graphs.

9. The method of claim 8, wherein the generated set of metrics from the networks comprises performance metrics related to a plurality of different network protocols.

10. The method of claim 8, wherein the controller is configured to implement a set of instructions to download the app, load pages and capture performance metrics for each page of the app across a plurality of scenarios from the server to each client using a different combinations of the network, operating system and browser.

11. The method of claim 8, wherein the performance testing comprises testing on different sets of at least two browsers.

12. The method of claim 8, wherein the performance testing comprises testing on sets of at least two operating systems.

13. The method of claim 8, further comprising:

storing by a database the set of performance metrics which is captured and enabling the controller to retrieve the set of performance metrics for automated analysis and trending by the performance engine to determine an improvement or regression of a feature of an app through an app development cycle.

14. The method of claim 8, further comprising:

providing a plurality of types of performance metrics of the client, server and network at least comprising;

client metrics of page loading and rendering time, component loading and rendering time, and client processor profiles;

server metrics of server processing time, request time of each layer of a server, and number of calls and time taken for app services, APIs executions at a server, and server database SQL statements; and

network metrics of requests, time in a network layer, and network navigation timing API metrics.

15. A system comprising:

at least one processor; and

at least one computer-readable storage device comprising instructions that when executed causes execution of a method of cloud automated testing for measuring performance metrics of apps during an app development cycle, the method comprising;

configuring a first module to connect a plurality of clients to an app server running the app;

configuring a second module to implement a plurality of operating systems;

configuring a third module to implement a plurality of web browsers;

configuring a fourth module to implement a plurality of network connections; and

downloading the app, and loading pages and capturing performance metrics for each page of the downloaded app across a plurality of scenarios from the app server to each client using a different combinations of the plurality of operating systems, web browsers and network connections; and

configuring a fifth module to implement a user interface for rendering the captured performance metrics during the app development cycle, the rendering comprising;

receiving data from processors of the client, server and network to cluster together for events across the client, server and network, and for particular discrete events of a page flow of creation time for each component on a page during the page flow to identify a bottleneck in an event and component processing at an event level across the client, server and network and at a component level across components of the page by a display of a waterfall graph and a flame graph for each event and component by;

in a first instance in the waterfall graph display of event logging and processing time for events across the client, server and network to;

visually monitor processing at different stages of instrumentation, determine lengths of processing time at each of the different stages and determine whether an event is processed faster at the server than at the network; and

in a second instance to present in the flame graph display for discrete events of creation time for each component to;

inspect component creation time on a page, and inspect the component creation time for a performance hot spot.

16. The system of claim 15, further comprising:

capturing a set of performance metrics of a client, server and network by a performance engine of each app from at least the client, server and network, and further organizing the set of performance metrics into categories based on an instrumentation and profile of each app wherein the categories of the performance metrics include clusters consisting of at least the client, server and network.

17. The system of claim 15, further comprising:

storing by a database the performance set of metrics which is captured and retrieving the performance set of metrics for automated analysis and trending by performance engine to determine an improvement or regression of a feature of an app through an app development cycle.

18. The system of claim 15, wherein the performance testing comprises testing on different sets of at least two operating systems.

19. The system of claim 15, wherein the performance testing comprise testing on different sets of at least two browsers.

20. The system of claim 15, wherein the generated set of metrics from the networks comprising performance metrics related to a plurality of different network protocols.

21. The system of claim 15, further comprising:

providing a plurality of types of performance metrics of the client, server and network at least comprising;

client metrics of page loading and rendering time, component loading and rendering time, and client processor profiles;

server metrics of server processing time, request time of each layer of a server, and number of calls and time taken for app services, APIs executions at a server, and server database SQL statements; and

network metrics of requests, time in a network layer, and network navigation timing API metrics.