Data driven emulation of application performance on simulated wireless networks

• US 10,548,034 B2
• Filed: 11/03/2017
• Issued: 01/28/2020
• Est. Priority Date: 11/03/2017
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Ganapathi, Tejaswini, Raghunath, Satish, Gal, Shauli, Chandrayana, Kartikeya, Che, Xu, Karapetov, Andrey
• Primary Examiner(s): Hsu, Alpus
• Application Number:
  US15/803,509
• Publication Number:
  US 20190141549A1
• Time in Patent Office: 816 Days
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 30/00 Computer-aided design [CAD]
• CPC Class Codes: G06F 30/20 Design optimisation, verification or simulation optimisation, verification or simulation of circuit designs G06F30/30
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 41/40 using virtualisation of network functions or resources, e.g. SDN or NFV entities
• CPC Class Codes: H04L 43/022 by sampling
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 43/08 Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 43/20 the monitoring system or the monitored elements being virtualised, abstracted or software-defined entities, e.g. SDN or NFV
• CPC Class Codes: H04L 43/50 Testing arrangements
• CPC Class Codes: H04W 24/02 Arrangements for optimising operational condition
• CPC Class Codes: H04W 24/06 Testing, supervising or monitoring using simulated traffic
• CPC Class Codes: H04W 84/042 Public Land Mobile systems, e.g. cellular systems

Abstract

A data driven approach to emulating application performance is presented. By retrieving historical network traffic data, probabilistic models are generated to simulate wireless networks. Optimal distribution families for network values are determined. Performance data is captured from applications operating on simulated user devices operating on a virtual machine with a network simulator running sampled tuple values.

Claims

1. A method, comprising:

retrieving historical network traffic data from a historical network traffic data store, wherein the historical network traffic data is associated with one or more parameterized data segments of a network request;

determining one or more representative statistics associated with a set of probability distribution function families associated with each parameterized data segment based on the historical network traffic data;

determining an optimal distribution family from the set of probability distribution function families associated with each parameterized data segment comprising the one or more representative statistics;

generating a tuple of network values based on sampling from the optimal distribution family associated with the network values;

generating a virtual machine operating on a simulated network, the simulated network using the tuple of network values;

causing operation of one or more applications on a simulated user device operating on the virtual machine;

determining at least one network parameter from the operation of the one or more applications on the simulated user device;

configuring the one or more applications based on the at least one network parameter to improve network performance of the one or more applications.

2. The method as recited in claim 1, further comprising generating an inference of performance based on the at least one network parameter.

3. The method as recited in claim 2, further comprising storing the inference of performance in a performance data store.

4. The method as recited in claim 1, further comprising:

receiving generated user requests at the one or more applications on the simulated user device operating on the virtual machine;

capturing at least one download outcome associated with a response to the generated user requests.

5. The method as recited in claim 4, further comprising:

capturing Transmission Control Protocol (TCP) parameters associated with the response.

6. The method as recited in claim 1, wherein the parameterized data segments of a network request comprises one or more of an application, a network type, a geography, or a time window.

7. The method as recited in claim 1, further comprising:

generating aggregate datasets of performance metrics comparing different combinations of network values of parameterized data segments that affect application performance based on the at least one network parameter from the operation of the one or more applications on the simulated user device.

8. One or more non-transitory computer-readable storage media, storing one or more sequences of instructions, which when executed by one or more processors cause performance of:

retrieving historical network traffic data from a historical network traffic data store, wherein the historical network traffic data is associated with one or more parameterized data segments of a network request;

determining one or more representative statistics associated with a set of probability distribution function families associated with each parameterized data segment based on the historical network traffic data;

determining an optimal distribution family associated with each parameterized data segment comprising the one or more representative statistics;

generating a tuple of network values based on sampling from the optimal distribution family associated with the network values;

generating a virtual machine operating on a simulated network, the simulated network using the tuple of network values;

causing operation of one or more applications on a simulated user device operating on the virtual machine;

determining at least one network parameter from the operation of the one or more applications on the simulated user device;

configuring the one or more applications based on the at least one network parameter to improve network performance of the one or more applications.

9. The one or more non-transitory computer-readable storage media as recited in claim 8, wherein the one or more sequences of instructions, which when executed by the one or more processors further cause performance of:

generating an inference of performance based on the at least one network parameter.

10. The one or more non-transitory computer-readable storage media as recited in claim 9, wherein the one or more sequences of instructions, which when executed by the one or more processors further cause performance of:

storing the inference of performance in a performance data store.

11. The one or more non-transitory computer-readable storage media as recited in claim 8, wherein the one or more sequences of instructions, which when executed by the one or more processors further cause performance of:

receiving generated user requests at the one or more applications on the simulated user device operating on the virtual machine;

capturing at least one download outcome associated with a response to the generated user requests.

12. The one or more non-transitory computer-readable storage media as recited in claim 11, wherein the one or more sequences of instructions, which when executed by the one or more processors further cause performance of:

capturing Transmission Control Protocol (TCP) parameters associated with the response.

13. The one or more non-transitory computer-readable storage media as recited in claim 8, wherein the one or more sequences of instructions, which when executed by the one or more processors further cause performance of:

generating aggregate datasets of performance metrics comparing different combinations of network values of parameterized data segments that affect application performance based on the at least one network parameter from the operation of the one or more applications on the simulated user device.

14. An apparatus, comprising:

a network modeler, implemented at least partially in hardware, configured to retrieve historical network traffic data from a historical network traffic data store, wherein the historical network traffic data is associated with one or more parameterized data segments of a network request;

wherein the network modeler is further configured to determine one or more representative statistics associated with a set of probability distribution function families associated with each parameterized data segment based on the historical network traffic data;

wherein the network modeler is further configured to determine an optimal distribution family associated with each parameterized data segment comprising the one or more representative statistics;

a tuple generator, implemented at least partially in hardware, configured to generate a tuple of network values based on sampling from the optimal distribution family associated with the network values;

a virtual machine generator, implemented at least partially in hardware, configured to generate a virtual machine operating on a simulated network, the simulated network using the tuple of network values;

a user device simulator, implemented at least partially in hardware, configured to cause operation of one or more applications on a simulated user device operating on the virtual machine;

a data modeler, implemented at least partially in hardware, configured to determine at least one network parameter from the operation of the one or more applications on the simulated user device;

a network optimizer, implemented at least partially in hardware, configured to configure the one or more applications based on the at least one network parameter to improve network performance of the one or more applications.

15. The apparatus as recited in claim 14, further comprising a performance inference generator, implemented at least partially in hardware, configured to generate an inference of performance based on the at least one network parameter.

16. The apparatus as recited in claim 15, further comprising a data store, implemented at least partially in hardware, configured to store the inference of performance in a performance data store.

17. The apparatus as recited in claim 14, further comprising:

wherein the user device simulator is further configured to receive generated user requests at the one or more applications on the simulated user device operating on the virtual machine;

a network parameter sampler, implemented at least partially in hardware, configured to capture at least one download outcome associated with a response to the generated user requests.

18. The apparatus as recited in claim 17, wherein the network parameter sampler is further configured to capture Transmission Control Protocol (TCP) parameters associated with the response.

19. The apparatus as recited in claim 14, wherein the parameterized data segments of a network request comprises one or more of an application, a network type, a geography, or a time window.

20. The apparatus as recited in claim 14, further comprising a data aggregator, implemented at least partially in hardware, configured to generate aggregate datasets of performance metrics comparing different combinations of network values of parameterized data segments that affect application performance based on the at least one network parameter from the operation of the one or more applications on the simulated user device.