Cloud computing benchmarking

• US 9,996,442 B2
• Filed: 03/25/2014
• Issued: 06/12/2018
• Est. Priority Date: 03/25/2014
• Status: Active Grant

• Current Assignee: Krystallize Technologies Inc.
• Original Assignee: Krystallize Technologies Inc.
• Inventors: France, Clinton
• Primary Examiner(s): Srivastava, Vivek
• Assistant Examiner(s): Barker, Todd L
• Application Number:
  US14/225,424
• Publication Number:
  US 20150278066A1
• Time in Patent Office: 1,540 Days
• CPC Class Codes: G06F 11/3428 Benchmarking
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis

Abstract

Cloud computing benchmarking is performed wherein the resource usage of a measuring benchmarking application is compensated for as to not impact measurement. The measurements are of a cloud instance'"'"'s benchmarking indicia which may include performance, functions and characteristics of the cloud instance. The benchmarking indicia use scalable measures as to allow the use of arithmetic operations such as those used in statistical functions. The benchmarking application is dispatched along with a configuration file and is controlled from a central controller to specified cloud instances. The dispatched benchmarking application takes measurements of the cloud instance based on the configuration file. The benchmarking application then stores the measurements in a results file for return back to the central controller. At the central controller, results files from one or more benchmarking applications are stored in a data store for comparative and statistical analysis.

Claims

1. A system to benchmark infrastructure, comprising:

a processor;

a memory communicatively coupled to the processor;

a benchmarking application that measures and stores one or more benchmark indicia of a cloud provider based at least in part on validating the cloud provider, the one or more benchmark indicia being associated with computing instances of the cloud provider; and

a central controller stored in the memory and operative on the processor, the central controller to;

dispatch, via a network connection between the central controller and the cloud provider, a configuration file and the benchmarking application to a first computing instance of the cloud provider and a second computing instance of the cloud provider, the first computing instance being associated with a secure execution of the benchmarking application and the second computing instance being associated with a non-secure execution of the benchmarking application, wherein the configuration file specifies the one or more benchmark indicia;

detect, prior to execution of the benchmarking application on the first computing instance and the second computing instance, initial measurements of the one or more benchmark indicia on the first computing instance and the second computing instance;

cause the benchmarking application to perform a secure execution on the first computing instance and a non-secure execution on the second computing instance, the benchmarking application to measure corresponding benchmark indicia for the first computing instance and the second computing instance; and

determine a compensation measurement that quantifies a computational latency between the secure execution of the benchmarking application on the first computing instance, the non-secure execution of the benchmarking application on the second computing instance, and the initial measurements of the one or more benchmark indicia.

2. The system of claim 1, wherein the one or more benchmark indicia include one or more scalable indicia.

3. The system of claim 1, wherein the one or more benchmark indicia are a measure of any one of performance, function, or characteristics of the cloud provider.

4. The system of claim 1, wherein the compensation measurement makes use of a statistical model of a behavior of the cloud provider.

5. The system of claim 1, wherein the central controller is further configured to dispatch a second benchmarking application to a second cloud provider, and the second benchmarking application, is configured to execute on the second cloud provider and operative to measure and store one or more benchmark indicia of the second cloud provider, and configured to compensate for at least one resource used by the second benchmarking application in storing the benchmark indicia of the second cloud provider.

6. The system of claim 5, wherein the cloud provider and the second cloud provider have different service abstractions.

7. They system of claim 1, wherein the benchmark indicia are any one of compute indicia, memory indicia, disk indicia, operating system indicia, network indicia, database indicia, or cloud indicia.

20. The system of claim 1, wherein the central controller is further configured to:

select performance indicia to measure the one or more benchmark indicia for one or more computing instances on a cloud provider, the performance indicia including at least one compute indicia that is associated with a physical or virtual processor core used by the cloud provider, at least one memory indicia that is associated with a physical or virtual memory used by the cloud provider, and at least one disk indicia that is associated with storage capacity available on the cloud provider.

8. A method to benchmark a cloud computing instance of a cloud provider, the method comprising:

receiving at a central controller a first address of a first cloud computing instance and a second address of a second cloud computing instance;

dispatching, via a network connection between the central controller and a server executing the first cloud computing instance and the second cloud computing instance, a benchmarking application from the central controller to the first cloud computing instance at the first address and the second cloud computing instance at the second address, the first cloud computing instance being associated with a secure execution of the benchmarking application and the second cloud computing instance being associated with a non-secure execution of the benchmarking application; and

dispatching to the first cloud computing instance and a second cloud computing instance, a configuration file that specifies one or more benchmark indicia to be measured by the benchmarking application, the configuration file including at least a version identifier of the configuration file;

causing the benchmarking application to perform a secure execution on the first cloud computing instance and a non-secure execution on the second cloud computing instance, the benchmarking application to;

validate the configuration file based at least in part on the version identifier;

detect, prior to execution of the benchmarking application on the first cloud computing instance and the second cloud computing instance, initial measurements of the one or more benchmark indicia on the first cloud computing instance and the second cloud computing instance; and

quantify a computational latency between the secure execution of the benchmarking application on the first cloud computing instance, the non-secure execution of the benchmarking application on the second cloud computing instance, and the initial measurements of the one or more benchmark indicia.

9. The method of claim 8, wherein the configuration file is dispatched in response to a request from the benchmarking application for a new version of the configuration file.

10. The method of claim 9, further comprising identifying at least one benchmark indicium of the one or more benchmark indicia to be measured at the central controller based at least on the cloud computing instance, and generating the configuration file at the central controller based on the at least one benchmark indicium.

11. The method of claim 8, further comprising at the central controller, dispatching a command to the benchmarking application, wherein the benchmarking application contains a permit-to-run flag such that the benchmarking application executes if the permit-to-run flag is turned on, and the benchmarking application does not execute if the permit-to-run flag is turned off, and the command from the central controller is a command to toggle the permit-to-run flag between being turned on and turned off.

12. The method of claim 11, further comprising:

receiving, at the central controller, an indication to halt measurement of the one or more benchmark indicia, and

wherein, dispatching the command to the benchmarking application includes toggling the permit-to-run flag to off.

13. The method of claim 8, further comprising:

creating at the central controller a data store entry to index results files of the benchmarking application;

receiving, from the benchmarking application, a results file that includes the one or more benchmark indicia; and

storing at the data store entry at the central controller the one or more benchmark indicia from the results file.

14. The method of claim 13, further comprising prior to storing at the data store entry at the central controller the one or more benchmark indicia, performing, at the central controller, pre-calculations associated with the one or more benchmark indicia, based at least in part on the results file.

15. The method of claim 13, further comprises:

associating, by the central controller, the benchmarking application with a metadata tag, and

storing, at the data store entry at the central controller, an association of the one or more benchmark indicia with the metadata tag.

16. The method of claim 13, further comprising:

causing the benchmarking application to encrypt the one or more benchmark indicia.

17. A system to benchmark infrastructure, comprising:

a processor;

a memory communicatively coupled to the processor;

a central controller stored in the memory and operative on the processor, the central controller to;

dispatch a benchmarking application to a first computing instance of a cloud provider and a second computing instance of the cloud provider, the benchmarking application to include a plurality of benchmark indicia, the first computing instance being associated with a secure execution of the benchmarking application and the second computing instance being associated with a non-secure execution of the benchmarking application,

wherein the benchmarking application is configured to measure the plurality of benchmark indicia on the first computing instance and the second computing instance;

detect, prior to execution of the benchmarking application on the first computing instance and the second computing instance, initial measurements of the plurality of benchmark indicia on the first computing instance and the second computing instance;

cause the benchmarking application to perform a secure execution on the first computing instance and a non-secure execution on the second computing instance; and

quantify a computational latency between the secure execution of the benchmarking application on the first computing instance, the non-secure execution of the benchmarking application on the second computing instance, and the initial measurements of the one or more benchmarking indicia;

receive, from the benchmarking application, a results file that includes the computational latency and the plurality of benchmark indicia; and

develop a statistical model based at least in part on the plurality of benchmark indicia, the statistical model to extrapolate a performance measurement for the cloud provider.

18. The system of claim 17, wherein the central controller further comprises a data store and wherein the central controller is further operative on the processor to:

store the results file in the data store.

19. The system of claim 18, wherein the central controller is further operative on the processor to:

perform statistical operations on the results file stored in the data store; and

perform a comparative analysis based at least on the statistical operations.