CLOUD COMPUTING BENCHMARKING

• US 20150278066A1
• Filed: 03/25/2014
• Published: 10/01/2015
• Est. Priority Date: 03/25/2014
• Status: Active Grant

• Current Assignee: Krystallize Technologies Inc.
• Original Assignee: Krystallize Technologies Inc.
• Inventors: France, Clinton
• Granted Patent: US 9,996,442 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3428 Benchmarking
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis

Abstract

Cloud computing benchmarking is performed wherein the resource usage of a measuring benchmarking application is compensated for as to not impact measurement. The measurements are of a cloud instance'"'"'s benchmarking indicia which may include performance, functions and characteristics of the cloud instance. The benchmarking indicia use scalable measures as to allow the use of arithmetic operations such as those used in statistical functions. The benchmarking application is dispatched along with a configuration file and is controlled from a central controller to specified cloud instances. The dispatched benchmarking application takes measurements of the cloud instance based on the configuration file. The benchmarking application then stores the measurements in a results file for return back to the central controller. At the central controller, results files from one or more benchmarking applications are stored in a data store for comparative and statistical analysis.

Claims

1. A system to benchmark infrastructure, comprising:

a processor;

a memory communicatively coupled to the processor; and

a central controller stored in the memory and operative on the processor, the central controller dispatches a benchmarking application to the cloud provider via a network connection between the central controller and the cloud provider,

wherein the benchmarking application executes on the cloud provider to measure and store one or more benchmark indicia of the cloud provider, and compensates for at least one resource used by the benchmarking application in measuring and storing the benchmark indicia.

2. The system of claim 1, wherein the measured benchmark indicia are scalable indicia.

3. The system of claim 1, where the measured benchmark indicia are a measure of any one of performance, function, or characteristics of the cloud provider.

4. The system of claim 1 where the compensation by the benchmarking application for the at least one resource used by the benchmarking application makes use of a statistical model of a behavior of the cloud provider.

5. The system of claim 1, wherein the central controller is further configured to dispatch a second benchmarking application to a second cloud provider, and the second benchmarking application, is configured to execute on the second cloud provider and operative to measure and store one or more benchmark indicia of the second cloud provider, and configured to compensate for at least one resource used by the second benchmarking application in storing the benchmark indicia of the second cloud provider.

6. The system of claim 5, wherein the cloud provider and the second cloud provider have different service abstractions.

7. They system of claim 1, wherein the benchmark indicia are any one of compute indicia, memory indicia, disk indicia, operating system indicia, network indicia, database indicia, or cloud indicia.

8. A method to benchmark a cloud computing instance, comprising:

receiving at a central controller an address of the cloud computing instance;

dispatching a benchmarking application from the central controller to the cloud computing instance at the address via a network connection between the central controller and a server executing the cloud computing instance; and

executing the benchmarking application to measure an benchmark indicium for the cloud computing instance, wherein an aggregate of the benchmark indicium with an additional benchmark indicium measured by the benchmarking application preserves both an additive property and a scalable property of a mathematical metric.

9. The method of claim 8, further comprising dispatching to the cloud computing instance a configuration file which specifies at least one benchmark indicia to be measured by the dispatched benchmarking application.

10. The method of claim 9, further comprising at the central controller identifying at least one benchmark indicia to be measured based at least on the cloud computing instance, and generating the dispatched configuration file at the central controller based on the identified at least one benchmark indicia.

11. The method of claim 8, further comprising at the central controller dispatching a command to the dispatched benchmarking application.

12. The method of claim 11, wherein the benchmarking application contains a permit-to-run flag such that the benchmarking application executes if the permit-to-run flag is turned on, and the benchmarking application does not execute if the permit-to-run flag is turned off, and the dispatched command from the central controller is a command to toggle the permit-to-run flag between being turned on and turned off.

13. The method of claim 11, further comprising the central controller dispatching a second benchmarking application containing a permit-to-run flag to a second cloud computing instance, and in response to receiving an emergency stop input, the central controller dispatching commands to both the first benchmarking application and the second benchmarking application to toggle their respective permit-to-run flags to off.

14. The method of claim 8, further comprising creating at the central controller a data store entry to index results files of the benchmarking application, and

measuring by the benchmarking application the at least one benchmarking indicia;

creating by the benchmarking application a results file;

storing in the created results file by the benchmarking application at least one measured benchmarking indicia;

sending by the benchmarking application the results file with the at least one stored benchmarking indicia; and

storing at the data store entry at the central controller the at least one stored benchmarking indicia.

15. The method of claim 14, further comprising prior to storing at the data store entry at the central controller the at least one stored benchmarking indicia, precalculating at the central controller at least one benchmarking indicia to be stored.

16. The method of claim 14, further comprises the central controller associating the benchmarking application with a metadata tag, and the storing in the created results file by the benchmarking application at least one measured benchmarking indicia further comprises storing the at least one measured benchmarking indicia with the metadata tag.

17. The method of claim 14, wherein the storing in the created results file by the benchmarking application further comprises encrypting the at least one measured benchmarking indicia.

18. A system to benchmark infrastructure, comprising:

a processor;

a memory communicatively coupled to the processor; and

a central controller stored in the memory and operative on the processor, the central controller dispatches a benchmarking application to the cloud provider the benchmarking application executes on the cloud provider to measure and store in a results file a plurality of benchmark indicia of the cloud provider and to return the results file to the central controller,

wherein an aggregate of plurality of the benchmark indicia measured by the benchmarking application preserves both an additive property and a scalable property of a mathematical metric.

19. The system of claim 18, wherein the central controller further comprises a data store and the central controller stores returned results files in the data store.

20. The system of claim 19, wherein the central controller performs statistical operations on the returned results files stored in the data store, and performs comparative analysis based at least on the performed statistical operations.