REAL-TIME COMPRESSIVE DATA COLLECTION FOR CLOUD MONITORING

• US 20140047106A1
• Filed: 08/08/2012
• Published: 02/13/2014
• Est. Priority Date: 08/08/2012
• Status: Active Grant

• Current Assignee: Empire Technology Development LLC (Allied Inventors Management, LLC)
• Original Assignee: Empire Technology Development LLC (Allied Inventors Management, LLC)
• Inventors: Leung, Henry, Liu, Xiaoxiang
• Granted Patent: US 9,032,072 B2
• Time in Patent Office: Days
• US Class Current: 709/224
• CPC Class Codes: H04L 41/142 using statistical or mathematical methods
• CPC Class Codes: H04L 41/5012 determining service availability, e.g. which services are available at a certain point in time
• CPC Class Codes: H04L 41/5096 wherein the managed service relates to distributed or central networked applications
• CPC Class Codes: H04L 43/08 Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route
• CPC Class Codes: H04L 43/12 Network monitoring probes
• CPC Class Codes: H04L 43/20 the monitoring system or the monitored elements being virtualised, abstracted or software-defined entities, e.g. SDN or NFV
• CPC Class Codes: H04L 69/04 Protocols for data compression, e.g. ROHC

Abstract

Technologies are presented for implementing a compressive-sensing-based data collection system in a cloud environment. In some examples, high-dimensional sensor data may be compressed using sparsity transforms and compressive sampling. The resulting low-dimensional data messages may be steered through a switch network to a cloud service manager, which then reconstructs the compressed messages for subsequent analysis, reporting, and/or comparable actions.

Claims

1. A method for compressive sensing based data collection in cloud monitoring, the method comprising:

receiving multi-dimensional data associated with performance of a cloud infrastructure collected by a plurality of probes within the cloud infrastructure;

determining a sparsity feature of the received multi-dimensional data;

applying compressive sensing to compress the multi-dimensional data into single-dimensional data using the sparsity feature, the single-dimensional data being suitable for use to reconstruct the multi-dimensional data; and

reconstructing the single-dimensional data into multi-dimensional data.

4. The method according to claim 1, wherein each probe is a physical instance of a data source within the cloud infrastructure providing state data and the plurality of probes include one or more of a processing unit, a monitoring sensor associated with a virtual machine, and a server.

5. The method according to claim 4, wherein the state data is associated with a property of a probe including one or more of a processor utilization, a memory utilization, a disk input/output (I/O) utilization, an adverse event associated with an application, a running time associated with an application, a resource allocation, context information, network usage, and disk temperature.

8. The method according to claim 1, further comprising:

compressing the multi-dimensional data by separating readings of each dimension of a probe vector that includes readings from a plurality of data sources and processing the separate readings respectively.

9. The method according to claim 1, further comprising:

compressing the multi-dimensional data by vectorizing messages from each probe and concatenating the vectorized messages into a single vector.

10. The method according to claim 1, further comprising:

compressing the multi-dimensional data from the plurality of probes employing a sparse transformation.

11. The method according to claim 10, wherein the sparse transformation includes one of discrete cosine transform or a wavelet transform.

12. The method according to claim 10, further comprising:

applying compressive sampling to the transformed multi-dimensional data by multiplying a single-dimensional data vector with a random construction matrix, wherein each column of the random construction matrix includes a series of random numbers for a probe corresponding to the single-dimensional data vector.

2-3. -3. (canceled)

6-7. -7. (canceled)

13-21. -21. (canceled)

22. A method for compressive sensing based data collection in cloud monitoring, the method comprising:

receiving a plurality of messages from a plurality of probes within a cloud infrastructure at aggregator switches of the cloud infrastructure;

generating multi-dimensional data associated with performance of the cloud infrastructure from the received messages;

determining a sparsity feature of the multi-dimensional data;

applying compressive sensing to compress the multi-dimensional data into single-dimensional messages using the sparsity feature by separating readings of each dimension of a probe vector that includes readings from a plurality of data sources and processing the separate readings respectively; and

steering the single-dimensional messages to a service manager within the cloud infrastructure to enable reconstructing of the single-dimensional messages into multi-dimensional data.

26. The method according to claim 22, further comprising:

initializing the plurality of probes by initializing a producer thread and a consumer thread, wherein the producer thread collects data from the plurality of probes and the consumer thread reads the collected data; and

adjusting a single-dimensional message transmission rate within the cloud infrastructure according to a real-time monitoring service specification.

36. The method according to claim 22, further comprising:

reconstructing the single-dimensional messages employing a linear programming optimization;

extracting sparse domain information from the reconstructed single-dimensional messages; and

transforming the reconstructed single-dimensional messages to original state domain by applying a reverse sparse domain transform.

23-25. -25. (canceled)

27-30. -30. (canceled)

35. (canceled)

37. A cloud-based datacenter configured to employ compressive sensing based data collection in cloud monitoring, the datacenter comprising:

a plurality of probes configured to collect data associated with performance of a plurality of nodes of a cloud infrastructure;

a plurality of aggregators configured to;

receive the collected data from the plurality of probes;

generate multi-dimensional data from the received data;

determine a sparsity feature of the multi-dimensional data; and

apply compressive sensing to compress the multi-dimensional data into single-dimensional data using the sparsity feature; and

a cloud monitoring service configured to manage steering of the single-dimensional data to a service manager, wherein the service manager is configured to reconstruct the single-dimensional data into multi-dimensional data.

45. The datacenter according to claim 37, wherein the aggregators are configured to compress the multi-dimensional data from the plurality of probes employing a sparse transformation.

46. The datacenter according to claim 45, wherein the sparse transformation includes one of discrete cosine transform or a wavelet transform.

55. The datacenter according to claim 37, wherein the cloud monitoring service is configured to:

steer the single-dimensional data directly via multiple levels of switches to the service manager within the cloud infrastructure.

38-44. -44. (canceled)

47. (canceled)

48. The datacenter according to claim 47, wherein a random global seed is broadcast from a central node of the cloud infrastructure such that each probe generates its own seed using the global seed and a probe identification to avoid transmitting the random construction matrix throughout the cloud infrastructure.

49-54. -54. (canceled)

56-77. -77. (canceled)