Collecting and aggregating data using distributed resources

• US 7,917,541 B2
• Filed: 03/31/2008
• Issued: 03/29/2011
• Est. Priority Date: 03/31/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Yuan, Zhongsheng
• Primary Examiner(s): LEROUX, ETIENNE PIERRE
• Application Number:
  US12/058,789
• Publication Number:
  US 20090248712A1
• Time in Patent Office: 1,093 Days
• Field of Search: 707/2, 707/100, 707/802, 702/186
• US Class Current: 707/802
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 16/2465 Query processing support for facilitating data mining operations in structured databases

Abstract

A hierarchial tree is provided to harvest data from data sources. The hierarchial tree includes multiple tree node entities arranged in multiple levels. In one case, leaf node entities of the hierarchial tree are used to collect data from the data sources. The hierarchial tree includes storage resources for storing the collected data. The hierarchial tree further includes computing resources for aggregating the collecting data in one or more aggregation operations and performing other processing operations. A receiving entity can selectively and independently receive parts of the collected data and aggregated data.

Claims

1. A method for providing data, comprising:

passing policy information with node entity operation instructions from an initial node entity of a hierarchical tree of tree node entities to one or more target leaf node entities of the hierarchical tree, the passing comprising successively passing the policy information through a series of linked tree node entities of the hierarchical tree connecting the initial node entity to the one or more target leaf node entities;

based at least in part on the node entity operation instructions, collecting data using the hierarchical tree from a plurality of data sources, to thereby provide collected data, the hierarchical tree including a plurality of levels including a leaf level, tree node entities in the leaf level of the hierarchical tree corresponding to leaf node entities, the leaf node entities including the one or more target leaf node entities;

based at least in part on the node entity operation instructions, storing the collected data using storage resources provided by at least one portion of data processing equipment associated with the tree node entities of the hierarchical tree;

based at least in part on the node entity operation instructions, aggregating the collected data to form aggregated data using computing resources provided by one or both of;

the at least one portion or at least one other portion of the data processing equipment associated with the tree node entities of the hierarchical tree, the storage resources and the computing resources together comprising distributed resources provided by the hierarchical tree; and

based at least in part on the node entity operation instructions, selectively providing at least parts of the collected data and the aggregated data to a receiving entity.

2. The method of claim 1, wherein the collected data is made available to the receiving entity independently of the aggregated data.

3. The method of claim 1, wherein the collecting is performed using the leaf node entities.

4. The method of claim 1, wherein the collected data reflects performance of one or more devices associated with the leaf node entities and comprising one or more portions of the data processing equipment.

5. The method of claim 1, wherein the aggregating comprises performing successive aggregation operations at respective levels in the hierarchical tree,

wherein an initial aggregation operation performs aggregation directly on the collected data,

wherein each additional aggregation operation performs aggregation on aggregated data provided by an immediately preceding aggregation operation to yield further aggregated data,

wherein a final aggregation operation yields final aggregated data, and

wherein the receiving entity receives the final aggregated data from a representative of the plurality of tree node entities.

6. The method of claim 1, wherein the aggregating comprises at least one of:

computing a sum of a group of collected data items;

generating a count associated with the group of collected data items;

finding a minimum value in the group of collected data items;

or finding a maximum value in the group of collected data items.

7. The method of claim 1, wherein at least a subset of the tree node entities perform respective principal functions, and wherein the storage resources and computing resources of the hierarchical tree comprise spare resources that are not being utilized by the principal functions.

8. A computer-readable medium for storing computer-readable instructions, the computer-readable instructions providing an agent module when executed by one or more processing devices, the computer-readable instructions comprising:

collection module logic configured to provide collected data from at least one data source for storage in a data store; and

aggregation module logic configured to aggregate the collected data to form aggregated data,

the agent module being configured to selectively provide the collected data and the aggregated data to a receiving entity,

the agent module being provided on a production device, and the production device being a leaf member of a hierarchical tree of tree node entities; and

administrative module logic configured to receive information successively passed through the hierarchical tree, wherein the information provides rules governing use of resources of the production device by the agent module.

9. The computer-readable medium of claim 8, wherein the collected data reflects performance of the production device.

10. The computer-readable medium of claim 8, wherein the agent module is further configured to provide the aggregated data to a logically linked parent node entity of the hierarchical tree.

11. The computer-readable medium of claim 8, the computer-readable instructions further providing at least one other agent module when executed by one or more processing devices, the at least one other agent module being provided on the production device.

12. The computer-readable medium of claim 11, the production device being at least one other second leaf member of the hierarchical tree.