Flexible and safe monitoring of computers

• US 9,274,842 B2
• Filed: 06/29/2010
• Issued: 03/01/2016
• Est. Priority Date: 06/29/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Maltz, David A., Goldstein, Jonathan David, Greenberg, Albert, Loboz, Charles, Patel, Parveen K.
• Primary Examiner(s): Gilles, Jude Jean
• Assistant Examiner(s): Means, Jaren M
• Application Number:
  US12/825,336
• Publication Number:
  US 20110320586A1
• Time in Patent Office: 2,072 Days
• Field of Search: 709/224
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3072 where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06F 11/3093 Configuration details thereof, e.g. installation, enabling, spatial arrangement of the probes
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 2201/88 Monitoring involving counting
• CPC Class Codes: G06F 2209/5013 Request control
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals
• CPC Class Codes: H04L 43/0817 by checking functioning
• CPC Class Codes: H04L 43/0876 Network utilisation, e.g. volume of load or congestion level

Abstract

Described is a server monitoring technology that is scalable to large numbers of servers, e.g., in a datacenter. Agents on servers run queries to monitor data sources for that server, such as performance counters and other events. The agents monitor their resource usage and those of monitored events to stay within an administrator-specified resource budget (policy), e.g., by modifying the running queries and/or monitoring parameters. A controller receives results of the monitoring, analyzes the results, and takes action as needed with respect to server operation and monitoring. The controller may dynamically update an agent'"'"'s queries, monitoring parameters and/or monitored data post-processing operations. The controller may issue alerts and reports, including alerts indicative of inter-server problems between two or more servers.

Claims

1. In a computing environment, a system comprising:

one or more processors;

a dynamic query engine, implemented on the one or more processors, comprising;

an agent that runs one or more queries on a server to collect monitoring data from one or more data sources; and

a scheduler that monitors resource usage of the agent with respect to a policy-specified resource budget and dynamically modifies at least one of the one or more queries or monitoring parameters based on the resource usage of the agent in order to remain within the policy-specified resource budget as the agent continues to collect the monitoring data.

2. The system of claim 1, wherein the dynamic query engine computes estimates of the resource usage of creating events corresponding to the monitoring data, and includes the estimates as part of monitoring the agent'"'"'s resource usage.

3. The system of claim 1, wherein the scheduler dynamically modifies a query by pausing the query, suspending the query, stopping the query or changing a resource priority of the query.

4. The system of claim 1, wherein the dynamic query engine dynamically modifies one or more of the queries or the monitoring parameters by sampling to obtain less than all of the data provided by a data source, or by combining the data provided by a data source into a lesser amount of data, or both by sampling to obtain less than all of the data provided by a data source and by combining the data provided by a data source into a lesser amount of data.

5. The system of claim 1, wherein the dynamic query engine logs at least some of the data provided by at least one data source for later processing as more resources become available.

6. The system of claim 1, wherein the policy-specified resource budget corresponds to a defined policy for the server, the defined policy comprising a super-policy and a subordinate policy, wherein the subordinate policy controls the resource usage by changing at least one monitoring parameter, performing sampling, or combining data into a lesser amount of data, or any combination of changing at least one monitoring parameter, performing sampling, or combining data into a lesser amount of data, and wherein the super-policy controls resource usage by at least one of pausing one or more queries, suspending one or more queries, stopping one or more queries or changing a resource priority of one or more queries in response to the subordinate policy being unable to control the resource usage to remain within the policy-specified resource budget.

7. The system of claim 1, wherein the query engine performs post-processing on the monitoring data to provide monitoring results for delivery to a data warehouse.

8. The system of claim 7, further comprising:

a controller that obtains and analyzes the monitoring data from the dynamic query engine.

9. The system of claim 1, wherein the dynamic query engine receives one or more new queries or one or more new monitoring parameters, or both, from a controller that obtains and analyzes the collected monitoring data, and in response, changes one or more of the data sources, one or more of the queries being run, one or more of the monitoring parameters, or one or more post-processing operations on the data, or any combination of one or more of the data sources, one or more of the queries being run, one or more of the monitoring parameters, or one or more post-processing operations on the data.

10. The system of claim 1, wherein output of a first query is used as input of a second query in a first query engine, or wherein output of the first query is used as input of the second query in a second query engine, or wherein output of the first query is used as input of the second query in the first query engine and as input of a third query in the second query engine.

11. In a computing environment, a system comprising:

one or more servers;

one or more agents implemented on the one or more servers and operating to;

collect data related to monitoring the one or more servers via one or more queries;

monitor resource usage of the one or more agents during the data collection;

enforce a policy-specified budget during the data collection, including to adjust at least one of a query plan or a task schedule for an agent based on the monitored resource usage associated with the agent; and

process the collected data into monitoring results for the one or more servers.

12. The system of claim 11, further comprising:

a controller implemented on a server other than the one or more servers, the controller operating to;

receive the monitoring results from the one or more agents;

post-process the monitoring results to obtain another result; and

adjust at least one of monitoring parameters, queries, or agent operation for at least one agent based on the other result.

13. The system of claim 12, wherein the controller processes historical data to determine one or more patterns corresponding to previous operating states of the one or more servers, and wherein the post-processing is based at least in part upon a comparison of the monitoring results that indicate a current operating state of the one or more servers versus one or more of the patterns.

14. The system of claim 11, further comprising:

an aggregation agent implemented on a server other than the one or more servers, the aggregation agent operating to;

receive the monitoring results from the one or more agents; and

aggregate the received monitoring results for analysis by a controller.

15. The system of claim 12, wherein the controller receives monitoring results associated with a server from an agent implemented on that server, analyzes the monitoring results associated with that server, and uses the analysis associated with that server to configure another server running a similar or same application as that server, or having similar or the same machine configuration parameters, or both to configure the other server running the similar or same application as that server and having the similar or the same machine configuration parameters.

16. The system of claim 12, wherein the controller separates a query into an online component and an offline component, the controller reconfiguring an agent with the online component, and using the offline component to combine results from a server corresponding to the online component with results received from another server.

17. In a computing environment, a method performed on at least one processor comprising:

running, by a dynamic query engine, one or more queries on a server to collect monitoring data from one or more data sources associated with the server;

monitoring resource usage of the dynamic query engine with respect to a policy-specified resource budget during the monitoring data collection; and

modifying at least one of the one or more queries or monitoring parameters during the monitoring data collection based on the monitored resource usage in order to remain within the policy-specified resource budget as the dynamic query engine continues to collect the monitoring data.

18. The method of claim 17, further comprising:

providing the monitoring data to a controller for analysis; and

receiving at least one of modified queries or modified monitoring parameters from the controller based at least in part on the provided monitoring data and at least in part upon health metrics.

19. The method of claim 17, further comprising:

logging at least some of the collected monitoring data from at least one data source for later processing as more resources become available.

20. The method of claim 17, further comprising:

providing the monitoring data to a controller for analysis, wherein the controller processes historical data to obtain a historical pattern; and

receiving at least one of modified queries or modified monitoring parameters from the controller based at least in part on the provided monitoring data and at least in part on results relative to the historical patterns.