Flexible and safe monitoring of computers

• US 10,298,477 B2
• Filed: 01/22/2016
• Issued: 05/21/2019
• Est. Priority Date: 06/29/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Maltz, David A., Goldstein, Jonathan David, Greenberg, Albert, Loboz, Charles, Patel, Parveen K.
• Primary Examiner(s): Joshi, Suraj M
• Assistant Examiner(s): Means, Jaren
• Application Number:
  US15/004,113
• Publication Number:
  US 20160149786A1
• Time in Patent Office: 1,215 Days
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3072 where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06F 11/3093 Configuration details thereof, e.g. installation, enabling, spatial arrangement of the probes
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 2201/88 Monitoring involving counting
• CPC Class Codes: G06F 2209/5013 Request control
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals
• CPC Class Codes: H04L 43/0817 by checking functioning
• CPC Class Codes: H04L 43/0876 Network utilisation, e.g. volume of load or congestion level

Abstract

Described is a server monitoring technology that is scalable to large numbers of servers, e.g., in a datacenter. Agents on servers run queries to monitor data sources for that server, such as performance counters and other events. The agents monitor their resource usage and those of monitored events to stay within an administrator-specified resource budget (policy), e.g., by modifying the running queries and/or monitoring parameters. A controller receives results of the monitoring, analyzes the results, and takes action as needed with respect to server operation and monitoring. The controller may dynamically update an agent'"'"'s queries, monitoring parameters and/or monitored data post-processing operations. The controller may issue alerts and reports, including alerts indicative of inter-server problems between two or more servers.

Claims

1. In a computing environment, a system comprising:

at least one processor; and

a dynamic query engine, implemented on the at least one processor and configured to at least;

monitor its own resource consumption based on a resource budget;

monitor the at least one processor using one or more queries;

collect monitoring data from one or more data sources via one or more input adapters;

determine that its own resource consumption has reached a limit relative to the resource budget; and

discard at least some portion of the monitoring data collected from the one or more data sources according to a policy in order to remain within the resource budget as the dynamic query engine continues to collect the monitoring data.

2. The system of claim 1, wherein the collected monitoring data from the one or more data sources includes at least one of event data, trace data, log data, performance data, provider data, instrumentation data, network data, or network traffic data.

3. The system of claim 1, wherein the dynamic query engine modifies at least one of the one or more queries based on the resource consumption of the dynamic query engine.

4. The system of claim 1, wherein the dynamic query engine receives the monitoring data from the one or more data sources as directed by the one or more queries running inside the dynamic query engine.

5. The system of claim 1, wherein the one or more data sources include at least one of tracing channels, network packet traces, event logs, provider events, instrumentation events, a network sniffer, or a performance counter.

6. The system of claim 1, wherein the dynamic query engine is further configured to:

determine, according to the policy, to pause at least one of the one or more queries in order to remain within the resource budget as the dynamic query engine continues to collect the monitoring data.

7. The system of claim 6, wherein the dynamic query engine is further configured to preserve monitoring data collected in a pipeline.

8. The system of claim 1, wherein the dynamic query engine receives one or more new queries or one or more new monitoring parameters, or both, from a controller that obtains and analyzes the collected monitoring data, and in response, changes one or more of the data sources, one or more of the queries being run, one or more current monitoring parameters, or one or more post-processing operations on the data, or any combination of one or more of the data sources, one or more of the queries being run, one or more of the monitoring parameters, or one or more post-processing operations on the data.

9. The system of claim 1, wherein the dynamic query engine further operates to estimate a resource requirement of creating an event corresponding to the one or more queries and includes the estimate in monitoring its own resource consumption relative to the one or more queries.

10. The system of claim 1, wherein the dynamic query engine is further configured to transmit output data associated with a first query of the one or more queries via one or more output adapters, and receive, at a first input adapter of the one or more input adapters, input data, the input data comprising at least a portion of the output data associated with the first query from a first output adapter of the one or more output adapters.

11. The system of claim 1, wherein the policy comprises a super-policy with a hard resource budget, and at least one subordinate policy, wherein the subordinate policy, when implemented, may prevent the dynamic query engine from exceeding the hard resource budget.

12. In a computing environment, a system comprising:

one or more computer servers;

one or more agents implemented on the one or more computer servers, an individual agent operating to;

collect data via one or more data sources of a server using one or more queries related to monitoring the server on which the individual agent is implemented, wherein a first query of the one or more queries corresponds to a pipeline of operations and a first operation of the pipeline of operations is a join operation;

monitor resource usage of the individual agent during the data collection based on;

a resource consumption budget relative to the individual agent during the data collection;

determine that the resource usage of the individual agent has reached a limit of the resource consumption budget;

discard at least some portion of the collected data from the one or more data sources according to a policy in order to remain within the resource consumption budget as the individual agent continues to collect data; and

process the remaining collected data into monitoring results for the server on which the individual agent is implemented.

13. The system of claim 12, wherein the one or more data sources include at least one of tracing channels, network packet traces, event logs, provider events, instrumentation events, a network sniffer, or a performance counter.

14. The system of claim 12, further comprising:

an aggregation agent implemented on another server other than the one or more servers, the aggregation agent operating to;

receive the monitoring results from the one or more agents; and

aggregate the received monitoring results for analysis by a controller.

15. The system of claim 14, wherein the controller receives monitoring results associated with the server from the agent implemented on that server, analyzes the monitoring results associated with that server, and uses the analysis associated with that server to configure another server running a similar or same application as that server, or having similar or the same machine configuration parameters, or both to configure the other server running the similar or same application as that server and having the similar or the same machine configuration parameters.

16. The system of claim 12, wherein the one or more agents receive the data from the one or more data sources as directed by the one or more queries running inside the one or more agents.

17. The system of claim 12, wherein the individual agent of the one or more agents further comprises:

a scheduler that monitors the resource usage of the individual agent with respect to the resource consumption budget.

18. In a computing environment, a method performed on at least one processor comprising:

running, by a dynamic query engine, one or more queries on a server to collect monitoring data from one or more data sources associated with the server, the monitoring data including at least one of event data, trace data, log data, performance data, provider data, instrumentation data, network data, or network traffic data via one or more input adapters;

monitoring, by the dynamic query engine, resource usage of the dynamic query engine with respect to a policy-specified resource budget during the monitoring data collection;

determine that the resource usage of the dynamic query engine has reached a limit relative to the policy-specified resource budget; and

filter at least some portion of the monitoring data collected from the one or more data sources in order to remain within the policy-specified resource budget as the dynamic query engine continues to collect the monitoring data.

19. The method of claim 18, further comprising:

logging at least some of the collected monitoring data from at least one data source for later processing as more resources become available.

20. The method of claim 18, wherein at least one of the one or more queries is directed to detect an anomaly and, in response to detecting the anomaly, obtain data related to logs corresponding to a time period relative to the detection of the anomaly.