Storage of mass data for monitoring

• US 8,381,039 B1
• Filed: 06/29/2009
• Issued: 02/19/2013
• Est. Priority Date: 10/20/2008
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Sarma, Prashant L., Vanderbilt, Monty, Azari, David R., Osiecki, Daniel L., Schmidt, Caitlyn R.
• Primary Examiner(s): Manoskey, Joseph D
• Application Number:
  US12/493,558
• Time in Patent Office: 1,331 Days
• Field of Search: 714/25, 714/26, 714/32, 714/33, 714/37, 714/38.1, 714/45, 714/47.1, 714/47.2, 714/48, 707/957
• US Class Current: 714/45
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3072 where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06F 11/34 Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
• CPC Class Codes: G06F 16/2458 Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
• CPC Class Codes: H04L 43/02 Capturing of monitoring data
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

Disclosed in various embodiments are systems and methods providing for storage of mass data such as metrics. A plurality of data models are generated in the server from a stream of metrics describing a state of a system. Each of the metrics is associated with one of a plurality of consecutive periods of time, and each data model represents the metrics associated with a corresponding one of the consecutive periods of time. The data models are stored in a data store and each of the metrics is discarded after use in generating at least one of the data models.

Claims

1. A method, comprising the steps of:

receiving a stream of metrics in a server generated by a system over a plurality of consecutive periods of time;

aggregating the metrics in the server generated within each consecutive period of time, thereby generating at least one aggregate metric for each one of the consecutive periods of time, wherein a degree of the aggregation of the metrics for each consecutive period of time results in the at least one aggregate metric that enables monitoring of a performance of the system with respect to at least one predefined operational threshold, and wherein a first one of the consecutive periods of time occurs before a second one of the consecutive periods of time, wherein a portion of the metrics generated during the first one of the consecutive periods of time are aggregated after aggregating at least a portion of the metrics generated during the second one of the consecutive periods of time;

storing the aggregate metrics in a memory in association with corresponding ones of the consecutive periods of time;

discarding the metrics after aggregating the metrics;

storing the at least one aggregate metric for a predefined period of time before discarding the at least one aggregate metric; and

generating an output based upon the aggregate metrics that indicates a performance of the system.

2. The method of claim 1, wherein the step of aggregating the metrics in the server generated within each consecutive period of time further comprises the steps of identifying minimum and maximum ones of the metrics received from the system for each of the consecutive periods of time.

3. The method of claim 1, wherein the step of aggregating the metrics in the server generated within each consecutive period of time further comprises the step of calculating an average of the metrics received from the system for each of the consecutive periods of time.

4. A method, comprising the steps of:

generating a plurality of data models in the server from a stream of metrics describing a state of a system, each of the metrics being associated with one of a plurality of consecutive periods of time, each data model representing the metrics associated with a corresponding one of the consecutive periods of time;

storing each of the data models for a predefined period of time;

discarding each of the data models at the end of the predefined period of time; and

discarding each of the metrics after use in generating at least one of the data models.

5. The method of claim 4, further comprising the step of receiving the stream of metrics in the server from the system, where the server is in data communication with the system.

6. The method of claim 4, further comprising the step of maintaining a list of the metrics for which at least one data model is stored.

7. The method of claim 6, further comprising the step of generating a directory of the metrics included in the list that is sent to a client.

8. The method of claim 4, further comprising the step of generating an output that indicates a performance of the system based upon the data models.

9. The method of claim 4, further comprising the step of updating a stored one of the data models with one of the metrics that is received out of temporal order.

10. The method of claim 4, wherein the step of generating the data models in the server from the metrics further comprises the step of maintaining a count of the metrics received from the system for each of the consecutive periods of time.

11. The method of claim 4, wherein the step of generating the data models in the server from the metrics further comprises the step of identifying a maximum one of the metrics received from the system for each of the consecutive periods of time.

12. The method of claim 4, wherein the step of generating the data models in the server from the metrics further comprises the step of calculating an average of the metrics received from the system for each of the consecutive periods of time.

13. The method of claim 4, wherein the step of generating the data models from the metrics further comprises the steps of aggregating the metrics in the server associated with each consecutive period of time, thereby generating at least one aggregate metric for each one of the consecutive periods of time.

14. The method of claim 13, wherein a degree of the aggregation of the metrics for each consecutive period of time results in at least one aggregate metric that enables monitoring of a performance of the system with respect to at least one predefined operational threshold.

15. A system, comprising:

a server;

logic executable in the server that generates a plurality of data models in the server from a plurality of metrics associated with a plurality of consecutive periods of time, each of the metrics describing a state of a system within a respective one of the consecutive periods of time;

logic executable in the server that stores the data models in a memory accessible to the server, each data model representing the metrics associated with a corresponding one of the consecutive periods of time; and

logic executable in the server that updates a stored one of the data models with one of the metrics that is received out of a temporal order.

16. The system of claim 15, further comprising logic executable in the server that discards the metrics after use in generating at least one of the data models.

17. The system of claim 15, wherein at least one of the data models further comprises a count of the metrics associated with a respective one of the consecutive periods of time.

18. The system of claim 15, wherein at least one of the data models maintains a minimum one of the metrics associated with a respective one of the consecutive periods of time.

19. The system of claim 15, wherein at least one of the data models maintains a maximum one of the metrics associated with a respective one of the consecutive periods of time.

20. The system of claim 15, wherein at least one of the data models further comprises an average of the metrics associated with a respective one of the consecutive periods of time.

21. The system of claim 15, further comprising logic executable in the server that stores each of the data models in a memory accessible to the server for a predefined period of time before discarding the data models.

22. The system of claim 15, further comprising logic executable in the server that generates an output that indicates a past performance of the system based upon the data models.

23. The system of claim 15, wherein the logic executable in the server that generates the data models in the server from the metrics associated with the consecutive periods of time, further comprising logic executable in the server that aggregates the metrics associated with each consecutive period of time, thereby generating at least one aggregate metric for each one of the consecutive periods of time.

24. The system of claim 23, wherein a degree of the aggregation of the metrics for each consecutive period of time results in at least one aggregate metric that enables monitoring of a performance of the system with respect to at least one predefined operational threshold.