Storage of mass data for monitoring

• US 8,819,497 B1
• Filed: 02/18/2013
• Issued: 08/26/2014
• Est. Priority Date: 10/20/2008
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Osiecki, Daniel L., Sarma, Prashant L., Vanderbilt, Monty, Azari, David R., Schmidt, Caitlyn R.
• Primary Examiner(s): Manoskey, Joseph D
• Application Number:
  US13/769,468
• Time in Patent Office: 554 Days
• Field of Search: 714/25, 714/26, 714/32, 714/33, 714/37, 714/38.1, 714/45, 714/47.1, 714/47.2, 714/48, 707/957
• US Class Current: 714/45
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3072 where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06F 11/34 Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
• CPC Class Codes: G06F 16/2458 Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
• CPC Class Codes: H04L 43/02 Capturing of monitoring data
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

Disclosed are various in various embodiments are systems and methods providing for storage of mass data such as metrics. A plurality of data models are generated in the server from a stream of metrics describing a state of a system. Each of the metrics is associated with one of a plurality of consecutive periods of time, and each data model represents the metrics associated with a corresponding one of the consecutive periods of time. The data models are stored in a data store and each of the metrics is discarded after use in generating at least one of the data models.

Claims

1. A method comprising:

maintaining, by a computing device, a list of a plurality of metrics in an active metric list, the plurality of metrics being associated with an active metric status indicating that a respective one of the plurality of metrics has been stored in a memory for less than a predefined period of time;

aggregating, by the computing device, the plurality of metrics associated with the active metric status and generated within individual ones of a plurality of consecutive periods of time, thereby generating at least one aggregate metric for the individual ones of the plurality of consecutive periods of time, wherein;

a degree of aggregation of the plurality of metrics for the individual ones of the plurality of consecutive periods of time results in the at least one aggregate metric that enables monitoring of a performance of a system with respect to at least one predefined operational threshold;

generating, by the computing device, an output based at least in part on the at least one aggregate metric that indicates the performance of the system;

sending, by the computing device, the output to at least one requesting computing device; and

removing, by the computing device, at least a portion of the plurality of metrics in the active metric list having been stored in the memory for more than the predefined period of time.

2. The method of claim 1, wherein the aggregating, by the computing device, of the plurality of metrics generated within the individual ones of the plurality of consecutive periods of time further comprises identifying a minimum and a maximum of the plurality of metrics received from the system for the individual ones of the plurality of consecutive periods of time.

3. The method of claim 1, further comprising:

discarding, by the computing device, the plurality of metrics after aggregating the plurality of metrics to generate the at least one aggregate metric; and

storing, by the computing device, the at least one aggregate metric for a predefined period of time before discarding the at least one aggregate metric.

4. The method of claim 1, wherein the plurality of metrics represent a percent of utilization of a plurality of processor resources associated with the system.

5. The method of claim 1, wherein the plurality of metrics represent a traffic throughput of a plurality of network interfaces associated with the system.

6. The method of claim 1, wherein the plurality of metrics represent a number of bytes that are written to or read from a storage device in association with the system.

7. The method of claim 1, wherein the plurality of metrics represent a response latency, a request count, or a number of instances of a healthy or unhealthy host associated with the system.

8. A non-transitory computer-readable medium embodying a program executable in at least one computing device, comprising code that:

maintains a list of a plurality of metrics in an active metric list describing a state of a system, the plurality of metrics being associated with an active metric status indicating that a respective one of the plurality of metrics has been stored in a memory for less than a predefined period of time;

generates a plurality of data models from the plurality of metrics in the active metric list, individual ones of the plurality of metrics being associated with at least one of a plurality of consecutive periods of time;

aggregates the plurality of metrics associated with individual ones of the plurality of consecutive periods of time, thereby generating at least one aggregate metric for the individual ones of the plurality of consecutive periods of time; and

sends the at least one aggregate metric to at least one requesting computing device in response to a request received from the at least one requesting computing device for the plurality of data models.

9. The non-transitory computer-readable medium of claim 8, wherein the plurality of metrics represent a response latency, a request count, or a number of instances of a healthy or unhealthy host.

10. The non-transitory computer-readable medium of claim 8, wherein the program further comprises code that discards the plurality of data models after a predefined period of time.

11. The non-transitory computer-readable medium of claim 8, wherein the program further comprises code that generates a directory of the plurality of metrics to be sent to the at least one requesting computing device.

12. The non-transitory computer-readable medium of claim 8, wherein the program further comprises code that generates an output that indicates a performance of the system according to at least one of the plurality of data models.

13. The non-transitory computer-readable medium of claim 8, wherein the program further comprises code that updates a stored one of the plurality of data models with at least one of the plurality of metrics that is received out of a temporal order.

14. A system, comprising:

at least one computing device; and

a data modeling service executed in the at least one computing device, the data modeling service comprising logic that;

accesses a list of a plurality of metrics in an active metric list describing a state of a monitored system within at least one of a plurality of consecutive periods of time, wherein individual ones of the plurality of metrics correspond to a status indicating that a respective one of the plurality of metrics has been stored in the active metric list for less than a predefined period of time;

generates a plurality of data models from the plurality of metrics in the active metric list, wherein individual ones of the plurality of metrics are associated with a timestamp falling within the at least one of the plurality of consecutive periods of time; and

sends the plurality of data models in response to a request received from at least one requesting computing device, wherein individual ones of the plurality of data models represent the plurality of metrics associated with the at least one of the plurality of consecutive periods of time.

15. The system of claim 14, wherein the request is received from the at least one requesting computing device via an application programming interface (API).

16. The system of claim 14, wherein the data modeling service further comprises logic that updates a previously stored one of the plurality of data models with an individual one of the plurality of metrics that is received out of a temporal order.

17. The system of claim 14, wherein at least one of the plurality of data models further comprises a count of the plurality of metrics associated with a respective one of the plurality of consecutive periods of time.

18. The system of claim 14, wherein the data modeling service further comprises logic that stores the individual ones of the plurality of data models in a memory accessible to the at least one computing device for a predefined period of time before discarding the plurality of data models.

19. The system of claim 14, wherein the data modeling service further comprises logic that generates an output that indicates a past performance of the monitored system based at least in part on the plurality of data models.

20. The system of claim 14, wherein the data modeling service further comprises logic that aggregates the plurality of metrics associated with individual ones of the plurality of consecutive periods of time, thereby generating at least one aggregate metric for the individual ones of the plurality of consecutive periods of time.

21. The system of claim 20, wherein the data modeling service further comprises logic that sends the at least one aggregate metric in response to the request received from the at least one requesting computing device.

22. The system of claim 20, wherein a degree of aggregation of the plurality of metrics for the individual ones of the plurality of consecutive periods of time results in the at least one aggregate metric that enables monitoring of a performance of the monitored system with respect to at least one predefined operational threshold.