Storage of mass data for monitoring

• US 9,563,531 B2
• Filed: 08/12/2014
• Issued: 02/07/2017
• Est. Priority Date: 10/20/2008
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Osiecki, Daniel L., Sarma, Prashant L., Vanderbilt, Monty, Azari, David R., Schmidt, Caitlyn R.
• Primary Examiner(s): Manoskey, Joseph D
• Application Number:
  US14/457,300
• Publication Number:
  US 20140365650A1
• Time in Patent Office: 910 Days
• Field of Search: 714/45, 714/25, 714/26, 714/32, 714/33, 714/37, 714/38.1, 714/47.1, 714/47.2, 714/48, 707/957
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3072 where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06F 11/34 Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
• CPC Class Codes: G06F 16/2458 Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
• CPC Class Codes: H04L 43/02 Capturing of monitoring data
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

Disclosed are various in various embodiments are systems and methods providing for storage of mass data such as metrics. A plurality of data models are generated in the server from a stream of metrics describing a state of a system. Each of the metrics is associated with one of a plurality of consecutive periods of time, and each data model represents the metrics associated with a corresponding one of the consecutive periods of time. The data models are stored in a data store and each of the metrics is discarded after use in generating at least one of the data models.

Claims

1. A system, comprising:

at least one computing device; and

program instructions executable in the at least one computing device that, when executed, cause the at least one computing device to;

maintain a plurality of metrics obtained over a network in a memory device for a predefined amount of time, the plurality of metrics describing a state of a monitored system for at least one of a plurality of periods of time;

generate a data model comprising an aggregation of the plurality of metrics that consumes less storage space in the memory device than the plurality of metrics, the data model being generated for the at least one of the plurality of periods of time utilizing individual ones of the plurality of metrics in a list having a timestamp falling within the at least one of the plurality of periods of time;

remove the individual ones of the plurality of metrics utilized in generating the data model from the memory device after the data model has been generated;

generate an output describing the state of the monitored system, the output being generated based at least in part on the data model; and

send the output describing the monitored system to a requesting service.

2. The system of claim 1, further comprising program instructions that, when executed, causes the at least one computing device to discard the data model after storage for a predefined period of time.

3. The system of claim 2, wherein the predefined period of time is determined based at least in part on an available amount of storage space, a quantity of data represented by the data model, a determination of whether additional data needs to be stored, or a combination thereof.

4. The system of claim 1, wherein the list is an active metric list comprising a plurality of active metrics, wherein the plurality of active metrics comprise a subset of the plurality of metrics having been stored for less than a predefined period of time.

5. The system of claim 4, further comprising program instructions that, when executed, causes the at least one computing device to remove at least one of the plurality of metrics in the subset from the active metric list responsive to the at least one of the plurality of metrics having been stored longer than the predefined period of time.

6. The system of claim 1, wherein the monitored system is a server computing device or a web service.

7. A method, comprising:

maintaining, by at least one computing device, a list of a plurality of metrics obtained over a network describing a state of a monitored system for at least one of a plurality of periods of time;

generating, by the at least one computing device, a data model comprising an aggregation of the plurality of metrics that consumes less storage space in a memory device than the plurality of metrics, the data model being generated for the at least one of the plurality of periods of time utilizing individual ones of the plurality of metrics in the list having a timestamp falling within the at least one of the plurality of periods of time; and

disregarding, by the at least one computing device, the individual ones of the plurality of metrics utilized in generating the data model after the data model has been generated.

8. The method of claim 7, further comprising:

generating, by the at least one computing device, an output describing the monitored system, the output being generated based at least in part on the data model; and

sending, by the at least one computing device, the output describing the monitored system to a service in response to a request received from the service.

9. The method of claim 8, wherein the request is received from the service via an application programming interface (API).

10. The method of claim 7, further comprising updating, by the at least one computing device, the data model after generation with an individual one of the plurality of metrics that is received out of a temporal order.

11. The method of claim 7, wherein the plurality of metrics represent a percent of utilization of a plurality of processor resources associated with the monitored system;

a traffic throughput of a plurality of network interfaces associated with the monitored system;

a number of bytes that are written to or read from a storage device in association with the monitored system;

or a response latency, a request count, or a number of instances of a healthy or unhealthy host associated with the monitored system.

12. The method of claim 7, further comprising removing, by the at least one computing device, the data model after a predefined period of time, the predefined period of time being determined based at least in part on an available amount of storage space, a quantity of data represented by the data model, a determination of whether additional data needs to be stored, or a combination thereof.

13. The method of claim 7, wherein the list is an active metric list comprising a plurality of active metrics, wherein the plurality of active metrics comprise a subset of the plurality of metrics having been stored for less than a predefined period of time.

14. The method of claim 13, further comprising removing, by the at least one computing device, at least one of the plurality of metrics in the subset from the active metric list responsive to the at least one of the plurality of metrics having been stored longer than the predefined period of time.

15. A non-transitory computer-readable medium embodying program code executable in at least one computing device that, when executed, causes the at least one computing device to:

receive a request over a network from a service for an output describing a performance of a monitored system;

access a list of a plurality of metrics describing the performance of the monitored system for at least one of a plurality of periods of time;

generate an aggregate metric describing an aggregation of the plurality of metrics that consumes less storage space in a memory device than the plurality of metrics, the aggregate metric being generated for the at least one of the plurality of periods of time utilizing individual ones of the plurality of metrics in the list having a timestamp falling within the at least one of the plurality of periods of time;

disregard the individual ones of the plurality of metrics utilized in generating the aggregate metric after the aggregate metric has been generated; and

generate the output describing the performance of the monitored system utilizing the aggregate metric for transmission to the service.

16. The non-transitory computer-readable medium of claim 15, further comprising program code that, when executed, causes the at least one computing device to update the aggregate metric after generation with an individual one of the plurality of metrics that is received out of a temporal order.

17. The non-transitory computer-readable medium of claim 15, wherein the plurality of metrics represent a percent of utilization of a plurality of processor resources associated with the monitored system;

a traffic throughput of a plurality of network interfaces associated with the monitored system;

a number of bytes that are written to or read from a storage device in association with the monitored system;

or a response latency, a request count, or a number of instances of a healthy or unhealthy host associated with the monitored system.

18. The non-transitory computer-readable medium of claim 15, further comprising program code that, when executed, causes the at least one computing device to remove the aggregate metric after a predefined period of time, the predefined period of time being determined based at least in part on an available amount of storage space, a quantity of data represented by the aggregate metric, a determination of whether additional data needs to be stored, or a combination thereof.

19. The non-transitory computer-readable medium of claim 15, wherein the list is an active metric list comprising a plurality of active metrics, wherein the plurality of active metrics comprise a subset of the plurality of metrics having been stored for less than a predefined period of time.

20. The non-transitory computer-readable medium of claim 19, further comprising program code that, when executed, causes the at least one computing device to remove at least one of the plurality of metrics in the subset from the active metric list responsive to the at least one of the plurality of metrics having been stored longer than the predefined period of time.