Flexible pivot querying of monitoring data with zero setup

• US 9,817,864 B1
• Filed: 06/13/2013
• Issued: 11/14/2017
• Est. Priority Date: 06/13/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Dunagan, John David, Pai, Derek Solomon, Shim, Caitlyn Rose, Heitmann, John W., Foster, Michael R.
• Primary Examiner(s): Beausoliel, Jr., Robert
• Assistant Examiner(s): Rayyan, Susan F
• Application Number:
  US13/917,586
• Time in Patent Office: 1,615 Days
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/2453 Query optimisation

Abstract

Monitoring data can be sent, received, stored, and accessed with zero setup. Customers can send monitoring data to be stored without the customers having to perform any setup ahead-of-time, such as arbitrary lists of key-value pairs. Customers can perform zero setup pivot queries using stored monitoring data. For example, users can perform pivot queries using any available stored information, such as available metric types and dimensions, without any setup needed ahead-of-time.

Claims

1. A method, implemented by a multi-tenant monitoring system comprising one or more computing devices, for performing a zero setup pivot query, the method comprising:

determining, by the one or more computing devices, available metric types and dimensions associated with monitoring data stored in data files, wherein the monitoring data stored in the data files is un-aggregated and is not aggregated across any dimensions;

receiving, from a customer by the one or more computing devices, a pivot query, wherein the pivot query indicates one or more metric types and one or more dimensions, wherein the one or more metric types and one or more dimensions are defined at query time, and wherein the one or more metric types and one or more dimensions are based, at query time, at least in part upon the available metric types and dimensions associated with the monitoring data;

performing, by the one or more computing devices, the pivot query using the monitoring data stored, in an un-aggregated format, in the data files according to the one or more metric types and the one or more dimensions; and

outputting results from performing the pivot query;

wherein the monitoring data is associated with the customer of the multi-tenant monitoring system, and wherein the monitoring data is grouped together by the customer within the data files; and

wherein each of the data files supports storing monitoring data for a plurality of customers of the multi-tenant monitoring system, and wherein the monitoring data stored within the data files is not indexed at a level of individual observations other than an indication of which data file contains a given customer'"'"'s data.

2. The method of claim 1 wherein determining the available metric types and dimensions associated with monitoring data comprises:

reading the monitoring data for one or more time periods; and

accumulating metric types and dimensions that are available in the read monitoring data.

3. The method of claim 2 further comprising:

presenting, to the customer, an indication of at least a portion of the available metric types and dimensions that have been accumulated.

4. The method of claim 1 wherein performing the pivot query comprises:

reading, by the one or more computing devices, the monitoring data from the data files; and

aggregating the monitoring data read from the one or more data files according to the pivot query.

5. The method of claim 1 wherein performing the pivot query comprises:

for each time period of a plurality of time periods;

reading, by the one or more computing devices, a portion of the monitoring data corresponding to the time period from one or more of the data files associated with the time period.

6. The method of claim 1 wherein the pivot query is performed in linear time.

7. The method of claim 1 wherein the monitoring data stored in the data files is stored as a list of key-value pairs.

8. The method of claim 1 wherein no indexing is performed for the monitoring data stored within the data files other than the indication of which data file contains a given customer'"'"'s data.

9. A multi-tenant monitoring system comprising one or more computing devices for performing a zero setup pivot query, the multi-tenant monitoring system comprising:

one or more systems comprising processing units, memory, and file storage;

the one or more systems configured to perform operations for performing a zero setup pivot query, the operations comprising;

determining available metric types and dimensions associated with monitoring data stored in data files, wherein the monitoring data stored in the data files is un-aggregated and is not aggregated across any dimensions;

receiving a pivot query, wherein the pivot query indicates one or more metric types and one or more dimensions, wherein the one or more metric types and one or more dimensions are defined at query time, and wherein the one or more metric types and one or more dimensions are based at least in part upon the available metric types and dimensions associated with the monitoring data;

performing the pivot query using the monitoring data stored, in an un-aggregated format, in the data files according to the one or more metric types and the one or more dimensions; and

outputting results from performing the pivot query;

wherein the monitoring data is associated with a customer of the multi-tenant monitoring system, and wherein the monitoring data is grouped together by the customer within the data files; and

wherein each of the data files supports storing monitoring data for a plurality of customers of the multi-tenant monitoring system, and wherein the monitoring data stored within the data files is not indexed at a level of individual observations.

10. The multi-tenant monitoring system of claim 9 wherein determining the available metric types and dimensions associated with monitoring data comprises:

reading the monitoring data for one or more time periods; and

accumulating metric types and dimensions that are available in the read monitoring data.

11. The multi-tenant monitoring system of claim 10 the operations further comprising:

presenting, to the customer, an indication of at least a portion of the available metric types and dimensions that have been accumulated.

12. The multi-tenant monitoring system of claim 9 wherein performing the pivot query comprises:

reading the monitoring data from the data files; and

aggregating the monitoring data read from the one or more data files according to the pivot query.

13. The multi-tenant monitoring system of claim 9 wherein performing the pivot query comprises:

for each time period of a plurality of time periods;

reading a portion of the monitoring data corresponding to the time period from one or more of the data files associated with the time period.

14. The multi-tenant monitoring system of claim 9 wherein the one or more systems are further configured to perform operations for receiving monitoring data supporting zero setup pivot queries, the operations comprising:

receiving, from one or more computing devices, monitoring data comprising key-value pairs; and

storing the received monitoring data in the data files.

15. The multi-tenant monitoring system of claim 14 wherein the received monitoring data is stored in the data files as lists of key-value pairs without performing any aggregation across any dimensions and not according to a schema.

16. The multi-tenant monitoring system of claim 9 wherein no indexing is performed for the monitoring data stored within the data files other than the indication of which data file contains a given customer'"'"'s data.

17. A computer-readable storage medium storing computer-executable instructions for causing one or more computing devices to perform a method supporting zero setup pivot queries using monitoring data stored within a multi-tenant monitoring system, the method comprising:

determining available metric types and dimensions associated with monitoring data stored in data files, wherein the monitoring data stored in the data files is un-aggregated and is not aggregated across any dimensions;

receiving a pivot query, wherein the pivot query indicates one or more metric types and one or more dimensions, wherein the one or more metric types and one or more dimensions are defined at query time, and wherein the one or more metric types and one or more dimensions are based at least in part upon the available metric types and dimensions associated with the monitoring data;

performing the pivot query using the monitoring data stored, in an un-aggregated format, in the data files according to the one or more metric types and the one or more dimensions; and

outputting results from performing the pivot query;

wherein the monitoring data is associated with a customer of the multi-tenant monitoring system, and wherein the monitoring data is grouped together by the customer within the data files; and

wherein each of the data files supports storing monitoring data for a plurality of customers of the multi-tenant monitoring system, and wherein the monitoring data stored within the data files is not indexed at a level of individual observations.

18. The computer-readable storage medium of claim 17 wherein determining the available metric types and dimensions associated with monitoring data comprises:

reading the monitoring data for one or more time periods;

accumulating metric types and dimensions that are available in the read monitoring data; and

presenting, to the customer, an indication of at least a portion of the available metric types and dimensions that have been accumulated.

19. The computer-readable storage medium of claim 17 the method further comprising:

receiving, from one or more computing devices, monitoring data comprising key-value pairs; and

storing the received monitoring data in the data files.

20. The computer-readable storage medium of claim 19 wherein the received monitoring data is stored in the data files as lists of key-value pairs without performing any aggregation across any dimensions and not according to a schema.