Multitenant monitoring system storing monitoring data supporting flexible pivot querying

• US 9,104,392 B1
• Filed: 06/13/2013
• Issued: 08/11/2015
• Est. Priority Date: 06/13/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Dunagan, John David, Pai, Derek Solomon, Shim, Caitlyn Rose, Heitmann, John W., Foster, Michael R.
• Primary Examiner(s): Lee, Wilson
• Application Number:
  US13/917,595
• Time in Patent Office: 789 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24575 using context

Abstract

Monitoring data can be received and managed in a multi-tenant environment. Monitoring data can be received for multiple tenants. The received monitoring data can be sorted. Sorting can be performed based on various criteria, such as by tenant. The sorted monitoring data can be written to data files. Each data file can store monitoring data for a plurality of tenants. Receiving, sorting, and writing monitoring data can be performed within a multi-tenant monitoring system.

Claims

1. A method, implemented at least in part by a multi-tenant monitoring system comprising one or more computing devices, for storing monitoring data for use during pivot querying, the method comprising:

for each of a plurality of time periods each having a first time period length;

receiving, by the multi-tenant monitoring system, monitoring data for multiple tenants;

sorting, by the multi-tenant monitoring system, the received monitoring data, wherein the sorting comprises grouping the received monitoring data by each tenant of the multiple tenants; and

writing, by the multi-tenant monitoring system, the sorted monitoring data, wherein the sorted monitoring data is written to storage using a plurality of data files, and wherein each of the data files supports storing data for a plurality of the multiple tenants;

wherein the sorted monitoring data that is written to storage is not aggregated across dimensions.

2. The method of claim 1 wherein sorting the received monitoring data comprises:

further grouping the received monitoring data by at least one dimension within the grouped monitoring data of one or more of the multiple tenants.

3. The method of claim 1 wherein the monitoring data stored within the plurality of data files is not indexed at a level of individual observations.

4. The method of claim 1 wherein each time period of the plurality of time periods uses a new plurality of data files.

5. The method of claim 4 further comprising, after the receiving, sorting, and writing have been performed for each of the plurality of time periods:

for each of the plurality of time periods, reading the sorted monitoring data from the plurality of data files;

re-sorting the read monitoring data comprising combining sorted monitoring data to use fewer time periods having a second time period length longer than the first time period length; and

writing the re-sorted monitoring data, wherein the re-sorted monitoring data is written to storage using a plurality of data files each storing re-sorted monitoring data for the second time period, and wherein each data file, of the plurality of data files storing re-sorted monitoring data, supports storing grouped data for a plurality of the multiple tenants.

6. The method of claim 1 wherein the received monitoring data is unsorted.

7. The method of claim 1 wherein the multi-tenant monitoring system is configured to read the stored monitoring data, aggregate the read data monitoring data, and perform pivot queries using the aggregated monitoring data.

8. The method of claim 1 wherein receiving the monitoring data comprises;

buffering the monitoring data in random access memory (RAM) for a period of time, wherein the period of time is based at least in part upon an amount of available RAM.

9. The method of claim 1 further comprising:

scaling out the multi-tenant monitoring system comprising partitioning the multiple tenants among a plurality of groups of computing resources.

10. A monitoring system comprising one or more computing devices for storing data for use during pivot querying, the monitoring system comprising:

one or more systems comprising processing units, memory, and file storage;

the one or more systems configured to perform operations for storing data for use during pivot querying, the operations comprising;

receiving data associated with multiple sources;

sorting the received data; and

writing the sorted data, wherein the sorted data is written to storage using a plurality of data files, and wherein each of the data files supports storing data for a plurality of the multiple sources;

wherein the sorted data that is written to storage is not aggregated across dimensions.

11. The monitoring system of claim 10 wherein sorting the received data comprises:

grouping the received data by each source of the multiple sources.

12. The monitoring system of claim 10 wherein individual data points of the data stored within the plurality of data files are not indexed.

13. The monitoring system of claim 10 wherein the receiving, the sorting, and the writing are performed for each of a plurality of time periods, wherein each time period of the plurality of time periods is of a first time period length, and wherein each time period of the plurality of time periods uses a new plurality of data files.

14. The monitoring system of claim 13 further comprising, after the receiving, sorting, and writing have been performed for each of the plurality of time periods:

for each of the plurality of time periods, reading the sorted data from the plurality of data files;

re-sorting the read data comprising combining sorted data to use fewer time periods having a second time period length longer than the first time period length;

writing the re-sorted data, wherein the re-sorted data is written to storage using a plurality of data files each storing re-sorted data for the second time period, and wherein each data file, of the plurality of data files storing re-sorted data, supports storing grouped data for a plurality of the multiple sources.

15. The monitoring system of claim 10 wherein receiving the data comprises;

buffering the data in random access memory (RAM) for a period of time, wherein the period of time is based at least in part upon an amount of available RAM.

16. The monitoring system of claim 10 further comprising:

scaling out the monitoring system comprising partitioning the multiple sources among a plurality of groups of computing resources.

17. A computer-readable storage medium storing computer-executable instructions for causing one or more computing devices to perform a method for storing monitoring data for use during pivot querying, the method comprising:

receiving monitoring data for multiple tenants;

sorting the received monitoring data, wherein the sorting comprises grouping the received monitoring data by each tenant of the multiple tenants; and

writing the sorted monitoring data, wherein the sorted monitoring data is written to storage using a plurality of data files associated with the time period, and wherein each of the data files supports storing data for a plurality of the multiple tenants;

wherein the sorted monitoring data that is written to storage is not aggregated across dimensions.

18. The computer-readable storage medium of claim 17 wherein the monitoring data stored within the plurality of data files is not indexed at a level of individual observations.

19. The computer-readable storage medium of claim 17 wherein the receiving, the sorting, and the writing are performed for each of a plurality of time periods, wherein each time period of the plurality of time periods is of a first time period length, and wherein each time period of the plurality of time periods uses a new plurality of data files.

20. The computer-readable storage medium of claim 19, the method further comprising, after the receiving, sorting, and writing have been performed for each of the plurality of time periods:

for each of the plurality of time periods, reading the sorted monitoring data from the plurality of data files associated with the time period;

re-sorting the read monitoring data comprising combining sorted monitoring data to use fewer time periods having a second time period length longer than the first time period length; and

writing the re-sorted monitoring data, wherein the re-sorted monitoring data is written to storage using a plurality of data files each storing re-sorted monitoring data for the second time period, and wherein each data file, of the plurality of data files storing re-sorted monitoring data, supports storing grouped data for a plurality of the multiple tenants.

21. The computer-readable storage medium of claim 17 wherein receiving the monitoring data comprises;

buffering the monitoring data in random access memory (RAM) for a period of time, wherein the period of time is based at least in part upon an amount of available RAM.