Method and system for collecting and analyzing time-series data

• US 9,990,385 B2
• Filed: 05/18/2015
• Issued: 06/05/2018
• Est. Priority Date: 03/14/2006
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Nordstrom, Paul G., Thompson, Aaron C.
• Primary Examiner(s): SNYDER, STEVEN G
• Application Number:
  US14/715,453
• Publication Number:
  US 20150248446A1
• Time in Patent Office: 1,114 Days
• Field of Search: None
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 11/1448 Management of the data involved in backup or backup restore
• CPC Class Codes: G06F 11/30 Monitoring
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/301 where the computing system is a virtual computing platform, e.g. logically partitioned systems virtual machines G06F9/45533; logical partitioning of resources G06F9/5077
• CPC Class Codes: G06F 11/3058 Monitoring arrangements for monitoring environmental properties or parameters of the computing system or of the computing system component, e.g. monitoring of power, currents, temperature, humidity, position, vibrations thermal management in cooling arrangements of a computing system G06F1/206
• CPC Class Codes: G06F 11/3442 for planning or managing the needed capacity
• CPC Class Codes: G06F 11/3447 Performance evaluation by modeling
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 2201/835 Timestamp
• CPC Class Codes: G06F 2201/875 Monitoring of systems including the internet

Abstract

A computer-implemented data collection and analysis method comprises receiving from a client computing device, at least one data analysis datapoint, adding the received at least one data analysis datapoints to a data structure, and adding an additional data analysis datapoint to the data structure, storing, in a data repository, the data structure further comprising a data processing result, maintaining at least partially consistent copies of the data structure across the plurality of host computers, in response to a request regarding an operation from the client computing device, retrieving the data processing result from the data repository, and providing the data processing result using routing information to the client computing device. The data structure comprises an index of data analysis datapoints from the plurality of host computing devices. A portion of the additional data analysis datapoint is added to the data value to generate a data processing result.

Claims

1. A computer-implemented data collection and analysis method comprising:

receiving, at a host computing device, at least one data analysis datapoint from a client computing device,

wherein a data analysis datapoint comprises a data value corresponding to data to be processed and a data interval corresponding to a time interval associated with the data analysis datapoint,

wherein the host computing device is one of a plurality of host computing devices configured to collect data from a plurality of client computing devices;

adding the received at least one data analysis datapoint to a data structure, wherein the data structure comprises an index of data analysis datapoints, routing information for at least a portion of the data structure from one of the plurality of host computing devices to a requesting client computing device, and a set of calculation descriptors indicating data for performing data processing to obtain a data processing result;

storing, in a data repository, the data structure;

maintaining at least partially consistent copies of the data structure across the plurality of host computing devices;

receiving an additional data analysis datapoint from the client computing device;

generating the data processing result based at least in part on the data analysis datapoints and the set of calculation descriptors included in the data structure, the data processing result corresponding at least in part to adding the additional data analysis datapoint to the data structure, wherein the additional data analysis datapoint is to be used in connection with one of the data analysis datapoints in the index, wherein a portion of the additional data analysis datapoint is added to the data value; and

in response to a request regarding an operation from the client computing device, providing the data processing result using the routing information to the client computing device.

2. The method of claim 1, wherein the received at least one data analysis datapoint is added to the data structure in the form of a plurality of calculation objects.

3. The method of claim 1, wherein the index of data analysis datapoints comprises a multi-dimensional index that indexes characteristics of a plurality of calculation objects.

4. The method of claim 3, wherein the characteristics of calculation objects comprise at least one of the data value, the data interval, a session identifier, or an identifier associated with a text string.

5. The method of claim 1 further comprising initiating processes on the plurality of host computers, wherein individual processes are initiated responsive to the addition of an additional data analysis datapoint to the data structure.

6. The method of claim 1 further comprising providing a library of process functions for generating the data processing result, wherein the process functions comprise at least one of a combination function, an addition function, or a publication function.

7. The method of claim 6 further comprising:

receiving a user selection of one of the process functions; and

processing the selected process function to generate a user view comprising data generated from the received at least one data analysis datapoint or the additional data analysis datapoint.

8. The method of claim 1, wherein providing the data processing result using the routing information to the client computing device comprises providing the data processing result on a web page to the client computing device.

9. The method of claim 1 further comprising providing a graphical user interface accessible to users to specify the at least one data analysis datapoint or the additional data analysis datapoint.

10. The method of claim 9, wherein the user interface is a web-based interface.

11. The method of claim 2, wherein a graphical user interface is configured to provide an interface to receive information corresponding to a calculation object of the plurality of calculation objects.

12. The method of claim 1 further comprising:

decomposing the operation into operation components; and

allocating the operation components across the plurality of host computing devices.

13. A system for collecting and analyzing data comprising:

one or more computer processors;

at least one computer memory accessible by at least one of the one or more computer processors;

a server computing device configured to execute one or more processes, the server computing device operable to;

receive, from a client computing device, at least one data analysis datapoint,

wherein a data analysis datapoint comprises a data value for processing,

wherein the server computing device is one of a plurality of host computing devices configured to collect data from a plurality of client computing devices;

aggregate the received at least one data analysis datapoint in a data structure,

wherein the data structure comprises an index of data analysis datapoints, routing information for at least a portion of the data structure from one of the plurality of host computing devices to a requesting client computing device, and a set of calculation descriptors indicating data for performing data processing to obtain a data processing result;

receive, at the server computing device, an additional data analysis datapoint;

store, in a data repository, the data structure;

maintain at least partially consistent copies of the data structure across the plurality of host computing devices;

generate the data processing result based at least in part on the at least one data analysis datapoint and the set of calculation descriptors included in the data structure, the data processing result corresponding at least in part to combining the additional data analysis datapoint within the data structure, wherein the additional data analysis datapoint is to be used in connection with one of the data analysis datapoints in the index of data analysis datapoints, wherein a portion of the additional data analysis datapoint is to be combined with the data value; and

in response to a request regarding an operation from the client computing device, provide the data processing result using the routing information to the client computing device.

14. The system of claim 13, wherein aggregating the received at least one data analysis datapoint in the data structure comprises aggregating the received at least one data analysis datapoint in the data structure as a plurality of calculation objects.

15. The system of claim 13, wherein providing the data processing result using the routing information to the client computing device comprises providing the data processing result to the client computing device using at least one of a web page or a graphical view.

16. The system of claim 13, wherein the server computing device is further operable to:

provide a user interface for specifying the at least one data analysis datapoint or the additional data analysis datapoint.

17. The system of claim 13, wherein the received at least one data analysis datapoint are indexed in the data structure as received from the client computing device.

18. A non-transitory computer readable medium storing instructions for data analysis and collection which, when executed, cause a processor to at least:

receive, at a host computing device, at least one data analysis datapoint from a client computing device,

wherein a data analysis datapoint comprises a data value for processing,

wherein the host computing device is one of a plurality of host computing devices configured to collect data from a plurality of client computing devices;

aggregate the received at least one data analysis datapoint in a data structure, wherein the data structure comprises an index of data analysis datapoints routing information for at least a portion of the data structure from one of the plurality of host computing devices to a requesting client computing device, and a set of calculation descriptors indicating data for performing data processing to obtain a data processing result;

maintain at least partially consistent copies of the data structure across the plurality of host computing devices;

receive, at the host computing device, an additional data analysis datapoint;

generate the data processing result based at least in part on the at least one data analysis datapoint and the set of calculation descriptors included in the data structure, the data processing result corresponding at least in part to combining the additional data analysis datapoint within the data structure, wherein the additional data analysis datapoint is to be used in connection with one of the received at least one data analysis datapoint in the data structure, wherein a portion of the additional data analysis datapoint is to be combined with the data value; and

in response to a request regarding an operation from the client computing device, provide the data processing result using the routing information to the client computing device.

19. The non-transitory computer readable medium of claim 18, wherein aggregating the received at least one data analysis datapoint in a data structure comprises aggregating the received at least one data analysis datapoint in the data structure as a plurality of objects.

20. The non-transitory computer readable medium of claim 18, wherein the instructions further cause the processor to at least:

provide a library of processes for generating the data processing result regarding at least one of the received data analysis datapoints or the additional data analysis datapoint.

21. The non-transitory computer readable medium of claim 20, wherein the processes for generating the data processing result comprise at least one of a combination process, an addition process, and a web page publication process.