Efficient processing of time series data
First Claim
1. A method for processing data, comprising:
- receiving time series measurement data about a plurality of application metrics over a plurality of time periods, wherein each application metric measures the performance of a component of a software application;
writing said received time series measurement data to a first file in a spool format, the first file having blocks of data each including a header indicating a begin time for the time period associated with the block of data and storing all time series measurement data received within the time period associated with the block of data;
converting the first file into a second file having a header and blocks of data containing said time series measurement data in a second format suited for compact storage, wherein each block of data stores only time series measurement data for a single application metric, the header indicating the time period between each instance of time series measurement data stored in the block of data;
receiving a query for a particular application metric; and
searching the second file for time series measurement data for the particular application metric in response to the query.
3 Assignments
0 Petitions
Accused Products
Abstract
A system receiving time series data will store the data in a first format suited for quick writing. That data will subsequently be converted from the first format to a second format better suited than the first format for compact storage and queries. Over time, the data in the second format can be aggregated and/or down sampled. In one embodiment, the first format includes a set of blocks of information, with each block of information storing data for a set of metrics during a time period. The second format includes a header, a set of blocks of data and a footer. Each bock of data in the second format stores information for a metric over multiple time periods. The footer includes pointers to each block of data.
-
Citations
23 Claims
-
1. A method for processing data, comprising:
-
receiving time series measurement data about a plurality of application metrics over a plurality of time periods, wherein each application metric measures the performance of a component of a software application; writing said received time series measurement data to a first file in a spool format, the first file having blocks of data each including a header indicating a begin time for the time period associated with the block of data and storing all time series measurement data received within the time period associated with the block of data; converting the first file into a second file having a header and blocks of data containing said time series measurement data in a second format suited for compact storage, wherein each block of data stores only time series measurement data for a single application metric, the header indicating the time period between each instance of time series measurement data stored in the block of data; receiving a query for a particular application metric; and searching the second file for time series measurement data for the particular application metric in response to the query. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10)
-
-
11. A method for processing data, comprising:
-
receiving time series measurement data for one or more metrics, each metric measuring an activity of a component of a software application; writing said time series measurement data to a first file in a spool format, wherein the first file stores time series measurement data in blocks of data, each block of data being associated with a different sample period; and converting said time series measurement data from said spool format to a query format, wherein the query format includes a second file having blocks of data and a footer, each block of data includes time series measurement data for a single metric received during the plurality of sample periods, said footer including a record for each of said blocks in said query format, each record includes a field that identifies said single metric for one of said blocks in said query format, each record contains a pointer to said block for said single metric identified in said field. - View Dependent Claims (12, 13, 14, 15, 16)
-
-
17. A method for processing data about a system, comprising:
-
adding probes to a software application, each probe configured to acquire performance data about a metric associated with the software application, each metric having a metric name; acquiring performance data about said software application using said probes in real time; sending said performance data from a plurality of agents to a data manager, each agent having a unique name; storing said performance data in one or more searchable files according to a first format, wherein the performance data in the first format is organized into blocks of data, each block of data includes all instances of performance data acquired during a sample period and, for each instance of performance data stored in the block of data, identifies the metric and agent associated with the performance data; and periodically converting a subset of said performance data stored in said first format to searchable files according to a second format, wherein the performance data in the searchable files in said second format is organized into blocks of data, each block of data stores all performance data acquired by the agent for a particular metric and includes a unique combination of agent name and metric name to identify the metric and agent associated with each instance of performance data. - View Dependent Claims (18, 19)
-
-
20. One or more processor readable storage devices storing processor readable code for programming one or more processors, said processor readable code comprising:
-
a writer, said writer receives data about a set of metrics associated with a software application and stores said data in blocks of data in a first file in a first format, each block of data contains a beginning sample time and all data acquired from the beginning sample time through a sample period, wherein each instance of data stored in a block of data is associated with a metric; and a converter, said converter accesses said data in said first format and converts said data from said first format to a second format in a second file having a header and blocks of data, the header includes a start time indicating the earliest sample time associated with the data stored in the blocks of data, a stop time indicating the latest sample time associated with the data stored in the blocks of data and a sample period for each instance of data stored in a block of data based on the beginning sample times contained in the blocks of data in the first format, each block of data stores all data about a single metric acquired between the start sample time and the stop sample time indicated in the header, said second format further includes a footer, said footer includes pointers to each block of data in the second format, each pointer points to a block for a single metric. - View Dependent Claims (21, 22, 23)
-
Specification