EFFICIENT PROCESSING OF TIME SERIES DATA

• US 20100281075A1
• Filed: 07/16/2010
• Published: 11/04/2010
• Est. Priority Date: 01/12/2005
• Status: Active Grant

• Current Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Original Assignee: Computer Associates Think Inc. (Broadcom, Inc.)
• Inventors: Bley, John B.
• Granted Patent: US 8,321,479 B2
• Time in Patent Office: Days
• US Class Current: 707/809
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 16/2474 Sequence data queries, e.g. querying versioned data
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database

Abstract

A system receiving time series data will store the data in a first format suited for quick writing. That data will subsequently be converted from the first format to a second format better suited than the first format for compact storage and queries. Over time, the data in the second format can be aggregated and/or down sampled. In one embodiment, the first format includes a set of blocks of information, with each block of information storing data for a set of metrics during a time period. The second format includes a header, a set of blocks of data and a footer. Each bock of data in the second format stores information for a metric over multiple time periods. The footer includes pointers to each block of data.

Claims

1. A method for processing data, comprising:

receiving time series measurement data about a plurality of application metrics over a plurality of time periods, wherein each application metric measures the performance of a component of a software application;

writing said received time series measurement data to a first file in a spool format, the first file having blocks of data each including a header indicating a begin time for the time period associated with the block of data and storing all time series measurement data received within the time period associated with the block of data; and

converting the first file into a second file having a header and blocks of data containing said time series measurement data in a second format suited for compact storage, wherein each block of data stores only time series measurement data for a single application metric, the header indicating the time period between each instance of time series measurement data stored in the block of data.

2. A method according to claim 1, wherein:

said spool format stores data in blocks such that each block is associated with a different time period.

3. A method according to claim 2, wherein:

each block of data in the second file stores a set of records associated with each time series measurement data, each record corresponding to different time periods; and

each record is compressed individually.

4. A method according to claim 1, wherein:

said second file a footer, said footer includes pointers to each block of data.

5. A method according to claim 1, wherein:

said writing includes writing a first file during a first time period; and

said converting includes writing a second file after said first time period.

7. A method according to claim 1, wherein:

said steps of receiving, writing and converting are performed by application performance analysis software.

8. A method according to claim 7, wherein said application performance analysis software further performs the step of:

adding probes and an Agent to a software application, said Agent receives data from said probes, said time series management data is received from said Agent.

9. A method according to claim 1, wherein:

said writing includes creating a set of files in said spool format;

said converting includes creating a set of files in said second format, each of said set of files in said second format corresponds to a different time period; and

said method further comprises aggregating said set of files in said second format and down sampling said set of files.

10. A method according to claim 9, wherein:

said receiving and writing are performed continuously;

said converting is performed every hour;

said aggregating is performed each day; and

said down sampling is performed after a week.

6. A method according to claim 6, further comprising:

receiving a first query of said time series measurement data during said first time period;

performing said first query of said time series measurement data using said first file during said first time period;

receiving a second query of said time series measurement data after said first time period; and

performing said second query of said time series measurement data in said second file.

11. A method for processing data, comprising:

receiving time series measurement data for one or more metrics, each metric measuring an activity of a component of a software application;

writing said time series measurement data to a first file in a spool format, wherein the first file stores time series measurement data in blocks of data, each block of data being associated with a different sample period; and

converting said time series measurement data from said spool format to a query format, wherein the query format includes blocks of data, each block of data includes time series measurement data for a single metric received during the plurality of sample periods.

12. A method according to claim 11, wherein:

each block of data in the query format includes a set of records each corresponding to different time periods; and

each record is compressed individually.

13. A method according to claim 11, wherein said writing includes writing a first file during a first time period and said converting includes writing a second file after said first time period, said method further comprising:

receiving a specific query of said time series measurement data during said first time period;

performing said specific query of said time series measurement data using said first file during said first time period;

receiving a particular query of said time series measurement data after said first time period; and

performing said particular query of said time series measurement data in said second file.

14. A method according to claim 13, wherein:

said steps of receiving, writing and converting are performed by application performance analysis software.

15. A method according to claim 14, wherein said application performance analysis software further performs the step of:

adding probes and an Agent to a software application, said Agent receives data from said probes, said time series management data is received from said Agent.

16. A method according to claim 11, wherein:

said converting includes creating a set of files in said query format, each of said set of files corresponds to a different time period; and

said method further comprises aggregating said set of files and down sampling said set of files.

17. A method for processing data about a system, comprising:

adding probes to a software application, each probe configured to acquire performance data about a metric associated with the software application;

acquiring performance data about said software application using said probes in real time;

sending said performance data to a data manager;

storing said performance data in one or more searchable files according to a first format, wherein the performance data in the first format is organized into blocks of data, each block of data includes all instances of performance data acquired during a sample period and, for each instance of performance data stored in the block of data, identifies the metric associated with the performance data; and

periodically converting a subset of said performance data stored in said first format to searchable files according to a second format, wherein the performance data in the searchable files in said second format is organized into blocks of data, each block of data stores all performance data acquired by the same probe together and identifies the metric associated with each instance of performance data at the beginning of the block of data.

18. A method according to claim 17, further comprising:

aggregating at least a subset of said searchable files according to said second format.

19. A method according to claim 18, further comprising:

down sampling said aggregated searchable files.

20. One or more processor readable storage devices storing processor readable code for programming one or more processors, said processor readable code comprising:

a writer, said writer receives data about a set of metrics associated with a software application and stores said data in a blocks of data in a first file in a first format, each block of data contains a beginning sample time and all data acquired from the beginning sample time through a sample period, wherein each instance of data stored in a block of data is associated with a metric; and

a converter, said converter accesses said data in said first format and converts said data from said first format to a second format in a second file having a header and blocks of data, the header includes a start time indicating the earliest sample time associated with the data stored in the blocks of data, a stop time indicating the latest sample time associated with the data stored in the blocks of data and a sample period for each instance of data stored in a block of data based on the beginning sample times contained in the blocks of data in the first format, each block of data stores all data about a single metric acquired between the start sample time and the stop sample time indicated in the header.

21. One or more processor readable storage devices according to claim 20, wherein said processor readable code further comprises:

an appender, said converter creates a set of data structures for a set of time periods, said appender aggregates said set of data structures to a single data structure.

22. One or more processor readable storage devices according to claim 20, wherein said processor readable code further comprises:

a down sampler, said down sampler creates down sampled data structures from original data structures in said second format.

23. One or more processor readable storage devices according to claim 20, wherein:

said second format further includes a footer, said footer includes pointers to each block of data in the second format.

24. One or more processor readable storage devices according to claim 23, wherein:

said information for a metric over multiple time periods includes a set of records that are individually compressed.