Efficient processing of time series data

• US 20060173878A1
• Filed: 01/12/2005
• Published: 08/03/2006
• Est. Priority Date: 01/12/2005
• Status: Active Grant

• Current Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Original Assignee: Wily Technology, Inc. (Broadcom, Inc.)
• Inventors: Bley, John B.
• Granted Patent: US 7,783,679 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 16/2474 Sequence data queries, e.g. querying versioned data
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database

Abstract

A system receiving time series data will store the data in a first format suited for quick writing. That data will subsequently be converted from the first format to a second format better suited than the first format for compact storage and queries. Over time, the data in the second format can be aggregated and/or down sampled. In one embodiment, the first format includes a set of blocks of information, with each block of information storing data for a set of metrics during a time period. The second format includes a header, a set of blocks of data and a footer. Each bock of data in the second format stores information for a metric over multiple time periods. The footer includes pointers to each block of data.

Claims

1. A method for processing data, comprising:

receiving time series measurement data;

writing said time series measurement data to a first format suited for fast writing; and

converting said time series measurement data from said first format to a second format suited for compact storage.

2. A method according to claim 1, wherein:

said first format stores data in blocks such that each block is associated with a different time period; and

said second format stores data in blocks of data for individual metrics.

3. A method A method according to claim 2, wherein:

each block of data includes a set of records each corresponding to different time periods; and

each record is compressed individually.

4. A method according to claim 1, wherein:

said first format includes a set of blocks of information, each block of information stores for data for a set of metrics during a respective time period; and

said second format includes a header and a set of blocks of data and a footer, each bock of data stores data for a metric over multiple time periods, said footer includes pointers to each block of data

5. A method according to claim 4, wherein said converting comprises:

(a) reading buffers of information from said blocks of information;

(b) preparing said header based on said buffers;

(c) finding a lowest metric ID in said buffers;

(d) writing data to a block of data based on information regarding said metric ID stored in said blocks of information corresponding to said buffers of information; and

(e) repeating steps (c) and (d) for additional metric IDs.

6. A method according to claim 1, wherein:

said writing includes writing a first file during a first time period; and

said converting includes writing a second file after said first time period.

7. A method according to claim 6, further comprising:

receiving a first query of said time series measurement data during said first time period;

performing said first query of said time series measurement data using said first file during said first time period;

receiving a second query of said time series measurement data after said first time period; and

performing said second query of said time series measurement data in said second file.

8. A method according to claim 7, wherein:

said second file has a header, a set of blocks of data and a footer;

each bock of data stores information for a metric over multiple time intervals;

said footer includes pointers to each block; and

said performing said second query includes reading said header, reading said footer, following a pointer to corresponding block, reading the corresponding block and reporting based on information in said corresponding block.

9. A method according to claim 1, wherein:

said steps of receiving, writing and converting are performed by application performance analysis software.

10. A method according to claim 9, wherein said application performance analysis software further performs the step of:

adding probes and an Agent to a software application, said Agent receives data from said probes, said time series management data is received from said Agent.

11. A method according to claim 1, wherein:

said writing includes creating a set of files in said first format;

said converting includes creating a set of files in said second format, each of said set of files in said second format corresponds to a different time period; and

said method further comprises aggregating said set of files in said second format and down sampling said set of files.

12. A method according to claim 11, wherein:

said receiving and writing are performed continuously;

said converting is performed every hour;

said aggregating is performed each day; and

said down sampling is performed after a week.

13. A method for processing data, comprising:

receiving time series measurement data;

writing said time series measurement data to a first format suited for fast writing; and

converting said time series measurement data from said first format to a second format suited for performing queries faster than said first format.

14. A method according to claim 13, wherein:

said first format stores data in blocks such that each block is associated with a different time period; and

said second format stores data in blocks of data for individual metrics.

15. A method A method according to claim 13, wherein:

each block of data includes a set of records each corresponding to different time periods; and

each record is compressed individually.

16. A method according to claim 13, wherein said writing includes writing a first file during a first time period and said converting includes writing a second file after said first time period, said method further comprising:

receiving a specific query of said time series measurement data during said first time period;

performing said specific query of said time series measurement data using said first file during said first time period;

receiving a particular query of said time series measurement data after said first time period; and

performing said particular query of said time series measurement data in said second file.

17. A method according to claim 13, wherein:

said steps of receiving, writing and converting are performed by application performance analysis software.

18. A method according to claim 17, wherein said application performance analysis software further performs the step of:

adding probes and an Agent to a software application, said Agent receives data from said probes, said time series management data is received from said Agent.

19. A method according to claim 13, wherein:

said converting includes creating a set of files in said second format, each of said set of files corresponds to a different time period; and

said method further comprises aggregating said set of files and down sampling said set of files.

20. A method for processing data, comprising:

receiving measurement data about a system, said measurement data includes a set of data for each time period of a set of time periods, each set of data includes information for a set of metrics about said system during a respective time period;

saving said measurement data in a first format such that information from a particular set of data is stored together; and

converting said measurement data from said first format to a second format such that information for each metric is stored together.

21. A method according to claim 20, wherein:

said second format includes blocks such that there is a block for each metric;

each block includes a set of records corresponding to time samples; and

each record is compressed individually.

22. A method according to claim 20, wherein:

said first format includes a set of blocks of information, each block of information stores data for a set of metrics pertaining to a common time period; and

said second format includes a header and a set of blocks of data and a footer, each bock of data stores information for one metric over multiple time periods, said footer includes pointers to each block of data

23. A method according to claim 20, wherein:

said saving includes writing a first file during a first time period; and

said converting includes writing a second file after said first time period.

24. A method according to claim 23, further comprising:

receiving a specific query of said time series measurement data during said first time period;

performing said specific query of said time series measurement data using said first file during said first time period;

receiving a particular query of said time series measurement data after said first time period; and

performing said particular query of said time series measurement data in said second file.

25. A method according to claim 20, wherein:

said steps of receiving, saving and converting are performed by performance analysis software.

26. A method according to claim 25, wherein said performance analysis software further performs the step of:

adding probes and an Agent to a software application, said Agent receives data from said probes, said management data is received from said Agent.

27. A method according to claim 20, wherein:

said converting includes create a set of files in said second format, each of said set of files corresponds to a different time period; and

said method further comprises aggregating said set of files and down sampling said set of files.

28. A method for processing data, comprising:

receiving time series data that includes multiple samples of said data over time;

writing said time series data to a first format suited for fast writing; and

converting said time series data from said first format to a second format having blocks of records, said converting includes compressing said records individually, said blocks have data from multiple samples of said data.

29. A method according to claim 28, wherein:

said first format includes a set of blocks of information, each block of information stores a time sample for a set of metrics; and

said second format includes a header and a set of blocks of data and a footer, each bock of data stores information for one metric over multiple time samples, said footer includes pointers to each block of data.

30. A method according to claim 28, further comprising:

receiving a first query of said time series data during a first time period;

performing said first query of said time series data in said first format during said first time period;

receiving a second query of said time series data after said first time period; and

performing said second query of said time series data in said second format.

31. A method according to claim 30, wherein:

said steps of receiving, writing and converting are performed by performance analysis software.

said method further comprises adding probes and an Agent to a software application, said Agent receives data from said probes, said time series data is received from said Agent.

32. A method according to claim 28, wherein:

said converting includes create a set of files in said second format, each of said set of files corresponds to a different time period; and

said method further comprises aggregating said set of files and down sampling said set of files.

33. A method for processing data, comprising:

receiving time series data;

writing said time series data to a first format suited for fast writing during a first time period;

converting said time series data from said first format to a second format suited long term storage after said first time period;

receiving a query of said time series data during said first time period;

performing said query of said time series data during said first time period using said time series data in said first format;

receiving a query of said time series data after said first time period; and

performing said query of said time series data after said first time period using said time series data in said second format.

34. A method according to claim 33, wherein:

said first format includes a set of blocks of information, each block of information stores a time sample for a set of metrics; and

said second format includes a header and a set of blocks of data and a footer, each bock of data stores information for a metric over multiple time samples, said footer includes pointers to each block of data, said information for a metric over multiple time samples includes a set of records that are individually compressed.

35. A method for processing data about a system, comprising:

adding probes to a software application;

acquiring performance data about said software application using said probes in real time;

sending said performance data to a data manager;

storing said performance data in one or more searchable files according to a first format; and

periodically converting a subset of said performance data to searchable files according to a second format.

36. A method according to claim 35, further comprising:

aggregating at least a subset of said searchable files according to said second format.

37. A method according to claim 36, further comprising:

down sampling said aggregated searchable files.

38. A method according to claim 35, wherein:

said first format stores data in blocks by time periods; and

said second format stores data in blocks of data for individual metrics.

39. One or more processor readable storage devices storing processor readable code for programming one or more processors, said processor readable code comprising:

a writer, said writer receives data about a set of metrics and stores said data in a first format having blocks sorted by time; and

a converter, said converter accesses said data in said first format and converts said data from said first format to a second format having blocks sorted by metric.

40. One or more processor readable storage devices according to claim 39, wherein said processor readable code further comprises:

an appender, said converter creates a set of data structures for a set of time periods, said appender aggregates said set of data structures to a single data structure.

41. One or more processor readable storage devices according to claim 39, wherein said processor readable code further comprises:

a down sampler, said down sampler creates down sampled data structures from original data structures in said second format.

42. One or more processor readable storage devices according to claim 39, wherein:

said first format includes a set of blocks of information, each block of information stores data for a set of metrics pertaining to a time period; and

said second format includes a header and a set of blocks of data and a footer, each bock of data stores information for a metric over multiple time periods, said footer includes pointers to each block of data.

43. One or more processor readable storage devices according to claim 42, wherein:

said information for a metric over multiple time periods includes a set of records that are individually compressed.

44. An apparatus for processing data, comprising:

a communication interface;

a storage device; and

one or more processors in communication with said storage device and said communication interface, said one or more processors perform a method comprising;

receiving measurement data about a system, said measurement data includes a set of data for a set of metrics during each time period of a set of time periods,

storing said measurement data in real time in a first format,

converting said stored measurement data from said first format to a second format and storing said measurement data in said second format in a set of one or more files associated with contiguous time periods,

aggregating said set of one or more files; and

down sampling said aggregated set of one or more files.

45. An apparatus according to claim 44, wherein:

said first format includes a set of blocks of information, each block of information stores data for said metrics during a respective time period; and

said second format includes a header and a set of blocks of data and a footer, each bock of data stores information for a metric over multiple time periods, said footer includes pointers to each block of data.

46. One or more processor readable storage devices storing processor readable code for programming one or more processors, said processor readable code comprising:

code means for receiving time series measurement data;

code means for writing said time series measurement data to a first format suited for fast writing; and

code means for converting said time series measurement data from said first format to a second format suited for compact storage.

47. One or more processor readable storage devices according to claim 46, wherein:

said second format is suited for faster querying than said first format.

48. One or more processor readable storage devices according to claim 46, further comprising:

code means for down sampling information converted to said second format.