Method and apparatus for managing time series database

• US 10,229,129 B2
• Filed: 09/22/2014
• Issued: 03/12/2019
• Est. Priority Date: 09/27/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Chen, Xiao Yan, Chen, Yao Liang, Huang, Sheng, Liu, Kai, Sun, Guo Rong, Wang, Chen
• Primary Examiner(s): Kim, Taelor
• Application Number:
  US14/492,423
• Publication Number:
  US 20150095381A1
• Time in Patent Office: 1,632 Days
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/235 Update request formulation
• CPC Class Codes: G06F 16/24 Querying
• CPC Class Codes: G06Q 10/067 Enterprise or organisation modelling

Abstract

A method for managing a time series database, includes: monitoring multiple operations that access the time series database, so as to identify types of the multiple operations, the types of the multiple operations comprising at least one of the query types or insert types; with respect to a storage mode among multiple storage modes, obtaining costs that the multiple operations access the time series database based on the types, respectively; selecting a storage mode with the minimum cost from the multiple storage modes; and during a predetermined time period, storing into the time series database data values that are collected from multiple measurement points according to the selected storage mode. In one embodiment, there is provided an apparatus for managing a time series database. By means of the method and apparatus of the present invention, the storage and query efficiency with respect to the time series database can be increased.

Claims

1. A method for managing a time series database, comprising:

monitoring multiple operations that access the time series database, so as to identify types of the multiple operations, the types of the multiple operations comprising at least one of a query type and an insert type;

with respect to a storage mode among multiple storage modes, obtaining estimated access costs from a cost model repository, the estimated access costs associated with the multiple operations accessing the time series database based on the operation types;

selecting a storage mode, from a storage mode repository including a plurality of storage modes each corresponding to a different application environment differentiated based on data generation characteristics, to reduce resource overhead related to storing and querying data collected from a plurality of sensors based on collection point amount values and data collecting frequency values measured from an application environment of the different application environments, the selected storage mode corresponding to a minimum access cost for the operation types accessing the time series database dictated by a particular application environment; and

during a predetermined time period, adjusting the time series database according to history operations involving the collection point amount values and the data collecting frequency values stored in the selected storage mode to reduce memory utilization.

2. The method according to claim 1, further comprising:

monitoring status information of a computer device running the time series database; and

revising an access cost based on the status information.

3. The method according to claim 2, wherein the multiple storage modes at least comprise any of:

a group storage mode;

based on groups resulting from dividing multiple measurement points by a measurement time period, storing into at least one group data blob data values that are collected at the multiple measurement points during the time period; and

a high-frequency storage mode;

storing into at least one high-frequency data blob data values that are collected at a measurement point during a time period.

4. The method according to claim 2, wherein the obtaining access costs associated with the multiple operations accessing the time series database based on the types comprises:

calculating the access costs according to a cost model corresponding to the multiple storage modes, based on the types.

5. The method according to claim 4, wherein the calculating the access costs according to a cost model corresponding to the multiple storage modes comprises at least one of:

calculating the access costs based on a query cost model in response to a first part of operations among the multiple operations belonging to the query type, wherein the query cost model describes the access costs of performing query operations to the database; and

calculating the access costs based on an insert cost model in response to a second part of operations among the multiple operations belonging to the insert type, wherein the insert cost model describes the access costs of performing insert operations to the database.

6. The method according to claim 5, wherein the calculating the costs based on a query cost model in response to a first part of operations among the multiple operations belonging to the query type comprises:

extracting from the first part of operations an amount of multiple measurement points associated with the first part of operations and an amount of multiple query time points, wherein the first part of operations represent operations of querying in the time series database data values collected from the multiple measurement points at the multiple query time points; and

estimating the costs at least based on the amount of the multiple measurement points and the amount of the multiple query time points according to the query cost model.

7. The method according to claim 5, the calculating the costs based on an insert cost model in response to a second part of operations among the multiple operations belonging to the insert type comprises:

extracting from the second part of operations the sampling frequency f of inserting data into the database which is associated with the second part of operations, and an amount of measurement points associated with the sampling frequency f, wherein the second part of operations represent inserting data values that are collected from the measurement points into the time series database at the frequency f; and

estimating the costs at least based on the sampling frequency f and the amount according to the query cost model.

8. The method according to claim 3, wherein:

in the group storage mode, at least identifiers of measurement points in the group, the collected data values and time offsets are stored in a data node of one of the at least one group data blob; and

in the high-frequency storage mode, at least the collected data values are stored in a data node of one of the at least one high-frequency data blob.

9. The method according to claim 4, wherein the query cost model and the insert cost model respectively comprise:

cost models corresponding to the group storage mode and the high-frequency storage mode, respectively.

10. The method according to claim 1, wherein the method is executed periodically.

11. A method for managing a time series database, comprising:

storing the collected data values into the time series database according to a method as recited in claim 1; and

in response to a query operation to the time series database, reading data from the time series database based on a data storage structure corresponding to the selected storage mode.

12. An apparatus for managing a time series database, comprising:

a monitoring module configured to monitor multiple operations that access the time series database, so as to identify types of the multiple operations, the types of the multiple operations comprising at least one of a query type and an insert type;

an obtaining module configured to, with respect to a storage mode among multiple storage modes, obtain estimated access costs from a cost model repository, the estimated access costs associated with the multiple operations accessing the time series database based on the operation types;

a selecting module configured to select a storage mode, from a storage mode repository including a plurality of storage modes each corresponding to a different application environment differentiated based on data generation characteristics, to reduce resource overhead related to storing and querying data collected from a plurality of sensors based on collection point amount values and data collecting frequency values measured from an application environment of the different application environments, the selected storage mode corresponding to a minimum access cost for the operation types accessing the time series database dictated by a particular application environment; and

a storage module configured to, during a predetermined time period, adjust the time series database according to history operations involving the collection point amount values and the data collecting frequency values stored in the selected storage mode to reduce memory utilization.

13. The apparatus according to claim 12, further comprising:

a monitoring module configured to monitor status information of a computer device running the time series database, and revise an access cost based on the status information.

14. The apparatus according to claim 13, wherein the multiple storage modes at least comprise any of:

a group storage mode;

based on groups resulting from dividing multiple measurement points by a measurement time period, storing into at least one group data blob data values that are collected at the multiple measurement points during the time period; and

a high-frequency storage mode;

storing into at least one high-frequency data blob data values that are collected at a measurement point during a time period.

15. The apparatus according to claim 13, wherein the obtaining module comprises:

a calculating module configured to calculate the access costs according to a cost model corresponding to the multiple storage modes, based on the types.

16. The apparatus according to claim 15, wherein the calculating module comprises at least one of:

a first calculating module configured to calculate the access costs based on a query cost model in response to a first part of operations among the multiple operations belonging to the query type, wherein the query cost model describes the access costs of performing query operations to the database; and

a second calculating module configured to calculate the access costs based on an insert cost model in response to a second part of operations among the multiple operations belonging to the insert type, wherein the insert cost model describes the access costs of performing insert operations to the database.

17. The apparatus according to claim 16, wherein the first calculating module comprises:

a first extracting module configured to extract from the first part of operations an amount of multiple measurement points associated with the first part of operations and an amount of multiple query time points, wherein the first part of operations represent operations of querying in the time series database data values collected from the multiple measurement points at the multiple query time points; and

a first estimating module configured to estimate the costs at least based on the amount of the multiple measurement points and the amount of the multiple query time points according to the query cost model.

18. The apparatus according to claim 16, the second calculating module comprises:

a second extracting module configured to extract from the second part of operations the sampling frequency f of inserting data into the database which is associated with the second part of operations, and an amount of measurement points associated with the sampling frequency f, wherein the second part of operations represent inserting data values that are collected from the measurement points into the time series database at the frequency f; and

a second estimating module configured to estimate the costs at least based on the sampling frequency f and the amount according to the query cost model.

19. The apparatus according to claim 14, wherein

in the group storage mode, at least identifiers of measurement points in the group, the collected data values and time offsets are stored in a data node of one of the at least one group data blob;

and

in the high-frequency storage mode, at least the collected data values are stored in a data node of one of the at least one high-frequency data blob.

20. The apparatus according to claim 15, wherein the query cost model and the insert cost model respectively comprise:

cost models corresponding to the group storage mode and the high-frequency storage mode, respectively.