Processing time series data from multiple sensors

• US 10,176,208 B2
• Filed: 03/06/2017
• Issued: 01/08/2019
• Est. Priority Date: 10/31/2012
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Chen, Xiao Yan, Huang, Sheng, Liu, Kai, Wang, Chen
• Primary Examiner(s): Nguyen, An T
• Application Number:
  US15/450,606
• Publication Number:
  US 20170177646A1
• Time in Patent Office: 673 Days
• Field of Search: 702181
• CPC Class Codes: G06F 16/20 of structured data, e.g. relational data
• CPC Class Codes: G06F 16/21 Design, administration or maintenance of databases
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers
• CPC Class Codes: G06Q 40/04 Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: Y04S 40/18 Network protocols supporting networked applications, e.g. including control of end-device applications over a network

Abstract

Processing time sequence data for multiple sensors, wherein the multiple sensors are divided into multiple sensor groups and each data comprises a time stamp and a value associated with the timestamp. The method comprises: receiving time series data from each sensor; assigning the time series data received to a sensor group to which the sensor belongs; storing time series data in a first database of a first memory, such that multiple time series data assigned to the same sensor group in the multiple sensor groups are stored in at least one database record of the first database; obtaining the time series data of each sensor among the multiple sensors from the first database; storing time series data in a second database of a second memory, such that the multiple time series data from the same sensor are stored in at least one database record of the second database.

Claims

1. A method for processing time series data from multiple sensors, wherein the multiple sensors are divided into multiple sensor groups and each time series data comprises a time stamp and a value associated with the timestamp, the method comprising:

receiving, at a processor, time series data from each sensor of the multiple sensors;

assigning, by the processor, the time series data received from each sensor to a sensor group to which the sensor belongs among the multiple sensor groups;

storing, by the processor, time series data in a first database of a first memory, such that multiple time series data assigned to the same sensor group in the multiple sensor groups are stored in at least one database record of the first database;

obtaining, by the processor, the time series data of each sensor among the multiple sensors from the first database;

storing, by the processor, time series data in a second database of a second memory, such that the multiple time series data from the same sensor are stored in at least one database record of the second database, compressing, by the processor, the time series data for each sensor by compressing the timestamps in the time series data to generate compressed time series data for storage in said at least one database record of the first database; and

wherein the compressing the timestamps in the time series data by the processor comprises;

obtaining a base timestamp denoting a base time for obtaining data from a particular sensor in the time series data;

generating for each sensor of a sensor group, a plurality of three-tuples, each three-tuple comprising an identifier of a sensor in the group, the obtained sensor value for that sensor, and a corresponding offset between the timestamp corresponding to the sensor value and the base time stamp.

2. The method according to claim 1, wherein the storing of time series data in a database of a second memory further comprises:

storing time series data as per the timestamps.

3. The method according to claim 1, wherein the storing of time series data in a database of a second memory further comprises:

storing time series data as per the values.

4. The method according to claim 3, further comprising:

calculating a probability distribution function PDF for values from each sensor, and

the storing time series data as per the values comprising;

dividing the values from each sensor into multiple value ranges based on the calculated PDF; and

storing time series data based on the divided value ranges, such that one database record comprises time series data corresponding to values within the same value range.

5. The method according to claim 1, wherein the compressing time series data comprises:

performing linear compression to the values in the time series data from each sensor.

6. The method according to claim 5, wherein the storing time series data in a first database of a first memory comprises:

storing the compressed time series data in the first database of the first memory.

7. The method according to claim 5, wherein the compressing time series data comprises:

performing linear compression to values in time series data from each sensor, and the storing time series data in a second database of a second memory comprises;

storing the compressed time series data in the second database of the second memory.

8. The method according to claim 1, further comprising:

receiving, by the processor, a query to the time series data; and

performing, by the processor, a query to the time series data in the second memory.

9. The method according to claim 8, wherein the query step comprises:

converting, by the processor, the received query into a query to the compressed time series data in the second memory;

performing, by the processor, a query to the compressed time series data in the second memory to obtain a query result; and

decompressing, by the processor, the compressed time series data in the obtained query result.

10. An apparatus for processing time series data from multiple sensors, wherein the multiple sensors are divided into multiple sensor groups and each time series data comprises a time stamp and a value associated with the timestamp, the apparatus comprising:

a hardware processor;

a memory storage device for storing programming instructions configuring the hardware processor to;

receive time series data from each sensor of the multiple sensors;

assign the time series data received from each sensor to a sensor group to which the sensor belongs among the multiple sensor groups;

store time series data in a first database of a first memory, such that multiple time series data assigned to the same sensor group in the multiple sensor groups are stored in at least one database record of the first database;

obtain the time series data of each sensor among the multiple sensors from the first database;

store time series data in a second database of a second memory, such that the multiple time series data from the same sensor are stored in at least one database record of the second database, generate compressed time series data for each sensor by compressing the timestamps in the time series data of the sensors; and

wherein to compress the timestamps in the time series data of the sensors, the hardware processor is further configured to;

obtain a base timestamp denoting a base time for obtaining data from a particular sensor in the time series data;

generating for each sensor of a sensor group, a plurality of three-tuples, each three-tuple comprising an identifier of a sensor in the group, the obtained sensor value for that sensor, and an offset between the timestamp corresponding to the sensor value and the base time stamp.

11. The apparatus according to claim 10, wherein the hardware processor is further configured to further store time series data as per the timestamps.

12. The apparatus according to claim 10, wherein the hardware processor is further configured to further store time series data as per the values.

13. The apparatus according to claim 12, wherein the hardware processor is further configured to:

calculate a probability distribution function PDF for values from each sensor, divide the values from each sensor into multiple value ranges based on the calculated PDF; and

store time series data based on the divided value ranges, such that one database record comprises time series data corresponding to values within the same value range.

14. The apparatus according to claim 10, wherein the hardware processor is further configured to:

perform linear compression to the values in the time series data from each sensor.

15. The apparatus according to claim 14, wherein the hardware processor is further configured to:

store the compressed time series data in the first database of the first memory.

16. The apparatus according to claim 14, wherein the hardware processor is further configured to:

store the compressed time series data in the second database of the second memory.

17. The apparatus according to claim 10, wherein the hardware processor is further configured to:

receive a query to the time series data; and

perform a query to the time series data in the second memory.

18. The apparatus according to claim 17, wherein the hardware processor is further configured to:

convert the received query into a query to the compressed time series data in the second memory;

perform a query to the compressed time series data in the second memory to obtain a query result; and

decompress the compressed time series data in the obtained query result.