Method and device for searching indexes for sensor tag data

• US 10,706,054 B2
• Filed: 08/22/2018
• Issued: 07/07/2020
• Est. Priority Date: 08/17/2018
• Status: Active Grant

• Current Assignee: Machbase Incorporated
• Original Assignee: Machbase Incorporated
• Inventors: Kim, Sung Jin
• Primary Examiner(s): Truong, Loan L. T.
• Application Number:
  US16/109,344
• Publication Number:
  US 20200057818A1
• Time in Patent Office: 685 Days
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating
• CPC Class Codes: G06F 16/221 Column-oriented storage; Management thereof
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24553 of query operations
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency ensuring consistency in mirrored systems G06F11/2064

Abstract

An embodiment of the invention provides a device for searching indexes for sensor tag data. The can include a memory that has sensor tag data stored as a time series; an interface that is configured to receive tag names and time segment information for searching sensor tag data; and a processor that is configured to determine partitions of the memory having the sensor tag data stored therein based on the time segment information and configured to determine corresponding indexes and leaf nodes for the determined partitions based on the tag names and the time segment information. An embodiment of the invention can provide an optimized index structure for and a method of storing and searching time series sensor tag data with which most query patterns relating to time series sensor tag data can be satisfied.

Claims

1. A device for searching indexes for sensor tag data, the device comprising:

a memory having sensor tag data stored as a time series;

an interface configured to receive tag names and time segment information for searching sensor tag data; and

a processor configured to determine partitions of the memory having the sensor tag data stored therein based on the time segment information, the processor configured to determine corresponding indexes and leaf nodes for the determined partitions based on the tag names and the time segment information,

wherein data stored in the memory is stored as records consisting of <

time, tag name, value>

, the records are stored for each partition until a number of the stored records reaches a maximum count,

an index is generated for the partition for which storage is completed with a key in a form of <

tag name, time>

, and fields <

value, RID>

are recorded in a data region of the index, and

wherein, when partition index files configured for the partitions are generated to a particular number, the processor merges the partition index files to generate a larger index file, the processor repeats the merging to generate one index file for one hundred million or more pieces of data,

and when generating a second index file of a subsequent level is completed from a partitioned first index file generated in a previous level, then the processor records a completed status of the second index file in a head region and a tail region of the second index file and deletes the first index file.

2. The device for searching indexes for sensor tag data according to claim 1, wherein an additional data column of the record is stored in the memory such that the additional data column is readable using the RID of the index at a column data file,

and a particular time value of a particular tag name is readable at the leaf node of the corresponding index.

3. The device for searching indexes for sensor tag data according to claim 1, wherein the merging comprises a primary merging and a secondary merging,

a plurality of blocks before the primary merging are configured as a plurality of records having fields <

Tag, Time, Value>

,

blocks after the primary merging performed by the processor are configured to have fields <

Tag, Count, Time, Value, Row ID>

,

blocks after the secondary merging performed by the processor are configured to have fields <

Tag &

Meta, Time, Value, Row ID>

,

the “

Tag &

Meta”

field is configured to include fields of Tag, Offset, and Count and metafiles of Time, Value, and Row ID,

and the metafiles of Time, Value, and Row ID each have subfields of Min, Max, Original Size, Compressed Size, Page, and Offset.

4. The device for searching indexes for sensor tag data according to claim 1, wherein the processor reads a column data file by obtaining a value of the leaf node and the RID.

5. The device for searching indexes for sensor tag data according to claim 1, wherein the processor performs a fault recovery algorithm to restart the device in an event of an abnormal termination caused by an abnormal operation of a hardware and software of the device,

and the processor does not perform a fault recovery operation for sensor data for which a merging of the index files is complete such that data is no longer newly recorded and there are no data updates.

6. A device for searching indexes for sensor tag data, the device comprising:

a memory having sensor tag data stored as a time series;

an interface configured to receive tag names and time segment information for searching sensor tag data; and

a processor configured to determine partitions of the memory having the sensor tag data stored therein based on the time segment information, the processor configured to determine corresponding indexes and leaf nodes for the determined partitions based on the tag names and the time segment information,

wherein data stored in the memory is stored as records consisting of <

time, tag name, value>

, the records are stored for each partition until a number of the stored records reaches a maximum count,

an index is generated for the partition for which storage is completed with a key in a form of <

tag name, time>

, and fields <

value, RID>

are recorded in a data region of the index,

wherein the processor performs a fault recovery algorithm to restart the device in an event of an abnormal termination caused by an abnormal operation of a hardware and software of the device,

and the processor does not perform a fault recovery operation for sensor data for which a merging of the index files is complete such that data is no longer newly recorded and there are no data updates, and

wherein the processor determines whether or not an error due to an abnormal operation during a merging of the index files occurred based on a completed status of a head region and a tail region of the index file,

and if the error occurred, the processor deletes the index file and performs a record search by using first index files of a level before the merging.

7. The device for searching indexes for sensor tag data according to claim 6, wherein the processor verifies whether or not an IO (input/output) error has occurred in units of pages with respect to a data file for which the index file has not been generated,

and the processor processes records of which a recording has not been completed in the data file by way of a rollback.