System for storage, querying, and analysis of time series data
First Claim
1. A method of offering a data service for storage and querying of real-time asset operational data, comprising:
- accessing, from a data source, time stamped data;
creating a plurality of data blocks, each data block comprising a portion of the time stamped data with at least one common metadata attribute and a plurality of indices, each index associated with a corresponding data block, each index comprising metadata associated with the corresponding data block;
storing the plurality of indices in an index database;
storing the plurality of data blocks in a time series database comprising a plurality of computing devices, each of which stores a portion of the plurality of data blocks;
receiving, at a query layer from a requestor, a query that specifies criteria defining a set of data retrieved from the service;
requesting from the index database the indices associated with data blocks stored in the time series database comprising time stamped data accessed to evaluate the query;
preparing a sub-query that produces appropriate data matching the criteria, the sub-query comprising the criteria and a logical operation performed on data matching the criteria;
sending the sub-query to an evaluator resident on each of the computing devices that corresponds to the data blocks identified in the requesting operation;
receiving at the evaluator the sub-query from the query layer;
evaluating the criteria specified in the sub-query with respect to the data blocks stored on the same computing devices as the evaluator in order to select a subset of data;
performing the logical operation specified in the sub-query on the subset of data to produce a sub-result;
returning the sub-result to an output handler;
receiving, at an output handler, the sub-results produced in response to each sub-query;
combining the sub-results into a query result; and
returning the query result to the requestor.
2 Assignments
0 Petitions
Accused Products
Abstract
A system for storing time series data includes an ingester that prepares metadata indices associated with blocks of incoming time series data and stores the blocks of data in a time series database and the indices in a separate index database. The time series database distributes storage of the data blocks among multiple data nodes. A query layer receives queries and uses the index database to determine which data blocks are needed to process the query, and then requests only those data blocks from the time series database. Processing of the query is performed within the time series database only on those data nodes that contain relevant data, and partial results are passed to an output layer for formation into a final query result.
-
Citations
20 Claims
-
1. A method of offering a data service for storage and querying of real-time asset operational data, comprising:
-
accessing, from a data source, time stamped data; creating a plurality of data blocks, each data block comprising a portion of the time stamped data with at least one common metadata attribute and a plurality of indices, each index associated with a corresponding data block, each index comprising metadata associated with the corresponding data block; storing the plurality of indices in an index database; storing the plurality of data blocks in a time series database comprising a plurality of computing devices, each of which stores a portion of the plurality of data blocks; receiving, at a query layer from a requestor, a query that specifies criteria defining a set of data retrieved from the service; requesting from the index database the indices associated with data blocks stored in the time series database comprising time stamped data accessed to evaluate the query; preparing a sub-query that produces appropriate data matching the criteria, the sub-query comprising the criteria and a logical operation performed on data matching the criteria; sending the sub-query to an evaluator resident on each of the computing devices that corresponds to the data blocks identified in the requesting operation; receiving at the evaluator the sub-query from the query layer; evaluating the criteria specified in the sub-query with respect to the data blocks stored on the same computing devices as the evaluator in order to select a subset of data; performing the logical operation specified in the sub-query on the subset of data to produce a sub-result; returning the sub-result to an output handler; receiving, at an output handler, the sub-results produced in response to each sub-query; combining the sub-results into a query result; and returning the query result to the requestor. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15)
-
-
16. A system providing a service for storage and querying of real-time asset operational data, the system comprising:
-
a data pipe comprising a connection through which data flows from a data source to an ingester, the ingester configured to receive data and separate the data into data blocks by performing operations comprising; receiving, via the data pipe, time stamped data from the data source; reading the received time stamped data and metadata associated with the time series data; creating a data block based on a common metadata attribute shared by data in the data block and an index associated with the time stamped data, the index comprising metadata associated with the data block; storing the index in an index database; and storing the data block in a time series database comprising a control node and a plurality of data nodes;
a query pipe, comprising a connection through which a query flows from a requestor to a query layer, the query layer configured to execute the query by performing operations comprising;receiving, via the query pipe, the query that specifies criteria defining a set of data stored in the time series database and an analysis to be performed on the set of data; requesting from the index database the indices associated with data blocks stored in the time series database comprising time stamped data accessed to evaluate the query; preparing a sub-query that produces appropriate data matching the criteria, the sub-query comprising the criteria and a logical operation performed on data matching the criteria that allows the analysis to be completed; sending the sub-query to an evaluator resident on each data node that corresponds to data blocks identified in the requesting operation, above, each evaluator configured to execute the sub-query by performing operations comprising; receiving the sub-query from the query layer; evaluating the criteria specified in the sub-query with respect to the data blocks stored on the same data node as the evaluator in order to select a subset of data; performing the logical operation specified in the sub-query on the subset of data to produce a sub-result; and return the sub-result to an output handler; and a result pipe, comprising a connection through which a query result flows from the output handler to the requestor, the output handler configured to generate the query result by performing operations comprising; receiving the sub-results produced in response to each sub-query; combining the sub-results into the query result; and returning, via the result pipe, the query result to the requestor. - View Dependent Claims (17, 18, 19, 20)
-
Specification