SEGMENT DATA VISIBILITY AND MANAGEMENT IN A DISTRIBUTED DATABASE OF TIME STAMPED RECORDS

• US 20150032775A1
• Filed: 07/28/2014
• Published: 01/29/2015
• Est. Priority Date: 07/26/2013
• Status: Active Grant

• Current Assignee: Rill Data, Inc.
• Original Assignee: Metamarkets Group, Inc. (Snap, Inc.)
• Inventors: Yang, Fangjin, Tschetter, Eric
• Granted Patent: US 10,324,942 B2
• Time in Patent Office: Days
• US Class Current: 707/770
• CPC Class Codes: G06F 16/2477 Temporal data queries

Abstract

A distributed database of time stamped records can be used to store time series data such as events occurring on the Internet. A distributed database of time stamped records can store segments of data that contain events for different time intervals. The volume of events occurring on the Internet introduces a “Big Data” variable that makes collections of data sets so large and complex they are difficult to manage. Disclosed are systems and methods to manage segments of a distributed database of time stamped records for optimal size (for storage and performance reasons etc.) and for proper visibility to data when different segments contain data for overlapping time periods.

Claims

1. A non-transitory computer readable medium comprising computer executable instructions stored thereon to cause one or more processors to:

receive a query request at a broker node from a query client, the query request related to a data source and a query time interval;

determine data segments applicable to the query request via interaction with at least one of a coordination node and a zookeeper node, the determined data segments represented as a collection of blocks; and

return a query response to the query client, the query response retrieved from the collection of blocks.

2. The non-transitory computer readable medium of claim 1, wherein each member of the collection of blocks refers to a particular block representing a most recent set of available data for the data source.

3. The non-transitory computer readable medium of claim 1, further comprising instructions to cause the one or more processors to:

build a timeline for at least one of a plurality of data sources.

4. The non-transitory computer readable medium of claim 3, wherein instructions to cause the one or more processors to build the timeline comprise:

identify, via interaction with the zookeeper node, blocks corresponding to the data source and having a time period, wherein each block has a unique version related to the time that the block was created;

identify temporally overlapping portions between blocks; and

present portions of blocks that do not overlap with portions of other blocks having a newer version for the overlapping time period.

5. The non-transitory computer readable medium of claim 1, wherein each block identified by the broker node is located on one of a plurality of query nodes.

6. The non-transitory computer readable medium of claim 5, wherein each query node is one of a historical node and a real-time node.

7. The non-transitory computer readable medium of claim 6, wherein the instructions to determine data segments applicable to the query request further comprise instructions to check a cache for the data segments applicable to the query request.

8. The non-transitory computer readable medium of claim 7, wherein the query request is not sent to query nodes for segments located in the cache.

9. The non-transitory computer readable medium of claim 7, wherein the cache is local to the broker node.

10. The non-transitory computer readable medium of claim 1, wherein the query client comprises a graphical user interface on a client computer.

11. The non-transitory computer readable medium of claim 1, wherein the query client comprises a computer process executing on a computer different from the broker node.

12. The non-transitory computer readable medium of claim 1, wherein the collection of blocks returned to the query client represent a timeline view of blocks responsive to the query.

13. A non-transitory computer readable medium comprising computer executable instructions stored thereon to cause one or more processing units to:

determine a plurality of merger segments to merge into a single merged segment, wherein the determination of merger segments is based on at least one of a size of each of the merger segments, a size of a resulting merged segment, and machine resources providing infrastructure to a distributed database of time stamped records.

14. The non-transitory computer readable medium of claim 13, further comprising instructions to:

define a merged time interval spanning from an earliest start time of the plurality of merger segments to a latest finish time of the plurality of merger segments.

15. The non-transitory computer readable medium of claim 13, wherein each merger segment comprises data in the form of at least one of a dimension and a metric, further comprising instructions to:

combine merger segment data for each overlapping dimension and metric; and

assign a null value for each dimension and metric that does not overlap.

16. The non-transitory computer readable medium of claim 13, wherein each created segment has a unique version.

18. The non-transitory computer readable medium of claim 16, further comprising instructions to:

obtain, by a coordinator, at least one default segment rule; and

perform an action on one or more segments based on the at least one default segment rule when the rule matches at least one of the time range and the data source.

19. The non-transitory computer readable medium of claim 16, wherein the action is one of load a segment and drop a segment.

20. The non-transitory computer readable medium of claim 16, further comprising instructions to emit an error message when the rule does not match at least one of the time range and the data source.

17. A non-transitory computer readable medium comprising computer executable instructions stored thereon to cause one or more processing units to:

determine a time range for each of a plurality of segments;

determine a data source for each of the plurality of segments;

obtain, by a coordinator, at least one data source specific rule; and

perform an action on one or more of the plurality of segments based on the at least one data source specific rule when the rule matches at least one of the time range and the data source.