Hierarchical diff files

• US 10,078,650 B2
• Filed: 08/25/2016
• Issued: 09/18/2018
• Est. Priority Date: 06/19/2009
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Dole, Bryn R., Lindahl, Gregory B., Markson, Michael, Peters, Keith, Saliba, Robert Michael, Skrenta, Rich, Truel, Robert N.
• Primary Examiner(s): Yen, Syling
• Application Number:
  US15/247,749
• Publication Number:
  US 20160364425A1
• Time in Patent Office: 754 Days
• Field of Search: 707812, 707802, 709212, 709219, 709234, 711122
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2282 Tablespace storage structures; Management thereof
• CPC Class Codes: G06F 16/24554 Unary operations; Data partitioning operations
• CPC Class Codes: G06F 16/2462 Approximate or statistical queries
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A method, according to one embodiment, includes identifying data to be stored in one or more tables on a bucket, where the bucket is an allocation of a partitioned storage in a node of one or more nodes of a swarm database, the bucket having at least one replica, and where no two identical bucket replicas reside on a single node. An identifier and a data storage hierarchical level are assigned to the data. The data is mapped to an index and the data is stored in accordance with the index and the data storage hierarchical level, the storing including writing the data to a row in one of the one or more tables on the bucket and recording a write operation into a transaction log of the node. A plurality of write operations are received, and a plurality of write tasks of the bucket are combined for a predetermined time period.

Claims

1. A computer-implemented method, comprising:

identifying data to be stored in one or more tables on a bucket, where the bucket is an allocation of a partitioned storage in a node of one or more nodes of a swarm database, the bucket having at least one replica, and where no two identical bucket replicas reside on a single node;

assigning an identifier and a data storage hierarchical level to the data;

mapping the data to an index and storing the data in accordance with the index and the data storage hierarchical level, the storing including writing the data to a row in one of the one or more tables on the bucket and recording a write operation into a transaction log of the node;

receiving a plurality of write operations;

combining a plurality of write tasks of the bucket for a predetermined time period;

designating a merging limit value to the one or more tables on the bucket, wherein the merging limit value is indicative of one of a plurality of data storage hierarchical levels; and

limiting a merging of files in accordance with the merging limit value by refraining from merging data files in the one of the plurality of data storage hierarchical levels once the merging limit value is met.

2. The computer-implemented method of claim 1, further comprising delaying one or more of the plurality of write operations to collect additional write operations designated to the bucket.

3. The computer-implemented method of claim 1, wherein the predetermined time period includes a time period needed to collect a total amount of write operations that meet a predetermined file size.

4. The computer-implemented method of claim 1, further comprising:

storing a bloom filter for a plurality of row keys for each of a plurality of data storage hierarchical levels; and

consulting the bloom filter to determine whether one of the plurality of row keys is found in a given data storage hierarchical level of the plurality of data storage hierarchical levels.

5. The computer-implemented method of claim 1, further comprising metering the combined plurality of write tasks.

6. The computer-implemented method of claim 1, wherein the data storage hierarchical level includes a first hierarchical level indicative of oldest data in the one or more tables on the bucket.

7. The computer-implemented method of claim 6, wherein the data storage hierarchical level includes a second hierarchical level indicative of newer stored data in the one or more tables on the bucket.

8. The computer-implemented method of claim 7, wherein the data storage hierarchical level includes a third hierarchical level indicative of most recently stored data in the one or more tables on the bucket.

9. The computer-implemented method of claim 8, wherein the plurality of write tasks are written to the third hierarchical level, data stored in the third hierarchical level is merged to the second hierarchical level after a first predetermined time period, and the data stored in the second hierarchical level is merged to the first hierarchical level after a second predetermined time period.

10. A computer program product for organizing data in a database system, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, the program instructions executable by a processor to cause the processor to perform a method comprising:

identifying, utilizing the processor, data to be stored in one or more tables on a bucket, where the bucket is an allocation of a partitioned storage in a node of one or more nodes of a swarm database, the bucket having at least one replica, and where no two identical bucket replicas reside on a single node;

assigning, utilizing the processor, an identifier and a data storage hierarchical level to the data;

mapping, utilizing the processor, the data to an index and storing the data in accordance with the index and the data storage hierarchical level, the storing including writing the data to a row in one of the one or more tables on the bucket and recording a write operation into a transaction log of the node;

receiving, utilizing the processor, a plurality of write operations;

combining, utilizing the processor, a plurality of write tasks of the bucket for a predetermined time period;

designating, utilizing the processor, a merging limit value to the one or more tables on the bucket, wherein the merging limit value is indicative of one of a plurality of data storage hierarchical levels; and

limiting, utilizing the processor, a merging of files in accordance with the merging limit value by refraining from merging data files in the one of the plurality of data storage hierarchical levels once the merging limit value is met.

11. The computer program product of claim 10, further comprising delaying one or more of the plurality of write operations to collect additional write operations designated to the bucket.

12. The computer program product of claim 10, wherein the predetermined time period includes a time period needed to collect a total amount of write operations that meet a predetermined file size.

13. The computer program product of claim 10, further comprising:

storing a bloom filter for a plurality of row keys for each of a plurality of data storage hierarchical levels; and

consulting the bloom filter to determine whether one of the plurality of row keys is found in a given data storage hierarchical level of the plurality of data storage hierarchical levels.

14. The computer program product of claim 10, further comprising metering the combined plurality of write tasks.

15. The computer program product of claim 10, wherein the data storage hierarchical level includes a first hierarchical level indicative of oldest data in the one or more tables on the bucket.

16. The computer program product of claim 15, wherein the data storage hierarchical level includes a second hierarchical level indicative of newer stored data in the one or more tables on the bucket.

17. The computer program product of claim 16, wherein the data storage hierarchical level includes a third hierarchical level indicative of most recently stored data in the one or more tables on the bucket.

18. A system, comprising:

a processor and logic integrated with and/or executable by the processor, the logic being configured to;

identify data to be stored in one or more tables on a bucket, where the bucket is an allocation of a partitioned storage in a node of one or more nodes of a swarm database, the bucket having at least one replica, and where no two identical bucket replicas reside on a single node;

assign an identifier and a data storage hierarchical level to the data;

map the data to an index and storing the data in accordance with the index and the data storage hierarchical level, the storing including writing the data to a row in one of the one or more tables on the bucket and recording a write operation into a transaction log of the node;

receive a plurality of write operations;

combine a plurality of write tasks of the bucket for a predetermined time period;

designate a merging limit value to the one or more tables on the bucket, wherein the merging limit value is indicative of one of a plurality of data storage hierarchical levels; and

limit a merging of files in accordance with the merging limit value by refraining from merging data files in the one of the plurality of data storage hierarchical levels once the merging limit value is met.