Distributed Database with Modular Blocks and Associated Log Files

• US 20130311441A1
• Filed: 01/06/2012
• Published: 11/21/2013
• Est. Priority Date: 01/06/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Citus Data Bilgi Islemleri Ticaret A.S.
• Inventors: Erdogan, Ozgun Ali, Pathak, Sumedh Suhas
• Granted Patent: US 10,860,563 B2
• Time in Patent Office: Days
• US Class Current: 707/713
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3485 for I/O devices
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/2282 Tablespace storage structures; Management thereof
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/278 Data partitioning, e.g. horizontal or vertical partitioning
• CPC Class Codes: G06F 16/285 Clustering or classification

Abstract

A system includes a master node and worker nodes controlled by the master node. Each worker node stores 25 or more modular blocks of a distributed database, where each modular block has a size of 5 G bytes or less and has an associated log file.

Claims

1. A system, comprising:

a master node; and

a plurality of worker nodes controlled by the master node, wherein each worker node stores 25 or more modular blocks of a distributed database, wherein each modular block of the distributed database has a size of 5 G bytes or less and has an associated log file to record changes associated with the distributed database;

wherein the master node and the plurality of worker nodes communicate over a distributed network utilizing standard protocols.

2. The system of claim 1 wherein the master node receives a database query and partitions the database query to run in parallel across the plurality of worker nodes.

3. The system of claim 1 wherein the master node collects statistics about data in the modular blocks.

4. The system of claim 3 wherein the master node uses the statistics to optimize a database query.

5. The system of claim 3 wherein the master node collects statistics selected from modular block location, size, associated log file and update time.

6. The system of claim 5 wherein the master node communicates with the plurality of worker nodes to update the statistics.

7. The system of claim 1 wherein the master node maintains availability information for the plurality of worker nodes.

8. The system of claim 1 wherein the master node maintains distribution information for the modular blocks across the plurality of worker nodes.

9. The system of claim 7 wherein the master node determines the placement of new modular blocks using the availability information for the plurality of worker nodes.

10. The system of claim 8 wherein the master node determines the placement of new modular blocks based upon the distribution information for the modular blocks.

11. The system of claim 1 wherein the master node redistributes modular blocks across the plurality of worker nodes.

12. The system of claim 7 wherein the master node redistributes modular blocks based upon the availability information.

13. The system of claim 8 wherein the master node redistributes modular blocks based upon the distribution information for the modular blocks.

14. The system of claim 1 wherein the master node controls the replication of modular blocks across the plurality of worker nodes.

15. The system of claim 14 wherein the master node controls the replication of modular blocks based upon availability information for the plurality of worker nodes.

16. The system of claim 14 wherein the master node controls the replication of the modular blocks based upon distribution information for the modular blocks.

17. The system of claim 14 wherein replication is a configurable parameter.

18. The system of claim 1 wherein the modular blocks have a size configurable to any value less than 5 GB.

19. The system of claim 1 wherein the master node produces a new modular block when a current modular block exceeds a maximum size.

20. The system of claim 1 wherein the master node produces a new modular block with half of the information from a current modular block when the current modular block exceeds a maximum size.