LARGE DISTRIBUTED DATABASE CLUSTERING SYSTEMS AND METHODS
First Claim
1. A system for managing a large distributed database, the system comprising:
- a configuration component configured to organize a plurality of database partitions; and
a shard server configured to host one or more of the plurality of database partitions, wherein the shard server further comprises a replica set, which includes a primary system and at least two secondary systems for replication of a primary copy of the one or more of the plurality of the database partitions to secondary copies of the one or more partitions;
wherein the configuration component is further configured to;
manage cloning of the one or more partitions by adding configuration information for a new computer system into configuration information for the replica set implementing the shard server hosting the one or more partitions, andwherein the replica set is configured to split the one or more partitions into additional partitions by filtering at least some data from the one or more partitions into the additional partitions.
1 Assignment
0 Petitions
Accused Products
Abstract
Systems and methods are provided for managing asynchronous replication in a distributed database environment, while providing for scaling of the distributed database. A cluster of nodes can be assigned roles for managing partitions of data within the database and processing database requests. In one embodiment, each cluster includes a node with a primary role to process write operations and mange asynchronous replication of the operations to at least one secondary node. Each cluster or set of nodes can host one or more partitions of database data. Collectively, the cluster or set of nodes define a shard cluster that hosts all the data of the distributed database. Each shard cluster, individual nodes, or sets of nodes can be configured to manage the size of any hosted partitions, splitting database partitions, migrating partitions, and/or managing expansion of shard clusters to encompass new systems.
376 Citations
29 Claims
-
1. A system for managing a large distributed database, the system comprising:
-
a configuration component configured to organize a plurality of database partitions; and a shard server configured to host one or more of the plurality of database partitions, wherein the shard server further comprises a replica set, which includes a primary system and at least two secondary systems for replication of a primary copy of the one or more of the plurality of the database partitions to secondary copies of the one or more partitions; wherein the configuration component is further configured to; manage cloning of the one or more partitions by adding configuration information for a new computer system into configuration information for the replica set implementing the shard server hosting the one or more partitions, and wherein the replica set is configured to split the one or more partitions into additional partitions by filtering at least some data from the one or more partitions into the additional partitions.
-
-
2. The system of claim 2, further comprising a partition component configured to:
-
detect a partition size for the at least one of the plurality of database partitions that exceeds a size threshold; split the at least one of the database partitions into at least a first and a second partition; control a distribution of data within the first and the second partition based on a value for a database key associated with the data in the at least one of the plurality of database partitions, wherein controlling the distribution includes minimizing any data distributed to the second partition based on a maximum value for the database key associated with the data in the at least one of the plurality of database partitions. - View Dependent Claims (5, 6, 8, 9, 10)
-
-
3. The system according to claim 3, wherein the partition component is further configured to:
-
assign at least any data in the at least one of the plurality of database partitions having associated database key values less than the maximum value to the first partition; and assign at least any data in the at least one of the plurality of database partitions having database key values greater that the maximum value to the second partition.
-
-
4. The system according to claim 4, wherein the partition component is further configured to:
identify database partitions having a sequential database key.
-
7. The system according to claim 7, further comprising a rebalancing component configured to determine a state of the database based on a distribution of the plurality of partitions across the plurality of servers, wherein the rebalancing component is further configured to migrate at least one partition in response to the state indicating an imbalanced distribution of partitions.
-
11. A computer-implemented method for managing a distributed database, implementing a sharded data architecture and replica set configuration of computer systems hosting the distributed database, the method comprising:
-
organizing, by a computer system, a plurality of database partitions; hosting, by a shard server, one or more of the plurality of database partitions, wherein the shard server comprises a replica set; replicating, by the replica set, the one or more of the plurality of database partitions from a primary system to at least two secondary systems that comprise the replica set; cloning, automatically, the one or more of the plurality of database partitions by adding a new computer system to the replica set; and splitting, the one more of the plurality of database partitions into at least one additional partition during the act of cloning, wherein the act of splitting includes filtering at least some data from the one or more of the plurality of database partitions into the at least one additional partition. - View Dependent Claims (12)
-
- 15. The method according to claim 15, further comprising an act of identifying a sequential database key, wherein the act of controlling is executed responsive to the act of identifying the sequential database key.
-
17. The method according to claim 17, further comprising acts of:
-
determining a state of the distributed database based on a distribution of the plurality of partitions across the plurality of servers; and migrating at least one partition in response to the state indicating an imbalanced distribution of partitions. - View Dependent Claims (16, 18, 19)
-
-
22. A system for managing a large distributed database, the system comprising:
-
a distributed database; a configuration component configured to organize the distributed database into a plurality of database partitions, wherein the configuration component provides data routing information for accepting and routing client requests to the plurality of database partitions; a shard server configured to host one or more of the plurality of database partitions, wherein the shard server further comprises a replica set, wherein the replica set comprises; a primary node and at least two secondary nodes for replication of a primary copy of the one or more of the plurality of the database partitions to secondary copies of the one or more partitions hosted on the secondary nodes; and
wherein the replica set is configured to replicate data operations within the replica set based on an eventually consistent model. - View Dependent Claims (20, 21, 23, 24, 25)
-
-
26. A computer implemented method for managing a large distributed database, the method comprising:
-
organizing, by a computer system, a distributed database into a plurality of database partitions, wherein organizing the distributed database includes defining data routing information for accepting and routing client requests to the plurality of database partitions; hosting, by the computer system, one or more of the plurality of database partitions; replicating, by the computer system, a respective one of the one or more of the plurality of database partitions between a primary node and at least two secondary nodes; wherein the primary node and at least two secondary nodes host a primary copy of the one or more of the plurality of the database partitions at the primary node, and the act of replicating includes replicating the primary copy to secondary copies of the one or more partitions hosted on the secondary nodes. - View Dependent Claims (27, 28, 29)
-
Specification