Naïve, client-side sharding with online addition of shards

• US 9,619,545 B2
• Filed: 03/26/2014
• Issued: 04/11/2017
• Est. Priority Date: 06/28/2013
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Hardy, Alexandre, Tilak, Omkar
• Primary Examiner(s): Leroux, Etienne
• Assistant Examiner(s): Bibbee, Jared
• Application Number:
  US14/226,557
• Publication Number:
  US 20150006482A1
• Time in Patent Office: 1,112 Days
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/278 Data partitioning, e.g. horizontal or vertical partitioning

Abstract

Multiple clients can be enabled to perform operations relative to data items in a shard system asynchronously to each other without the use by those clients of exclusive locks. A rebalancing event, in which data items are redistributed automatically among a set of shards due to a modification of the quantity of shards in the system, can be performed without the use of exclusive locks by clients. Clients can continue to perform operations relative to at least some of the data items in the shard system even while rebalancing processes are redistributing at least some of the data items asynchronously during a system-wide rebalancing event. All of these benefits can be obtained without sacrificing data consistency within the shard system.

Claims

1. A computer-implemented method comprising:

determining a change in a quantity of shards in a multi-shard system that has a plurality of shards;

in response to determining the change in the quantity of shards, transitioning a client from a normal state to a rebalancing state, wherein during the normal state, the client performs a type of operation relative to data items stored in the multi-shard system in a first manner, wherein during the rebalancing state, the client performs the type of operation relative to the data items stored in the multi-shard system in a second manner, wherein the second manner is different than the first manner, wherein during the rebalancing state, the client performs the type of operation without the client acquiring one or more exclusive locks relative to any of the data items, and wherein during the rebalancing state, the type of operation is performed based on;

a first shard in the multi-shard system, the first shard including a data item stored at a first memory location in the first shard before the change in the quantity of shards, wherein the data item is one of the data items; and

a second shard in the multi-shard system, the second shard including the data item stored at a second memory location in the second shard after the change in the quantity of shards;

while the client is in the rebalancing state, determining, for one or more data items of the data items, one or more destination shards that are separate from one or more source shards, wherein the one or more source shards store the one or more data items during the normal state prior to transitioning the client from the normal state to the rebalancing state; and

moving the one or more data items from the one or more source shards to the one or more destination shards while the client is in the rebalancing state.

2. The computer-implemented method of claim 1, further comprising:

while the one or more data items are being moved from the one or more source shards to the one or more destination shards, the client performing, asynchronously to the moving of the one or more data items, and relative to a data item, an operation of the type of operation in the second manner without acquiring an exclusive lock.

3. The computer-implemented method of claim 1, further comprising:

determining the one or more destination shards for the one or more data items by hashing primary keys of the data items based at least in part on the quantity of shards in the multi-shard system following the change in the quantity of shards; and

in response to determining that the one or more data items have been moved from the one or more source shards to the one or more destination shards, transitioning the client from the rebalancing state to the normal state.

4. The computer-implemented method of claim 1, further comprising:

while the one or more data items are being moved from the one or more source shards to the one or more destination shards, the client performing an add operation in the second manner relative to a first data item having a primary key at least in part by;

determining, based at least in part on a quantity of shards in the multi-shard system preceding the change in the quantity of shards, a particular source shard on which the first data item is stored prior to the change in the quantity of shards;

determining, based at least in part on a quantity of shards in the multi-shard system following the change in the quantity of shards, a particular destination shard on which the first data item is to be added after the change in the quantity of shards;

determining that no data item on the source shard has the primary key of the first data item;

determining that no data item on the destination shard has both a true tombstone attribute value and the primary key of the first data item; and

inserting the first data item into the destination shard.

5. The computer-implemented method of claim 1, further comprising:

while the one or more data items are being moved from the source shards to the destination shards, the client performing an add operation in the second manner relative to a first data item having a primary key at least in part by;

determining, based at least in part on a quantity of shards in the multi-shard system preceding the change in the quantity of shards, a particular source shard on which the first data item is stored prior to the change in the quantity of shards;

determining, based at least in part on a quantity of shards in the multi-shard system following the change in the quantity of shards, a particular destination shard on which the first data item is to be added after the change in the quantity of shards;

determining that no data item on the source shard has the first data item'"'"'s primary key;

determining that a second data item on the destination shard has both a true tombstone attribute value and the first data item'"'"'s primary key;

assigning, to one or more attributes of the second data item, attribute values of the first data item; and

setting the second data item'"'"'s tombstone attribute value to false.

6. The computer-implemented method of claim 1, further comprising:

while the one or more data items are being moved from the source shards to the destination shards, the client performing an update type of operation in the second manner relative to a first data item having a primary key at least in part by;

determining, based at least in part on a quantity of shards in the multi-shard system preceding the change in the quantity of shards, a particular source shard on which the first data item is stored prior to the change in the quantity of shards;

determining, based at least in part on a quantity of shards in the multi-shard system following the change in the quantity of shards, a particular destination shard on which the first data item is to be located after the change in the quantity of shards;

determining that no data item on the source shard has the first data item'"'"'s primary key;

determining that a second data item on the destination shard has both a false tombstone attribute value and the first data item'"'"'s primary key; and

assigning, to one or more attributes of the second data item, attribute values of the first data item.

7. The computer-implemented method of claim 1, further comprising:

while the one or more data items are being moved from the source shards to the destination shards, the client performing an update type of operation in the second manner relative to a first data item having a primary key at least in part by;

determining, based at least in part on a quantity of shards in the multi-shard system preceding the change in the quantity of shards, a particular source shard on which the first data item is stored prior to the change in the quantity of shards;

determining, based at least in part on a quantity of shards in the multi-shard system following the change in the quantity of shards, a particular destination shard on which the first data item is to be located after the change in the quantity of shards;

determining that a second data item on the source shard has both the first data item'"'"'s primary key and a false tombstone attribute value;

determining that a third data item on the destination shard has the first data item'"'"'s primary key;

assigning, to one or more attributes of the third data item, attribute values of the first data item; and

setting a version number of the third data item to a value different from a version number of the second data item.

8. The computer-implemented method of claim 1, further comprising:

while the one or more data items are being moved from the source shards to the destination shards, the client performing an update type of operation in the second manner relative to a first data item having a primary key at least in part by;

determining, based at least in part on a quantity of shards in the multi-shard system preceding the change in the quantity of shards, a particular source shard on which the first data item is stored prior to the change in the quantity of shards;

determining, based at least in part on a quantity of shards in the multi-shard system following the change in the quantity of shards, a particular destination shard on which the first data item is to be located after the change in the quantity of shards;

determining that a second data item on the source shard has both the first data item'"'"'s primary key and a false tombstone attribute value;

determining no data item on the destination shard has the first data item'"'"'s primary key; and

inserting the first data item into the destination shard with a version number that differs from a version number of the second data item.

9. The computer-implemented method of claim 1, further comprising:

while the one or more data items are being moved from the source shards to the destination shards, the client performing a delete type of operation in the second manner relative to a first data item having a primary key at least in part by;

determining, based at least in part on a quantity of shards in the multi-shard system preceding the change in the quantity of shards, a particular source shard on which the first data item is stored prior to the change in the quantity of shards;

determining, based at least in part on a quantity of shards in the multi-shard system following the change in the quantity of shards, a particular destination shard on which the first data item is to be located after the change in the quantity of shards;

determining that a second data item on the source shard has the first data item'"'"'s primary key;

upserting the first data item into the destination shard with a version number that differs from a version number of the second data item; and

setting a tombstone attribute value of the first data item on the destination shard to true.

10. The computer-implemented method of claim 1, further comprising:

while the one or more data items are being moved from the source shards to the destination shards, the client performing a delete type of operation in the second manner relative to a first data item having a primary key at least in part by;

determining, based at least in part on a quantity of shards in the multi-shard system preceding the change in the quantity of shards, a particular source shard on which the first data item is stored prior to the change in the quantity of shards;

determining, based at least in part on a quantity of shards in the multi-shard system following the change in the quantity of shards, a particular destination shard on which the first data item is to be located after the change in the quantity of shards;

determining that no data item on the source shard has the first data item'"'"'s primary key;

determining that a second data item on the destination shard has the first data item'"'"'s primary key; and

setting a tombstone attribute value of the second data item to true.

11. The computer-implemented method of claim 1, further comprising:

while the one or more data items are being moved from the source shards to the destination shards, the client performing a get type of operation in the second manner relative to a primary key at least in part by;

determining, based at least in part on a quantity of shards in the multi-shard system preceding the change in the quantity of shards, a source shard on which a data item having the primary key is stored prior to the change in the quantity of shards;

determining, based at least in part on a quantity of shards in the multi-shard system following the-change in the quantity of shards, a destination shard on which a data item having the primary key is to be located after the change in the quantity of shards;

determining that a first data item on the destination shard has the primary key and a false tombstone attribute value; and

reading attribute values of the first data item.

12. The computer-implemented method of claim 1, further comprising:

while the one or more data items are being moved from the source shards to the destination shards, the client performing a get type of operation in the second manner relative to a primary key at least in part by;

determining, based at least in part on a quantity of shards in the multi-shard system preceding the change in the quantity of shards, a source shard on which a data item having the primary key is stored prior to the change in the quantity of shards;

determining, based at least in part on a quantity of shards in the multi-shard system following the change in the quantity of shards, a destination shard on which a data item having the primary key is to be located after the change in the quantity of shards;

determining that no data item on the destination shard has the primary key;

determining that a first data item on the source shard has the primary key and a false tombstone attribute value; and

reading attribute values of the first data item.

13. The computer-implemented method of claim 1, further comprising:

while the one or more data items are being moved from the source shards to the destination shards, the client performing a get type of operation in the second manner relative to a primary key at least in part by;

determining, based at least in part on a quantity of shards in the multi-shard system preceding the change in the quantity of shards, a source shard on which a data item having the primary key is stored prior to the change in the quantity of shards;

determining, based at least in part on a quantity of shards in the multi-shard system following the change in the quantity of shards, a destination shard on which a data item having the primary key is to be located after the change in the quantity of shards;

determining, at a first moment in time, that no data item on the destination shard has the primary key;

determining that no data item on the source shard has the primary key;

determining, at a second moment in time, that a first data item on the destination shard has the primary key and a false tombstone attribute value; and

reading attribute values of the first data item.

14. The computer-implemented method of claim 1, wherein moving the one or more data items from the source shards to the destination shards comprises:

incrementing a system-wide version number; and

for each first data item having a version number that is less than the system-wide version number, determining, based at least in part on a quantity of shards in the multi-shard system following the shard quantity change, a first destination shard on which the first data item is to be located after the change in the quantity of shards;

for each second data item that (a) has a version number that is less than the system-wide version number and (b) is located on a source shard that differs from a second destination shard on which the second data item is to be located after the change in the quantity of shards, inserting, into the second destination shard, a copy of the second item having the system-wide version number;

for each third data item that (a) has a version number that is less than the system-wide version number and (b) is already located on a third destination shard on which the third data item is to be located after the change in the quantity of shards, changing a version number of the third item to the system-wide version number.

15. The computer-implemented method of claim 1, wherein moving the one or more data items from the source shards to the destination shards comprises incrementing a system-wide version number, and further comprising:

determining a set of queries that are pending against one or more shards in the multi-shard system as of a time that the movement of the one or more data items completes;

waiting for all queries in the set of queries to finish;

after all queries in the set of queries have finished, removing, from all shards in the multi-shard system, all data item copies having version number attribute values that differ from the system-wide version number; and

after all queries in the set of queries have finished, removing, from all shards in the multi-shard system, all data item copies having true tombstone attribute values.

16. The computer-implemented method of claim 1, further comprising:

while the one or more data items are being moved from the source shards to the destination shards, and for each particular shard in the multi-shard system, populating a preliminary result queue for that particular shard with data items that both (a) are located on that particular shard and (b) satisfy specified query criteria;

for each particular shard'"'"'s preliminary result queue, sorting data items in that shard'"'"'s preliminary result queue based at least in part on primary keys of the data item in that shard'"'"'s preliminary result queue;

until a preliminary result queue of a shard that is non-empty does not exit, repeatedly performing operations comprising;

selecting a first data item from a set comprising data items currently located at tops of preliminary result queues of all of the shards in the multi-shard system, wherein selecting the first data item from the set comprising data items currently located at tops of preliminary result queues of all of the shards in the multi-shard system comprises;

selecting, from the set, a subset of data items having a smallest primary key of primary keys of data items in the set; and

selecting, as the first data item, a data item having a largest version number attribute value of version number attribute values of data item in the subset,

adding, to a final result set, each first data item that does not have a true tombstone attribute value, and

removing, from every shard'"'"'s preliminary result queue, all data item copies having a primary key that matches a primary key of the first data item; and

after every shard'"'"'s preliminary result queue is empty, returning data items in the final result set as query result.

17. The computer-implemented method of claim 1, further comprising:

while the one or more data items are being moved from the source shards to the destination shards, the client attempting and failing to perform a delete type of operation in the second manner relative to a first data item having a primary key at least in part by;

determining that no data item on the source shard has the first data item'"'"'s primary key;

determining that no data item on the destination shard has the first data item'"'"'s primary key; and

generating data that indicates that the delete operation failed due to the first data item not existing.

20. The computer-implemented method of claim 1, further comprising:

identifying a first quantity of shards, wherein the first quantity of shards indicates a quantity of shards in the multi-shard system before the change in the quantity of shards;

identifying a second quantity of shards, wherein the second quantity of shards indicates a quantity of shards in the multi-shard system after the change in the quantity of shards;

identifying the first shard based on the first quantity of shards, wherein the first shard is included in the plurality of shards; and

identifying the second shard based on the second quantity of shards, wherein the second shard is included in the plurality of shards.

18. A system comprising:

a plurality of database shards that store data items;

a plurality of clients that are configured to transition, in response to a change in a quantity of the plurality of database shards, from a normal state to a rebalancing state, wherein during the normal state, the client performs a type of operation relative to the data items stored in the multi-shard system in a first manner, wherein during the rebalancing state, the client performs the type of operation relative to the data items stored in the multi-shard system in a second manner, wherein the second manner is different than the first manner, wherein during the rebalancing state, the client performs the type of operation without the client acquiring one or more exclusive locks relative to any of the data items, and wherein during the rebalancing state, the type of operation is performed based on;

a first shard in the multi-shard system, the first shard including a data item stored at a first memory location in the first shard before the change in the quantity of the plurality of database shards, wherein the data items is of data items; and

a second shard in the multi-shard system, the second shard including the data item stored at a second memory location in the second shard after the change in the quantity of the plurality of database shards; and

at least one computing device configured to (a) determine, while the plurality of clients are in the rebalancing state, and for one or more data items of the data items, one or more destination shards that are separate from one or more source shards, wherein the one or more source shards store the one or more data items during the normal state while each client of the plurality of clients is in the normal state, and (b) move the one or more data items from the one or more source shards to the one or more destination shards while each client of the plurality of clients is in the rebalancing state.

19. A non-transitory computer-readable storage memory storing processor-executable instructions comprising:

instructions to cause one or more processors to determine a change in a quantity of shards in a multi-shard system that has a plurality of shards;

instructions to cause one or more processors to transition a client, in response to determining the change in the quantity of shards, from a normal state to a rebalancing state, wherein during the normal state, the client performs a type of operation relative to data items stored in the multi-shard system in a first manner, wherein during the rebalancing state, the client performs the type of operation relative to the data items stored in the multi-shard system in a second manner, wherein during the rebalancing state, the client performs the type of operation without the client acquiring one or more exclusive locks relative to any of the data items, and wherein during the rebalancing state, the type of operation is performed based on;

a first shard in the multi-shard system, the first shard including a data item stored at a first memory location in the first shard before the change in the quantity of shards, wherein the data is one of the data items; and

a second shard in the multi-shard system, the second shard including the data item stored at a second memory location in the second shard after the change in the quantity of shards;

instructions to cause one or more processors to determine, while the client is in the rebalancing state, and for one or more data items of the data items, one or more destination shards that are separate from one or more source shards, wherein the one or more source shards store the one or more data items during the normal state prior to transitioning the client from the normal state to the rebalancing state; and

instructions to cause one or more processors to-move the one or more data items from the one or more source shards to the one or more destination shards while the client is in the rebalancing state.