Data replication framework

• US 9,268,835 B2
• Filed: 06/17/2013
• Issued: 02/23/2016
• Est. Priority Date: 12/28/2010
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: O'Neill, Brian S., Holgers, Tobias L., Shivaprakash, Archit, Kuznetsova, Olga I., Raden, Gary P. II, Parakh, Vishal
• Primary Examiner(s): LE, UYEN T
• Application Number:
  US13/919,824
• Publication Number:
  US 20130282656A1
• Time in Patent Office: 981 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/184 implemented as replicated file system
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

Systems and methods are directed to an eventually consistent replicated data store that uses, for its underlying storage, a computer software library that provides a high-performance embedded database for data. The replicated data store employs a plurality of hosts interconnected to one another, allowing for writes to any host and full awareness of membership across all hosts. With the data replication framework, various modes are allowed to be built up on top of the core system.

Claims

1. A computing system for providing peer data replication, the computing system comprising:

a plurality of interconnected hosts, individual ones of the plurality of interconnected hosts comprising;

a data store configured to store data and host availability information that represents a list of available hosts configured to replicate data between one another; and

a client application programming interface configured to read local and remote data;

a membership module corresponding to one or more computing devices, the membership module configured to maintain the list of available hosts and to send the list to data stores of individual hosts; and

a data replication module corresponding to one or more computing devices, the data replication module configured to communicate with the membership module of individual hosts in order to replicate data across hosts determined to be available based on the list of available hosts.

2. The computing system of claim 1, wherein the data comprises a record and a record version number.

3. The computing system of claim 2, wherein the record version number comprises a timestamp.

4. The computing system of claim 2, wherein the record version number is configured to be incremented using a counter external to the computing system for providing peer data replication.

5. The computing system of claim 1, further comprising an acknowledgment receiving module configured to receive acknowledgments from the available hosts of receipt of the data.

6. The computing system of claim 1, wherein the membership module comprises a membership data store.

7. The computing system of claim 1, wherein the membership module assigns a unique membership identification number to respective hosts.

8. The computing system of claim 7, wherein the membership module associates the unique membership identification number with a network address associated with individual hosts.

9. A method of providing peer data replication between a plurality of hosts, the method comprising:

storing data and host availability information associated with individual hosts on respective data stores of the plurality of hosts, the host availability information represents a list of available hosts configured to replicate data between one another;

maintaining the list of available hosts at a membership agent in communication with the respective data stores;

sending the list of available hosts to the respective data stores; and

replicating data stored on respective data stores across hosts determined to be available based on the list of available hosts,

wherein the method is performed on a computing device comprising a processor and memory.

10. The method of claim 9, wherein the data comprises a record and a record version number.

11. The method of claim 10, wherein the record version number comprises a timestamp.

12. The method of claim 9, further comprising receiving a change set from an available host.

13. The method of claim 12, further comprising replicating the change set at individual data stores of other available hosts.

14. The method of claim 12, wherein the change set comprises a change identification number.

15. A method of providing peer data replication between a plurality of hosts, the method comprising:

monitoring availability of the plurality of hosts;

maintaining, on respective data stores of the plurality of hosts, host availability information that represent a list of available hosts configured to replicate data between one another;

communicating with a membership module to synchronize the host availability information among the plurality of hosts;

receiving, at the host, a data change record from another host, wherein the data change record comprises a record version number and represents a change to at least a part of data stored at the host;

comparing the record version number sent from the other host with a record version number stored in a local data store of the host;

accepting the data change record if the record version number sent is greater than the record version number stored in the host'"'"'s local data store, and rejecting the data change record if the record version number sent is less than the record version number stored in the host'"'"'s local data store;

persisting the data change record at the host'"'"'s local data store; and

causing transmission of an acknowledgment of receipt of the data change record to the other host;

wherein the method is performed on a computing device comprising a processor and memory.

16. The method of claim 15, further comprising receiving a data change record from a recorder and causing transmission of an acknowledgment of receipt of the data change record to the recorder.

17. The method of claim 15, further comprising maintaining the list of available hosts at a membership agent.

18. The method of claim 17, further comprising persisting the data change record at individual data stores of other available hosts.

19. The method of claim 15, wherein the data change record comprises a data change record identification number.

20. The method of claim 19, further comprising verifying the data change record identification number for the received data change record.