Replication processes in a distributed storage environment

• US 8,918,362 B2
• Filed: 05/23/2011
• Issued: 12/23/2014
• Est. Priority Date: 05/23/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Calder, Bradley Gene, Nilakantan, Niranjan, Srivastav, Shashwat, Wu, Jiesheng, Abbasi, Abdul Rafay, Mainali, Shane, Uddaraju, Padmanabha Chakravarthy
• Primary Examiner(s): Ruiz, Angelica
• Application Number:
  US13/113,718
• Publication Number:
  US 20120303581A1
• Time in Patent Office: 1,310 Days
• Field of Search: 707600-831, 707/899, 707999001- 99206
• US Class Current: 707/610
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/275 Synchronous replication
• CPC Class Codes: G06F 16/278 Data partitioning, e.g. horizontal or vertical partitioning

Abstract

Embodiments of the present invention relate to systems, methods, and computer storage media for replicating data in a distributed computing environment utilizing a combination of replication methodologies. A full-object replication may be utilized to replicate a full state of an object from a primary data store to a secondary data store. A checkpoint created after initiating the full-object replication may be parsed to identify changes to the object that have been entered since initiating the full-object replication. This replication process is referred to as a delta-checkpoint replication methodology. Additionally, in an embodiment, a log-based replication methodology may be utilized. The log-based replication may communicate data from a log of the primary data store to the secondary data store. It is also contemplated in an exemplary embodiment that when the log-based replication fails to maintain a throughput threshold, one of the other replication methodologies may be initiated, at least temporarily.

Claims

1. A computer-implemented method in a distributed storage environment utilizing a processor and memory for replicating data utilizing a combination of replication processes in the distributed computing environment, the method comprising:

identifying a first one or more checkpoints that describes complete contents of an object for a data partition range at a primary data store, wherein the data partition range maintains the object;

initiating replicating the first one or more checkpoints to the secondary data store utilizing a full-object replication process that replicates the complete contents of the object from the first one or more checkpoints to the secondary data store;

identifying a second one or more checkpoints for the data partition range at the primary data store;

initiating replicating the second one or more checkpoints to the secondary data store utilizing a delta-checkpoint replication process that updates the complete contents of the object at the secondary data store with changes made to the object at the primary data store after the full-object replication process; and

initiating replicating data from the data partition range at the primary data store to the secondary data store utilizing a log-based replication process.

2. The method of claim 1, wherein the first one or more checkpoints is comprised of at least two checkpoints of data maintained within the data partition range.

3. The method of claim 2 further comprising merging the at least two checkpoints of the first one or more checkpoints into a merged checkpoint;

wherein the initiating replicating of the first one or more checkpoints to the secondary data store includes the primary data store communicating the merged checkpoint to the secondary data store.

4. The method of claim 1, wherein the data partition range represents a range of data unique to an account.

5. The method of claim 1, wherein the secondary data store is in a different geographical location than the primary data store.

6. The method of claim 1, wherein the first one or more checkpoints is a snapshot of a state of the data partition range prior to the initiating replicating the first one or more checkpoints to the secondary data store utilizing the full-object replication process.

7. The method of claim 1, wherein the second one or more checkpoints are created at the primary data store after identifying the first one or more checkpoints.

8. The method of claim 7, wherein the second one or more checkpoints are identified after starting the replication of the first one or more checkpoints to the secondary.

9. The method of claim 1, wherein the delta-checkpoint replication process does not communicate the entire state of an object.

10. The method of claim 1, wherein the delta-checkpointing process communicates from the primary data store to the secondary data store changes to an object that occurred at the primary data store since the first one or more checkpoints were generated.

11. The method of claim 1 further comprising prior to utilizing the log-based replication process, determining the data partition range does not have an associated checkpoint that was generated after the second one or more checkpoints were identified.

12. The method of claim 1, wherein the log-based replication process communicates data from a log of the primary data store to the secondary data store, the data from the log is data that has not been replicated using the checkpoint-based approach prior to communicating from the primary data store to the secondary data store.

13. One or more computer storage media having computer-executable instructions embodied thereon, that when executed by a computing system having a processor and memory, cause the computing system to perform a method for replicating data in a distributed computing environment utilizing a plurality of replication processes, the method comprising:

initiating replicating data from a primary data store to a secondary data store utilizing a log-based replication process, wherein the data is maintained in a log of the primary data store;

determining the log-based replication process is performing below a predefined threshold of replication throughput from the primary data store to the secondary data store; and

in response to determining the log-based replication process is below the predefined threshold, switching from the log-based replication process for replicating data from the primary data store to the secondary data store to a checkpoint replication process.

14. The media of claim 13, wherein the data is maintained within a data partition range of the primary data store.

15. The media of claim 13, wherein the log-based replication processes is utilized to replicate data from the primary data store to the secondary data store unless an amount of the data in the log exceeds the predefined threshold.

16. The media of claim 13, wherein the data is annotated with information as part of the log-based replication process, wherein the information facilitates committing the data at the secondary data store.

17. The media of claim 13, wherein the determining the log-based replication process is performing below a predefined threshold of replication throughput is determined when the log-based replication process is replicating the data at a rate that is less than data is being written to a log of the primary data store.

18. The media of claim 13, wherein the method further comprises writing the data to a buffer.

19. The media of claim 13, wherein the checkpoint replication process is comprised of a full-object replication process or a delta-checkpoint replication process.

20. One or more computer storage media having computer-executable instructions embodied thereon, that when executed by a computing system having a processor and memory, cause the computing system to perform a method for replicating data in a distributed computing environment utilizing a plurality of replication processes, the method comprising:

replicating a first data of a first object from a primary data store to a secondary data store utilizing a full-object replication process based on a first checkpoint that describes contents of the first object, wherein the full-object replication replicates complete contents of the first object;

replicating a second data of the first object from the primary data store to the secondary data store utilizing a delta-checkpoint replication process based on a second checkpoint that describes changes made to the first object at the primary data store, wherein the delta-checkpoint replication process replicates delta differences of the first object that are determined from the changes described by the second checkpoint to the secondary data store, wherein the second checkpoint was generated after the full-object replication process initiated replicating the first data; and

replicating a third data of the first object from the primary data store to the secondary data store utilizing a log-based replication process, wherein the log-based replication process replicates the third data from a log of the primary data store, wherein the third data is not maintained in the first checkpoint or the second checkpoint.