Guaranteed delivery of replication message

• US 10,021,186 B2
• Filed: 03/04/2015
• Issued: 07/10/2018
• Est. Priority Date: 12/19/2014
• Status: Active Grant

• Current Assignee: Microsoft Corporation, Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Grebnov, Ilya, Banina, Samuel, Lamanna, Charles, Lam, Kevin
• Primary Examiner(s): Whipple, Brian
• Application Number:
  US14/638,202
• Publication Number:
  US 20160182633A1
• Time in Patent Office: 1,224 Days
• Field of Search: None
• CPC Class Codes: G06F 16/184 implemented as replicated file system
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Aspects of the present invention relate to guaranteeing delivery of replication messages in distributed storage systems. A request to perform an operation may be received. A first replication message corresponding to the request may be created, where the first replication message comprises instructions to replicate the result of the operation to one or more target storages. The replication message may be inserted in a message queue with a delayed visibility. The operation may be performed, where a result of the operation is persisted in a source storage in a data store. Upon determining that the result was successfully persisted in the source storage, a second replication message may be created, where the second replication message comprises instructions to replicate the result of the operation to the one or more target storages. The second replication message may be inserted in the message queue with an immediate visibility.

Claims

1. One or more computer-storage media storing computer-executable instructions that, when executed by a computing device having a processor, cause the computing device to perform a method for guaranteeing delivery of replication messages in distributed storage systems, the method comprising:

receiving a request to perform an operation;

creating a first replication message corresponding to the request, the first replication message comprising instructions to replicate the result of the operation to one or more target storages in corresponding data stores;

inserting the first replication message into a message queue, the first replication message having a delayed visibility;

performing the operation, a result of the operation being persisted in a source storage in a data store;

upon determining that the result was successfully persisted in the source storage, creating a second replication message corresponding to the request, the second replication message comprising instructions to replicate the result of the operation to the one or more target storages in the corresponding data stores; and

inserting the second replication message in the message queue, the second replication message having immediate visibility.

2. The media of claim 1, wherein the source storage is a source table on a local data store and the one or more target storages are target tables on remote data stores.

3. The media of claim 2, wherein the operation comprises writing a set of rows to the source table in the data store.

4. The media of claim 1, wherein the one or more target storages in corresponding data stores comprise target tables for replicating the result.

5. The media of claim 1, wherein the first replication message further comprises instructions corresponding to the request, the instructions including one or more of the one or more target storages, a time stamp, and unique key corresponding to the result of the operation to be replicated and persisted in the one or more target storages.

6. The media of claim 1, wherein the delayed visibility time is a quantified time period that is at least longer than a time during which the operation is performed and the result persisted in the source storage.

7. The media of claim 1, wherein the delayed visibility time is quantified to factor at least one immediate retry of the operation caused by transient errors.

8. The media of claim 1, wherein the second replication message optimizes replication latency in that replication of the result of the operation is performed immediately.

9. The media of claim 1, wherein the method is performed as a background process.

10. A computer-implemented method for guaranteeing delivery of messages, the method comprising:

receiving a request to perform an operation, the operation being associated with a result, wherein the result of the operation is persisted in a source storage in a data store, wherein the message is a replication message corresponding to the request, wherein the replication message comprises instructions to replicate the result of the operation to one or more target storages on corresponding data stores, and wherein the replication message is inserted into the message queue;

creating a message corresponding to the request, the message indicating a notification to be communicated upon the operation achieving the result;

inserting the message in a message queue, the message having a delayed visibility, the delayed visibility facilitating maintaining the message in the queue until the message is dequeued upon communicating the notification; and

determining that the operation was successfully persisted in the source storage;

upon determining that the message was successfully persisted in the source storage, creating a second replication message corresponding to the request, wherein the second replication message comprises instructions to replicate the result of the operation to the one or more target storages on corresponding data stores; and

inserting the second replication message into the message queue, the second replication message having an immediate visibility.

11. The method of claim 10, further comprising:

partially processing the operation to a source table upon identifying an indication of a failure to perform the operation to successfully persist an entirety of the result;

iteratively processing the replication message in the queue upon identifying the indication of the failure to perform the operation to successfully persist the entirety of the result, wherein the iteratively processing comprises replicating a portion of the result that is successfully persisted;

upon determining that the entirety of the result was successfully persisted in the source storage, processing the replication message in the queue to replicate the result of the operation; and

dequeueing the replication message.

12. The method of claim 10, wherein the operation comprises a plurality of rows to persist in a source storage of a local data store, wherein the plurality of rows are replicated on one or more target storages on corresponding data stores.

13. A system for performing processing of replication messages in distributed storages, comprising:

a source storage component configured to store a result of an operation;

a replication guarantee component configured to;

receive a request to perform the operation, the result of the operation being persisted in the source storage component on a data store;

create a replication message corresponding to the request; and

insert the replication message into the message queue, the message having a delayed visibility, wherein the delayed visibility facilitates maintaining the replication message in the queue until the replication message is dequeued upon communicating instructions to replicate the result of the operation on each of one or more target storages in corresponding data stores; and

a target storage component configured to store the result of the operation, wherein storing the result is based on the replication message, the replication message comprising instructions to replicate the result of the operation on the target storage component; and

the replication guarantee component further configured to;

perform a latency optimization operation, wherein the latency optimization operation comprises;

upon determining that the result was successfully persisted on the source storage, creating a second replication message corresponding to the request, wherein the second replication message comprises instructions to replicate the result of the operation to the one or more target storages in corresponding data stores; and

inserting the second replication message in the message queue, the second replication message having an immediate visibility.

14. The system of claim 13, wherein the replication guarantee component is configured for performing a latency optimization operation, wherein the latency optimization operation comprises:

upon determining that the message was successfully persisted on the source storage, modifying the delayed visibility time of the replication message to make the message visible immediately.

15. The system of claim 13, wherein the replication guarantee component is configured for guaranteeing replication of the result of the operation, wherein guaranteeing replication of the result of the operation comprises:

partially processing the operation to a source table upon identifying an indication of a failure to perform the operation to successfully persist an entirety of the result.

16. The system of claim 15, wherein guaranteeing replication of the result of the operation further comprises:

iteratively processing the replication message in the queue upon identifying the indication of the failure to perform the operation to successfully persist the entirety of the result, wherein the iteratively processing comprises replicating a portion of the result that is successfully persisted; and

upon determining that the entirety of the result was successfully persisted in the source storage, processing the replication message in the queue to replicate the result of the operation; and

dequeueing the replication message.

17. The system of claim 13, wherein the result of the operation is persisted in a source storage of the source storage component.