DATA REPLICATION FROM A CLOUD-BASED STORAGE RESOURCE

• US 20160224609A1
• Filed: 07/07/2015
• Published: 08/04/2016
• Est. Priority Date: 02/02/2015
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Seekircher, Norbert, Soetebeer, Frank, Pruessmann, Boris
• Granted Patent: US 10,180,945 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems

Abstract

A local computing device replicates data stored in a cloud-based storage resource in a way that is substantially transparent to end users. A counter generation module provides a source for sequentially increasing counter indices, each of which is associated with a creation timestamp. When a data record managed by the cloud-based storage resource is created, updated or deleted, an address of the modified data record is recorded in a journal, along with unique counter index and timestamp values. Later, when data records stored in the cloud-based storage resource are to be replicated at the local computing device, data records corresponding to the sequential counter indices listed in the journal are sent to the local computing device. Only those data records which correspond to blocks of uninterrupted sequential counter indices are transmitted to the local computing device, thereby ensuring consistency of the replicated data.

Claims

1. A method for replicating data records stored in a cloud-based storage resource at a local computing device, the method comprising:

receiving a read state counter index from the local computing device;

identifying, with reference to a journal, a plurality of journal records, each of which (a) is associated with a counter index that forms a sequence starting with the read state counter index, and (b) corresponds to a data record stored in the cloud-based storage resource;

identifying an unavailable journal record that interrupts the sequence and that is associated with an unavailable counter index;

transmitting a plurality of data records to the local computing device, wherein the transmitted plurality of data records correspond to the identified plurality of journal records; and

transmitting the unavailable counter index to the local computing device.

2. The method of claim 1, wherein the unavailable counter index is transmitted with instructions to replace the read state counter index with the unavailable counter index.

3. The method of claim 1, wherein the plurality of transmitted data records are transmitted to the local computing device in batches corresponding to a maximum page size.

4. The method of claim 1, further comprising queueing the plurality of transmitted data records in a cache before transmitting them to the local computing device.

5. The method of claim 1, wherein identifying the plurality of journal records further comprises:

identifying a subsequent journal record associated with a counter index that is greater than the unavailable counter index, such that a gap exists in the sequence; and

determining an age of the gap.

6. The method of claim 1, wherein identifying the plurality of journal records further comprises:

identifying a subsequent journal record associated with a counter index that is greater than the unavailable counter index, such that a gap exists in the sequence; and

determining a difference between a timestamp associated with the subsequent journal record and a current time.

7. The method of claim 1, wherein the cloud-based storage resource comprises a plurality of geographically distributed storage devices.

8. The method of claim 1, further comprising comparing the unavailable counter index with a last available counter index maintained by a counter generation module.

9. The method of claim 1, further comprising:

identifying a subsequent journal record associated with a counter index that is greater than the unavailable counter index, such that a gap exists in the sequence; and

invoking a full synchronization procedure in response to determining that the gap corresponds to a quantity of counter indices that exceeds a full synchronization threshold.

10. The method of claim 1, wherein the journal includes a record timestamp associated with each of the counter indices, the record timestamp corresponding to a time at which the associated counter index was generated.

11. A data replication system comprising:

a counter generation module comprising a log entry that correlates a user, a counter index, and a creation timestamp; and

a cloud storage gateway that comprises

a plurality of cloud storage devices that form a cloud storage resource, and that collectively store a plurality of data records,

a journal that comprises a plurality of journal records, each of which correlates (a) a network address that identifies where a particular data record is stored in the cloud storage resource with (b) a particular counter index associated with a particular operation having been performed on the particular data record, and

a cloud storage administration module configured to generate a list of journal records that corresponds to a plurality of operations recorded in the journal, the list comprising a continuous sequence of counter indices that terminates before the counter index included in the counter generation module log entry.

12. The data replication system of claim 11, further comprising a communication interface configured transmit the list of journal records to a client computing device.

13. The data replication system of claim 11, wherein the log entry includes a token that identifies the user.

14. The data replication system of claim 11, wherein the cloud storage gateway further comprises a journal administration module configured to request a new sequential counter index from the counter generation module in response to receipt of data that is to be stored in the cloud storage resource.

15. The data replication system of claim 11, wherein the cloud storage administration module is further configured to identify a nonsequential journal record associated with a counter index that is greater than the counter index included in the counter generation module log entry.

16. The data replication system of claim 11, wherein each of the journal records further correlates the network address with a particular creation timestamp corresponding to a time that the particular counter index was generated by the counter generation module.

17. A computer program product that, when executed by one or more processors, causes a data storage process to be carried out, the data storage process comprising:

receiving new data to be stored in a cloud-based storage resource;

in response to receiving the new data, generating a new journal record that comprises a new counter index and a timestamp, wherein the new counter index is extracted from a user-specific log maintained by a counter generation module;

saving an incremented counter index in the user-specific log; and

saving the new data in the cloud-based storage resource at a location identified by a network address;

wherein the new journal record correlates the new counter index with the network address, and wherein the new counter index is a sequential addition to an existing plurality of counter indices stored in a corresponding plurality of existing journal records that collectively represent a sequence of data modification operations performed on data stored in the cloud-based storage resource.

18. The computer program product of claim 17, wherein the timestamp corresponds to a time at which the new data was saved in the cloud-based storage resource.

19. The computer program product of claim 17, wherein:

the new data is saved in the cloud-based storage resource at a plurality of locations identified by a corresponding plurality of network addresses; and

the new journal record correlates the new counter index with the plurality of network addresses.

20. The computer program product of claim 17, wherein the new data is received by a cloud storage gateway that is in communication with the counter generation module.