Shadowing storage gateway

• US 9,294,564 B2
• Filed: 06/30/2011
• Issued: 03/22/2016
• Est. Priority Date: 06/30/2011
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Sorenson, James Christopher III, Lin, Yun, Salyers, David C., Khetrapal, Ankur
• Primary Examiner(s): Vu, Viet
• Assistant Examiner(s): Edwards, James
• Application Number:
  US13/174,505
• Publication Number:
  US 20130007219A1
• Time in Patent Office: 1,727 Days
• Field of Search: 709/236
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/1724 Details of de-fragmentation performed by the file system saving storage space on storage systems G06F3/0608; management of blocks in storage devices G06F3/064
• CPC Class Codes: G06F 9/44 Arrangements for executing specific programs
• CPC Class Codes: H04L 65/40 Support for services or applications
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/56 Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching

Abstract

Methods, apparatus, and computer-accessible storage media for shadowing data stored on a local store to a remote store provided by a service provider. A gateway may be configured as a shadowing gateway on a customer network in response to receiving configuration information. The shadowing gateway may receive reads and writes to the local store. The gateway passes the requests to the local store, and also uploads write data indicated by the writes to the service provider to update a snapshot of the local store maintained by the service provider on the remote store. The write data may be buffered to a write log for uploading, and may be uploaded as blocks according to a block storage format used by the service provider. The shadowing process may be transparent to processes on the customer network. The shadowed data may be used to recover data on the local store.

Claims

1. A method, comprising:

initiating, by a storage gateway on a customer network, a process that registers the storage gateway with a remote service provider that provides a remote data store to customers of the service provider;

receiving, by the storage gateway, configuration information that specifies that the storage gateway is to serve as a shadowing gateway on the customer network that shadows data stored on a local data store within the customer network to the remote data store, wherein the local data store serves as a primary data store for the customer data;

the storage gateway configuring as the shadowing gateway on the customer network in response to the configuration information;

uploading at least a portion of the data stored on the local data store to the service provider to make a snapshot of the primary data store on the remote data store consistent with the primary data store on the customer network;

receiving, by the storage gateway from one or more processes on the customer network, read requests and write requests directed to the primary data store, wherein the storage gateway receives the read and write requests from the one or more processes on the customer network via a block storage protocol;

for the read requests, passing the read requests to the primary data store; and

for the write requests;

passing the write requests to the primary data store; and

sending write data indicated by the write requests to the service provider to update the snapshot of the primary data store on the remote data store with the write data.

2. The method as recited in claim 1, wherein receiving the read requests and the write requests, passing the read requests and the write to the primary data store, and sending write data indicated by the write requests to the service provider are performed by the storage gateway concurrently with uploading the at least a portion of the data stored on the local data store to the service provider to make the snapshot.

3. The method as recited in claim 1, further comprising:

the storage gateway entering a pass-through mode in which said sending write data indicated by the write requests to the service provider is suspended for a period;

the storage gateway exiting the pass-through mode; and

subsequent to said exiting the pass-through mode, the storage gateway uploading at least a portion of the data stored on the local data store to the service provider to make the snapshot of the primary data store consistent with the primary data store.

4. The method as recited in claim 1, wherein said sending the write data to the service provider is performed asynchronously to said passing the read requests and the write requests to the local data store and to returning responses to the read requests and the write requests from the local data store to the one or more processes.

5. The method as recited in claim 1, wherein the local data store comprises one or more storage devices, the method further comprising restoring at least some data on at least one of the one or more storage devices from the snapshot of the primary data store on the remote data store.

6. The method as recited in claim 1, wherein said sending write data indicated by the write requests to the service provider comprises:

buffering the write data to a write log; and

uploading, by an upload component of the storage gateway, the buffered write data to a corresponding upload component of the service provider.

7. The method as recited in claim 1, wherein the remote data store is provided by a storage service of the service provider, and wherein said sending write data indicated by the write requests to the service provider comprises the storage gateway sending the write data according to an interface to the storage service.

8. The method as recited in claim 1, wherein the one or more processes on the customer network initiate the read requests and the write requests to one or more data ports exposed by the storage gateway on the customer network.

9. A device, comprising:

at least one processor; and

a memory comprising program instructions, wherein the program instructions are executable by the at least one processor to implement a gateway process operable to;

upload at least a portion of data stored on a local data store on a customer network to a service provider to make a snapshot of the local data store on a remote data store consistent with the local data store on the customer network;

receive, from one or more processes on the customer network, read requests and write requests directed to the local data store on the customer network, wherein the gateway process receives the read and write requests from the one or more processes on the customer network via a block storage protocol;

pass the read requests and the write requests to the local data store; and

send write data indicated by the write requests to the service provider to update the snapshot of the local data store maintained on the remote data store by the service provider with the write data.

10. The device as recited in claim 9, wherein the gateway process is operable to perform said sending the write data to the service provider asynchronously to said passing the read requests and the write requests to the local data store and to returning responses to the read requests and the write requests from the local data store to the initiating processes.

11. The device as recited in claim 9, wherein the gateway process is further operable to initiate a process that registers the gateway process with the service provider.

12. The device as recited in claim 11, wherein the gateway process is further operable to configure as a shadowing gateway according to configuration information received from the service provider, wherein a shadowing gateway shadows data stored on a local data store to a remote data store.

13. The device as recited in claim 9, wherein the gateway process is further operable to restore at least some data in the local data store from the snapshot of the local data store.

14. The device as recited in claim 9, wherein, to send write data indicated by the write requests to a service provider, the gateway process is further operable to:

buffer the write data to a write log; and

upload the buffered write data from the write log to the service provider.

15. The device as recited in claim 14, wherein said uploading is performed according to a data deduplication technique.

16. The device as recited in claim 9, wherein the remote data store is provided by a storage service of the service provider, and wherein, to send write data to the service provider, the gateway process is further operable to send the write data in a format specified by an interface of the storage service.

17. The device as recited in claim 9, wherein the snapshot of the local data store is maintained on the remote data store by the service provider in a block format, and wherein, to send write data indicated by the write requests to a service provider, the gateway process is further operable to upload blocks of data modified by the write data to the service provider.

18. The device as recited in claim 17, wherein said uploading the blocks is performed according to a data deduplication technique.

19. The device as recited in claim 9, wherein the one or more processes on the customer network initiate the read requests and the write requests to one or more data ports exposed by the gateway process on the customer network.

20. A non-transitory computer-accessible storage medium storing program instructions computer-executable to implement a gateway process operable to:

configure as a shadowing gateway on a customer network according to configuration information received from a service provider, wherein a shadowing gateway shadows data stored on a local data store to a remote data store maintained by the service provider;

upload at least a portion of the data stored on the local data store to the service provider to make a snapshot of the local data store on the remote data store consistent with the local data store;

wherein, to shadow data stored on the local data store to the remote data store, the gateway process is operable to;

receive write requests directed to the local data store from the one or more processes on the customer network, wherein the gateway process receives the write requests from the one or more processes on the customer network via a block storage protocol; and

upload write data indicated by the write requests to the service provider to update the snapshot of the local data store maintained on the remote data store by the service provider with the write data.

21. The non-transitory computer-accessible storage medium as recited in claim 20, wherein the gateway process is further operable to receive and process read requests directed to the local data store.

22. The non-transitory computer-accessible storage medium as recited in claim 21, wherein the gateway process is further operable to pass the write requests to the local data store, and wherein the gateway process is operable to perform said uploading the write data to the service provider asynchronously to said passing the write requests to the local data store and to said processing the read requests.

23. The non-transitory computer-accessible storage medium as recited in claim 21, wherein the one or more processes on the customer network initiate the read requests and the write requests to one or more data ports exposed by the gateway process on the customer network.

24. The non-transitory computer-accessible storage medium as recited in claim 20, wherein, to upload write data indicated by the write requests to the service provider, the gateway process is further operable to:

buffer the write data to a write log; and

upload the buffered write data from the write log to the service provider according to a data deduplication technique.

25. The non-transitory computer-accessible storage medium as recited in claim 20, wherein, to upload write data to the service provider, the gateway process is further operable to upload the write data according to a Web service interface of the service provider.

26. The non-transitory computer-accessible storage medium as recited in claim 20, wherein, to upload write data to the service provider, the gateway process is further operable to upload blocks of data modified by the write data to the service provider.

27. The non-transitory computer-accessible storage medium as recited in claim 26, wherein said uploading is performed according to a data deduplication technique.