Snapshots and thin-provisioning in distributed storage over shared storage devices

• US 9,519,666 B2
• Filed: 04/28/2015
• Issued: 12/13/2016
• Est. Priority Date: 11/27/2014
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: E8 Storage Systems Ltd. (Amazon.com, Inc.)
• Inventors: Friedman, Alex
• Primary Examiner(s): Owyang, Michelle
• Application Number:
  US14/697,653
• Publication Number:
  US 20160154834A1
• Time in Patent Office: 595 Days
• Field of Search: 707705-780
• US Class Current: 1/1
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/2246 Trees, e.g. B+trees
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/278 Data partitioning, e.g. horizontal or vertical partitioning
• CPC Class Codes: G06F 16/86 Mapping to a database
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/953 Querying, e.g. by the use of web search engines
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A method for data storage includes, in a system that includes one or more storage controllers, one or more servers and one or more storage devices, defining one or more data structures that represent thinly-provisioned user volumes used by the servers in storing data on the storage devices. One or more of the data structures are shared among the storage controllers and the servers. One or more of the user volumes are accessed by the servers, using the shared data structures and without executing code on the storage controllers.

Claims

1. A method for data storage, comprising:

in a system that comprises one or more storage controllers, multiple servers and multiple storage devices that communicate over a network, running user applications on the servers;

storing, in a memory of the one or more storage controllers, a data structure, which (i) represents thinly-provisioned user volumes used by the servers in storing data on the storage devices, and (ii) specifies for the user volumes a logical-to-physical mapping between logical addresses and physical addresses of the storage devices;

generating, in a given server that is configured to run one or more of the user applications, storage commands that access one or more of the user volumes, by performing in the given server;

(i) retrieving at least a part of the logical-to-physical mapping from the memory of the one or more storage controllers to the given server, using direct remote access to the memory of the one or more storage controllers and without executing code on the one or more storage controllers;

(ii) translating the logical addresses accessed by the storage commands into the respective physical addresses, using the retrieved at least part of the logical-to-physical mapping; and

(iii) generating the storage commands that specify the translated physical addresses; and

sending the storage commands, which specify the physical addresses, from the given server that is configured to run the one or more of the user applications directly to one or more of the storage devices, without executing code on the one or more storage controllers.

2. The method according to claim 1, wherein the data structure comprises one or more tree structures that map the logical addresses in the user volumes onto the physical addresses in the storage devices.

3. The method according to claim 2, wherein storing the data structure further comprises indicating in the data structure which nodes of the one or more tree structures are referenced by more than a single user volume.

4. The method according to claim 2, wherein storing the data structure comprises defining back-references that point from nodes of the one or more tree structures to the user volumes that access the nodes.

5. The method according to claim 1, wherein generating the storage commands comprises detecting, using the data structure, that a given user volume has a clone, and performing a copy-on-write operation when writing to the given user volume.

6. The method according to claim 1, wherein storing the data structure comprises defining in the shared data structure back-references that map the physical addresses in the storage devices back to the logical addresses in the user volumes.

7. The method according to claim 1, wherein storing the data structure comprises updating the data structure in response to creation of a clone of a user volume.

8. The method according to claim 1, wherein storing the data structure comprises updating the data structure in response to a write to a previously-unallocated address in a user volume.

9. The method according to claim 1, and comprising generating in at least one of the servers a list of commands for updating reference counts associated with one or more of the user volumes, and sending the list to at least one of the storage controllers for background execution.

10. The method according to claim 1, wherein storing the data structure comprises caching in the given server a local copy of at least a portion of the data structure, and wherein generating the storage commands comprises accessing a given user volume by the given server using the cached local copy.

11. The method according to claim 10, and comprising, upon accessing the local copy, verifying that the local copy is consistent with the data structure, and otherwise updating the local copy by accessing the data structure stored in the storage controllers.

12. The method according to claim 1, and comprising moving multiple blocks of data from first addresses to second addresses in the storage devices using the data structure.

13. The method according to claim 12, wherein moving the multiple blocks comprises performing an XCOPY or an Offloaded Data Transfers (ODX) operation.

14. The method according to claim 1, and comprising, on each of the storage devices, queuing the storage commands received directly from the servers in multiple server-specific queues, such that each server-specific queue queues the storage commands from a respective one of the servers.

15. A computing system, comprising:

multiple physical servers, which are configured to run user applications and to communicate with multiple storage devices over a network; and

one or more physical storage controllers, which have a memory and are configured to store in the memory a data structure that (i) represents thinly-provisioned user volumes used by the servers in storing data on the storage devices, and (ii) specifies for the user volumes a logical-to-physical mapping between logical addresses and physical addresses of the storage devices,

wherein a given physical server, which is configured to run one or more of the user applications, is further configured to generate storage commands that access one or more of the user volumes, by performing;

(i) retrieving at least a part of the logical-to-physical mapping from the memory of the one or more storage controllers to the given server, using direct remote access to the memory of the one or more storage controllers and without executing code on the one or more storage controllers;

(ii) translating the logical addresses accessed by the storage commands into the respective physical addresses, using the retrieved at least part of the logical-to-physical mapping; and

(iii) generating the storage commands that specify the translated physical addresses,

and wherein the given server that is configured to run the one or more of the user applications is further configured to send the storage commands, which specify the physical addresses, directly from the given server to one or more of the storage devices, without executing code on the one or more storage controllers.

16. The system according to claim 15, wherein the data structure comprises one or more tree structures that map the logical addresses in the user volumes onto the physical addresses in the storage devices.

17. The system according to claim 16, wherein the data structure is further indicative of which nodes of the one or more tree structures are referenced by more than a single user volume.

18. The system according to claim 16, wherein the one or more storage controllers are configured to define back-references that point from nodes of the one or more tree structures to the user volumes that access the nodes.

19. The system according to claim 15, wherein the given physical server is configured to detect, using the data structure, that a given user volume has a clone, and to perform a copy-on-write operation when writing to the given user volume.

20. The system according to claim 15, wherein the one or more storage controllers are configured to define back-references that map the physical addresses in the storage devices back to the logical addresses in the user volumes.

21. The system according to claim 15, wherein the one or more storage controllers are configured to update the data structure in response to creation of a clone of a user volume.

22. The system according to claim 15, wherein the one or more storage controllers are configured to update the data structure in response to a write to a previously-unallocated address in a user volume.

23. The system according to claim 15, wherein the given physical server is configured to generate a list of commands for updating reference counts associated with one or more of the user volumes, and to send the list to at least one of the storage controllers for background execution.

24. The system according to claim 15, wherein the given physical server is configured to cache a local copy of at least a portion of the data structure, and to access a given user volume using the cached local copy.

25. The system according to claim 24, wherein, upon accessing the local copy, the given physical server is configured to verify that the local copy is consistent with the data structure, and otherwise to update the local copy by accessing the data structure stored in the storage controllers.

26. The system according to claim 15, wherein the given physical server is configured to move a bulk of data from first addresses to second addresses in the storage devices using the data structure.

27. The system according to claim 26, wherein the given physical server is configured to move the multiple blocks by performing an XCOPY or an Offloaded Data Transfers (ODX) operation.

28. A server, comprising:

a physical network interface for communicating over a communication network with multiple storage devices and with one or more storage controllers; and

a physical processor, which is configured to run one or more user applications, and to generate storage commands that access one or more thinly-provisioned user volumes on the storage devices, by;

(i) retrieving to the server, using direct remote access to a memory of the one or more storage controllers and without executing code on the one or more storage controllers, at least a part of a logical-to-physical mapping between logical addresses and physical addresses of the storage devices, wherein the logical-to-physical mapping is stored in the memory in a data structure that (a) represents the thinly-provisioned user volumes, and (b) specifies the logical-to-physical mapping for the user volumes;

(ii) translating the logical addresses accessed by the storage commands into the respective physical addresses, using the retrieved at least part of the logical-to-physical mapping; and

(iii) generating the storage commands that specify the translated physical addresses,

wherein the physical processor that is configured to run the one or more user applications is further configured to send the storage commands, which specify the physical addresses, directly to one or more of the storage devices without executing code on the one or more storage controllers.