Distributed storage over shared multi-queued storage device

• US 9,800,661 B2
• Filed: 07/13/2015
• Issued: 10/24/2017
• Est. Priority Date: 08/20/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: E8 Storage Systems Ltd. (Amazon.com, Inc.)
• Inventors: Ori, Zivan
• Primary Examiner(s): WANG, LIANG CHE A
• Application Number:
  US14/797,272
• Publication Number:
  US 20160057224A1
• Time in Patent Office: 834 Days
• Field of Search: 709202, 709203, 709223, 709226, 709245
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching

Abstract

A method for data storage includes, in a system that includes one or more storage controllers, multiple servers and multiple multi-queue storage devices, assigning in each storage device server-specific queues for queuing data-path storage commands exchanged with the respective servers. At least some of the data-path storage commands are exchanged directly between the servers and the storage devices, not via the storage controllers, to be queued and executed in accordance with the corresponding server-specific queues.

Claims

1. A method for data storage, comprising:

in a system that comprises one or more storage controllers, multiple servers and multiple multi-queue storage devices, assigning in each storage device server-specific queues for queuing data-path storage commands exchanged with the respective servers; and

exchanging at least some of the data-path storage commands directly between the servers and the storage devices, not via the storage controllers, to be queued and executed in accordance with the corresponding server-specific queues,

wherein exchanging the data-path storage commands comprises accessing the storage devices by;

locally-caching in a given server, or looking-up in the storage controllers by the given server, at least a portion of a volume map that specifies user volumes for use by the servers; and

accessing the storage devices by the given server, by querying the locally-cached or looked-up volume map.

2. The method according to claim 1, wherein exchanging the data-path storage commands comprises queuing the data-path storage commands in the given server in one or more storage-device-specific queues, which respectively correspond to one or more of the storage devices accessed by the given server.

3. The method according to claim 1, wherein exchanging the data-path storage commands comprises running in the given server a respective storage agent that communicates directly with the server-specific queues assigned to the given server in the storage devices.

4. The method according to claim 1, wherein exchanging the data-path storage commands comprises translating in the given server between virtual and physical storage addresses, and specifying the data-path storage commands in the given server using the physical storage addresses.

5. The method according to claim 4, wherein translating between the virtual and the physical storage addresses comprises locally storing in the given server a mapping that maps each virtual address range to multiple redundant ranges of the physical storage addresses located on two or more of the storage devices, and translating between the virtual and the physical storage addresses by querying the mapping.

6. The method according to claim 1, and comprising granting to the given server a reservation on at least a portion of a user volume only in response to ensuring that all other servers that are attached to the user volume have temporarily stopped accessing the user volume and have been provided with an up-to-date state of the reservation.

7. The method according to claim 1, and comprising, in response to an access by the given server to a logical address that is not yet allocated a respective physical address, allocating the respective physical address by the storage controllers.

8. The method according to claim 1, and comprising, in response to replacement of a failed storage device, rebuilding on a new storage device data that was stored on the failed storage device, by copying redundant copies of the data from one or more other storage devices to the new storage device.

9. A computing system, comprising:

one or more storage controllers, which are configured to assign in each of multiple multi-queue storage devices server-specific queues for queuing data-path storage commands exchanged with multiple servers; and

multiple processors, which are comprised in the respective servers and are configured to exchange at least some of the data-path commands directly between the servers and the storage devices, not via the storage controllers, to be queued and executed in accordance with the corresponding server-specific queues,

wherein a given processor in a given server is configured to locally-cache, or look-up in the storage controllers, at least a portion of a volume map that specifies user volumes for use by the servers, and to access the storage devices by querying the locally-cached or looked-up volume map.

10. The system according to claim 9, wherein the given processor is configured to queue the data-path storage commands in one or more storage-device-specific queues, which respectively correspond to one or more of the storage devices accessed by the given server.

11. The system according to claim 9, wherein the given processor is configured to translate between virtual and physical storage addresses, and to specify the data-path storage commands exchanged with the storage devices using the physical storage addresses.

12. The system according to claim 11, wherein the given processor is configured to locally-store in the given server a mapping that maps each virtual address range to multiple redundant ranges of the physical storage addresses located on two or more of the storage devices, and to translate between the virtual and the physical storage addresses by querying the mapping.

13. The system according to claim 9, wherein the storage controllers are configured to grant to the given server a reservation on at least a portion of a user volume only in response to ensuring that all other servers that are attached to the user volume have temporarily stopped accessing the user volume and have been provided with an up-to-date state of the reservation.

14. The system according to claim 9, wherein, in response to an access by the given server to a logical address that is not yet allocated a respective physical address, the storage controllers are configured to allocate the respective physical address.

15. The system according to claim 9, wherein, in response to replacement of a failed storage device, the storage controllers are configured to rebuild on a new storage device data that was stored on the failed storage device, by copying redundant copies of the data from one or more other storage devices to the new storage device.