Write cache and write-hole recovery in distributed raid over shared multi-queue storage devices

• US 9,842,084 B2
• Filed: 02/06/2017
• Issued: 12/12/2017
• Est. Priority Date: 04/05/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: E8 Storage Systems Ltd. (Amazon.com, Inc.)
• Inventors: Friedman, Alex, Liakhovetsky, Alex, Divinsky, Yoni, Budilovsky, Evgeny, Gonshorovitz, Itay, Pinhas, Barak
• Primary Examiner(s): Fan, Hua
• Application Number:
  US15/424,921
• Publication Number:
  US 20170286362A1
• Time in Patent Office: 309 Days
• CPC Class Codes: G06F 15/167 using a common memory, e.g. mailbox
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching

Abstract

A method for data storage includes, in a system that includes multiple servers, multiple multi-queue storage devices and at least one storage controller that communicate over a network, storing data by the servers in a storage process that (i) caches the data in a cache memory that is accessible to the servers and to the storage devices and (ii) de-stages the cached data from the cache memory to the storage devices. A progress of the storage process is continually journaled, by the servers, in a plurality of server-specific journals residing in the cache memory. In response to a failure occurring during the storage process, the data is recovered by the storage controller based on the server-specific journals.

Claims

1. A method for data storage, comprising:

in a system that comprises multiple servers, multiple multi-queue storage devices and at least one storage controller that communicate over a network, storing data by the servers in stripes in a storage process that (i) caches the data in a cache memory that is accessible to the servers and to the storage devices and (ii) de-stages the cached data from the cache memory to the storage devices;

continually journaling a progress of the storage process, by the servers, in a plurality of server-specific journals residing in the cache memory; and

in response to a failure associated with a given server occurring during the storage process, collecting from the server-specific journals entries relating to one or more stripes accessed by the given server, and recovering the data by the storage controller based on the collected entries of the server-specific journals,

wherein the stripes each consist of N data blocks and c redundancy blocks, and wherein recovering the data comprises reconstructing the stripes based on fewer than N+c of the server-specific journals, or based on fewer than N+c server-specific portions of the cache memory.

2. The method according to claim 1, wherein storing the data and journaling the progress are performed by remote direct memory access, without triggering or running code on any CPU of the storage controller.

3. The method according to claim 1, wherein recovering the data comprises obtaining at least some of the data or the redundancy from one or more of the servers, and reconstructing the stripes using the data or the redundancy obtained from the one or more of the servers.

4. The method according to claim 1, wherein recovering the data comprises performing a simulated run of a recovery process that recovers the data, and, only upon success of the simulated run, performing an actual run of the recovery process.

5. The method according to claim 1, wherein storing the data comprises storing a first copy of the cache memory on one or more of the servers, and storing a second copy of the cache memory on the storage controller or on one or more of the storage devices.

6. The method according to claim 1, wherein caching of the data in the cache memory and subsequent de-staging of the data to the storage devices comprises increasing an effective queue depth of a workload that generates the data.

7. The method according to claim 1, wherein the cache memory resides on the storage devices, such that storing the data comprises transferring the data over the network only once.

8. A system for data storage, comprising:

multiple servers, which are configured to store data in stripes over a network on multiple multi-queue storage devices in a storage process that (i) caches the data in a cache memory that is accessible to the servers and to the storage devices and (ii) de-stages the cached data from the cache memory to the storage devices, and to continually journal a progress of the storage process in a plurality of server-specific journals residing in the cache memory; and

at least one storage controller, which is configured, in response to a failure associated with a given server occurring during the storage process, to collect from the server-specific journals entries relating to one or more stripes accessed by the given server, and to recover the data based on the collected entries of the server-specific journals,

wherein the stripes each consist of N data blocks and c redundancy blocks, and wherein the storage controller is configured to reconstruct the stripes based on fewer than N+c of the server-specific journals, or based on fewer than N+c server-specific portions of the cache memory.

9. The system according to claim 8, wherein the servers are configured to store the data and to journal the progress by remote direct memory access, without triggering or running code on any CPU of the storage controller.

10. The system according to claim 8, wherein the storage controller is configured to obtain at least some of the data or the redundancy from one or more of the servers, and to reconstruct the stripes using the data or the redundancy obtained from the one or more of the servers.

11. The system according to claim 8, wherein the storage controller is configured to perform a simulated run of a recovery process that recovers the data, and, only upon success of the simulated run, to perform an actual run of the recovery process.

12. The system according to claim 8, wherein the servers are configured to store a first copy of the cache memory on one or more of the servers, and to store a second copy of the cache memory on the storage controller or on one or more of the storage devices.

13. The system according to claim 8, wherein, by caching the data in the cache memory and subsequent de-staging the data to the storage devices, the servers are configured to increase an effective queue depth of a workload that generates the data.

14. The system according to claim 8, wherein the cache memory resides on the storage devices, such that the servers are configured to store the data by transferring the data over the network only once.