NVRAM caching and logging in a storage system

• US 8,898,388 B1
• Filed: 01/23/2014
• Issued: 11/25/2014
• Est. Priority Date: 01/08/2014
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Kimmel, Jeffrey S.
• Primary Examiner(s): Blair, April Y
• Assistant Examiner(s): Thammavong, Prasith
• Application Number:
  US14/162,174
• Time in Patent Office: 306 Days
• Field of Search: 711/114
• US Class Current: 711/114
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 12/0246 in block erasable memory, e.g. flash memory
• CPC Class Codes: G06F 12/0802 Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
• CPC Class Codes: G06F 12/0804 with main memory updating G06F12/0806 takes precedence
• CPC Class Codes: G06F 12/0815 Cache consistency protocols
• CPC Class Codes: G06F 12/0866 for peripheral storage systems, e.g. disk cache
• CPC Class Codes: G06F 12/0891 using clearing, invalidating or resetting means
• CPC Class Codes: G06F 12/0893 Caches characterised by their organisation or structure
• CPC Class Codes: G06F 12/0895 of parts of caches, e.g. directory or tag array
• CPC Class Codes: G06F 2212/1032 Reliability improvement, data loss prevention, degraded operation etc
• CPC Class Codes: G06F 2212/214 Solid state disk
• CPC Class Codes: G06F 2212/221 Static RAM
• CPC Class Codes: G06F 2212/222 Non-volatile memory
• CPC Class Codes: G06F 2212/2228 Battery-backed RAM
• CPC Class Codes: G06F 3/0614 Improving the reliability of storage systems
• CPC Class Codes: G06F 3/0655 Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
• CPC Class Codes: G06F 3/0679 Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
• CPC Class Codes: G06F 3/0688 Non-volatile semiconductor memory arrays
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

In one embodiment, non-volatile random access memory (NVRAM) caching and logging delivers low latency acknowledgements of input/output (I/O) requests, such as write requests, while avoiding loss of data. Write data may be stored in a portion of an NVRAM configured as, e.g., a persistent write-back cache, while parameters of the request may be stored in another portion of the NVRAM configured as one or more logs, e.g., NVLogs. The write data may be organized into separate variable length blocks or extents and “written back” out-of-order from the write back cache to storage devices, such as solid state drives (SSDs). The write data may be preserved in the write-back cache until each extent is safely and successfully stored on SSD (i.e., in the event of power loss), or operations associated with the write request are sufficiently logged on NVLog.

Claims

1. A method comprising:

receiving at a storage system a plurality of input/output (I/O) requests each having a related data and a parameter;

in response to receiving each of the plurality of I/O requests, returning a respective acknowledgement;

organizing the related data of the plurality of I/O requests into a plurality of extents, the organizing to include combining the related data of two or more first I/O requests of the plurality of I/O requests into a first extent, forming the related data of a second I/O request of the plurality of I/O requests into a second extent, and splitting the related data of a third I/O request of the plurality of I/O requests into two or more third extents;

storing the plurality of extents in a first order in a persistent memory cache of the storage system;

writing the plurality of extents in a second order from the persistent memory cache to the one or more solid state drives (SSDs) attached to the storage system, wherein the second order differs from the first order;

receiving a confirmation that each of the plurality of extents was successfully stored in the one or more SSDs; and

deleting each of the plurality of extents in the persistent memory cache when each respective extent is confirmed to be successfully stored in the one or more SSDs.

2. The method of claim 1 wherein the plurality of extents have variable size, such that at least one extent has a different size from at least one other extent.

3. The method of claim 1 wherein the two or more first I/O requests have related data that is within a continuous logical block address range.

4. The method of claim 1 further comprising atomically storing each extent of the plurality of extents in the one or more SSDs.

5. The method of claim 1 further comprising:

recording the parameter of each I/O request of the plurality of I/O requests in a respective entry of a first non-volatile log of the storage system, thereby avoiding loss of the parameter when power is lost to the one or more SSDs.

6. The method of claim 5 wherein each entry of the first non-volatile log includes a pointer to the persistent memory cache.

7. The method of claim 6 further comprising:

recording an offset included in the parameter of each I/O request in a respective entry of a second non-volatile log of the storage system; and

storing the entries of the second non-volatile log on the one of more SSDs.

8. The method of claim 7 wherein deleting each of the plurality of extents in the persistent cache occurs in response to recording the pointer to the persistent memory cache in a third non-volatile log of the storage system.

9. The method of claim 7 wherein the plurality of extents is stored on a first subset of the one or more SSDs, wherein the entries of the second non-volatile log are stored on a second subset of the one or more SSDs, and wherein the first subset is different than the second subset.

10. The method of claim 1 wherein the third I/O request is split based on a maximum extent size.

11. A system comprising:

a first storage array having one or more solid state drives (SSDs);

a first node connected to a non-volatile memory via a system bus, the first node attached to the first storage array,

the first node configured to receive a plurality of input/output (I/O) requests each having a related data and a parameter,

the first node further configured to acknowledge each of the plurality of I/O requests,

the first node further configured to organize the related data of the plurality of I/O requests into a plurality of extents, by combining the related data of two or more first I/O requests of the plurality of I/O requests into a first extent, forming the related data of a second I/O request of the plurality of I/O requests into a second extent, and splitting the related data of a third I/O request of the plurality of I/O requests into two or more third extents,

the first node further configured to store the plurality of extents in a cache in a first order, the cache stored in the non-volatile memory,

the first node further configured to write the plurality of extents in a second order from the cache to the first storage array, wherein the second order differs from the first order,

the first node further configured to receive a confirmation that each of the plurality of extents was successfully stored in the first storage array, and

the first node further configured to delete the plurality of extents in the cache when each respective extent is confirmed to be successfully stored in the first storage array.

12. The system of claim 11 wherein the plurality of extents have variable size, such that at least one extent has a different size from at least one other extent.

13. The system of claim 11 wherein the two or more first I/O requests have related data that is first node is within a continuous logical block address range.

14. The system of claim 11 wherein the first node is further configured to atomically store each extent of the plurality of extents in the first array.

15. The system of claim 11 wherein the first node is further configured to record the parameter of each I/O request of the plurality of I/O requests in a respective entry of a first non-volatile log stored in the non-volatile memory, thereby avoiding loss of the parameter when power is lost to the first node.

16. The system of claim 15 wherein each entry of the non-volatile log includes a pointer to the cache.

17. The system of claim 15 further comprising:

a second node configured to receive an offset included in the parameter of a given I/O request from the first node,

the second node further configured to record the offset in an entry of a second non-volatile log, and

a third node further configured to store the entry of the second non-volatile log on a second storage array attached to the second node, the second storage array having one or more SSDs.

18. The system of claim 17 wherein the third node is different from the first node and the second node.

19. The system of claim 15 wherein the third I/O request is split based on a maximum extent size.

20. A system comprising:

a first storage array having one or more solid state drives (SSDs);

a first node having a non-volatile memory connected to a processor via a bus, the first node attached to the first storage array; and

a storage I/O stack executing on the processor of the first node, the storage I/O stack when executed operable to;

receive a plurality of input/output (I/O) requests each having a related data and a parameter,

organize the related data of the plurality of I/O requests into units by combining the related data of two or more first I/O requests of the plurality of I/O requests into a single first unit, forming related data of a second I/O request of the plurality of I/O requests into a second unit, and splitting the related data of a third I/O request of the plurality of I/O requests into two or more third units,

atomically store the plurality of units in the non-volatile memory in a first order,

pass the plurality of units in a first path of the storage I/O stack to the first storage array for storage on the one or more SSDs in a second order, wherein the second order differs from the first order,

pass the related parameters in a second path of the storage I/O stack, wherein the first path differs from the second path, for storage in a second storage array attached to a second node, and

receive confirmation that each unit is stored on the one or more SSDs in the first storage array prior to marking the respective unit released in the non-volatile memory.