GLOBAL IN-LINE EXTENT-BASED DEDUPLICATION

• US 20150193169A1
• Filed: 01/21/2014
• Published: 07/09/2015
• Est. Priority Date: 01/08/2014
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Sundaram, Rajesh, Kimmel, Jeffrey S., Lewis, Blake H.
• Granted Patent: US 9,152,335 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0688 Non-volatile semiconductor memory arrays
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

In one embodiment, a layered file system of a storage input/output (I/O) stack executes on one or more nodes of a cluster. The layered file system includes a flash-optimized, log-structured layer configured to provide sequential storage of data and metadata (i.e., a log-structured layout) on solid state drives (SSDs) of storage arrays in the cluster to reduce write amplification, while leveraging a data de-duplication feature of the storage I/O stack. An extent store layer of the file system performs and maintains mappings of the extent keys to SSD storage locations, while a volume layer of the file system performs and maintains mappings of the LUN offset ranges to the extent keys. Separation of the mapping functions between the volume and extent store layers enables different volumes with different offset ranges to reference a same extent key (and thus a same extent).

Claims

1. A method comprising:

receiving a first write request directed towards a first logical unit (LUN), the first write request having write data and having metadata that includes a first logical block address (LBA), the first write request processed at a node of a storage system, the storage system attached to a storage array of solid state drives (SSDs);

applying a hash function to the write data to generate a first hash value;

selecting an extent store from a plurality of extent stores based on the first hash value;

storing the write data in a segment of the selected extent store, wherein the selected extent store spans a set of SSDs of the storage array, wherein a key is formed from the first hash value, wherein the key is stored in an extent store metadata structure;

receiving a second write request directed towards a second LUN, the second write request having the write data and having metadata that includes a second LBA;

applying the hash function to the write data to generate a second hash value, wherein the first hash value is equal to the second hash value; and

in response to the second write request, and based on the first hash value being equal to the second hash value, returning an acknowledgment that the write data is stored without writing the write data again to the storage array.

2. The method of claim 1 wherein the first LUN is different from the second LUN.

3. The method of claim 2 wherein the first LUN is on a volume different from that of the second LUN.

4. The method of claim 1 further comprising:

forming the write data into an extent; and

compressing the extent prior to storage in the selected extent store.

5. The method of claim 1 further comprising:

reading the write data stored from the selected extent store; and

determining whether the write data in the selected extent store is identical to the write data in the second write request.

7. The method of claim 1 further comprising:

translating the first LBA to a first offset;

selecting a first volume metadata structure associated with the first LUN;

storing the key in the first volume metadata structure such that the key is associated with the first offset;

translating the second LBA to a second offset;

selecting a second volume metadata structure associated with the second LUN; and

storing the key associated with the write data in the second volume metadata structure such that the key is associated with the second offset, thereby de-duplicating storage of the write data.

8. The method of claim 7 wherein the volume metadata structure is stored on one or more of the plurality of extent stores.

9. The method of claim 8 wherein the volume metadata structure is not de-duplicated.

10. The method of claim 1 wherein the SSD includes flash components.

6. (canceled)

11. A method comprising:

receiving a first write request directed towards a first logical unit (LUN), the first write request having write data and having metadata that includes a first logical block address (LBA), the first write request processed at a node of a storage system, the storage system attached to a storage array of solid state drives (SSDs);

applying a hash function to the write data to generate a first hash value;

selecting an extent store from a plurality of extent stores based on the first hash value;

storing the write data in the selected extent store, wherein the selected extent store spans a set of SSDs of the storage array, wherein a key is formed from the first hash value;

translating the first LBA to a first offset;

selecting a first volume metadata structure associated with the first LUN;

storing the key in the first volume metadata structure such that the key is associated with the first offset;

receiving a second write request directed towards a second LUN, the second write request having the write data and having metadata that includes a second LBA;

applying the hash function to the write data to generate a second hash value, wherein the second hash value is identical to the first hash value;

translating the second LBA to a second offset;

selecting a second volume metadata structure associated with the second LUN; and

storing the key associated with the write data in the second volume metadata structure such that the key is associated with the second offset, thereby de-duplicating storage of the write data based on the first hash value being equal to the second hash value.

12. A system comprising:

a storage system having a memory connected to a processor via bus;

a storage array coupled to the storage system and having one or more solid state drives (SSDs);

a storage I/O stack executing on the processor of the storage system, the storage I/O stack when executed operable to;

receive a first write request directed towards a logical unit (LUN), the first write request having write data and having metadata that includes a logical block address (LBA);

apply a hash function to the write data to generate a first hash value;

select an extent store from a plurality of extent stores based on the first hash value;

store the write data in a segment of the selected extent store, wherein a key is formed from the first hash value, wherein the key is stored in an extent store metadata structure, wherein the segment spans a set of SSDs of the storage array;

receive a second write request directed towards a second LUN, the second write request having the write data and having metadata that includes a second LBA;

apply the hash function to the write data to generate a second hash value, wherein the first hash value is equal to the second hash value; and

in response to the second write request, and based on the first hash value being equal to the second hash value, return an acknowledgment that the write data is stored without writing the write data again to the storage array.

13. The system of claim 12 wherein the first LUN is different from the second LUN.

14. The system of claim 12 wherein the storage I/O stack when executed is further operable to:

form the write data into an extent; and

compress the extent prior to storage in the selected extent store.

15. The system of claim 12 wherein the storage I/O stack when executed is further operable to:

read the write data stored from the selected extent store; and

determine whether the write data in the selected extent store is identical to the write data in the second write request.

16. The system of claim 15 wherein the storage I/O stack when executed is further operable to:

store the write data in the second write request in the selected extent store, wherein the write data in the selected extent store is not identical to the write data in the second write request.

17. The system of claim 12, wherein the storage I/O stack when executed is further operable to:

translate the first LBA to a first offset;

select a first volume metadata structure associated with the first LUN;

store the key in the first volume metadata structure such that the key is associated with the first offset;

translate the second LBA to a second offset;

select a second volume metadata structure associated with the second LUN; and

store the key associated with the write data in the second volume metadata structure such that the key is associated with the second offset, thereby de-duplicating storage of the write data.

18. The system of claim 17 wherein the volume metadata structure is stored on one or more of the plurality of extent stores.

19. The system of claim 12 wherein the SSD includes flash components.

20. The system of claim 12 wherein the storage system comprises a cluster of nodes, and wherein the first write request is received at a first node and the second write request is received at a second node different from the first node.