Global in-line extent-based deduplication

• US 9,529,546 B2
• Filed: 01/08/2014
• Issued: 12/27/2016
• Est. Priority Date: 01/08/2014
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Sundaram, Rajesh, Kimmel, Jeffrey S., Lewis, Blake H.
• Primary Examiner(s): Yi, David X
• Assistant Examiner(s): RANKIN, CANDICE A
• Application Number:
  US14/150,689
• Publication Number:
  US 20150193170A1
• Time in Patent Office: 1,084 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0688 Non-volatile semiconductor memory arrays
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

In one embodiment, a layered file system includes a volume layer and an extent store layer configured to provide sequential log-structured layout of data and metadata on solid state drives (SSDs) of one or more storage arrays. The data is organized as variable-length extents of one or more logical units (LUNs). The metadata includes volume metadata mappings from offset ranges of a LUN to extent keys and extent metadata mappings of the extent keys to storage locations of the extents on the SSDs. The extent store layer maintaining the extent metadata mappings determines whether an extent is stored on a storage array, and, in response to determination that the extent is stored on the storage array, returns an extent key for the stored extent to the volume layer to enable global inline de-duplication that obviates writing a duplicate copy of the extent on the storage array.

Claims

1. A system comprising:

a central processing unit (CPU) of a node of a cluster having a plurality of nodes, each node coupled to one or more storage arrays of solid state drives (SSDs); and

a memory coupled to the CPU and configured to store a layered file system of a storage input/output (I/O) stack, the layered file system including a volume layer and an extent store layer configured to provide sequential log-structured layout of data and metadata on the SSDs, the data and metadata organized as variable-length extents of one or more logical units (LUNs) served by the nodes, the metadata including volume metadata mappings from offset ranges of a LUN to extent keys and extent metadata mappings of the extent keys to storage locations of the extents on the SSDs, wherein the extent store layer maintaining the extent metadata mappings is configured to

determine whether a data extent is stored on a storage array of the cluster and,

in response to determination that the data extent is stored on the storage array of the cluster, retrieve an extent key for the stored data extent from an extent metadata mapping having a storage location on a SSD for the stored data extent, and return the extent key for the stored data extent to the volume layer to enable selective inline de-duplication that obviates writing a duplicate copy of the data extent on the storage array, wherein a flag passed to the extent store layer enables the selective inline de-duplication such that a metadata extent is not de-duplicated.

2. The system of claim 1 wherein the volume layer cooperates with the extent store layer to provide a level of indirection that facilitates the log-structured layout of the extents on the SSDs by the extent store layer.

3. The system of claim 2 wherein the level of indirection allows relocation of the data extent without update to the volume metadata mappings such that the storage location of the extent on the SSD is virtualized by the extent key.

4. The system of claim 3 wherein the extent store layer is implemented as one or more processes, and wherein the virtualized storage location permits a cleaning process to occur in the extent store layer without update to the volume metadata mappings.

5. The system of claim 1 wherein the volume metadata mappings and the extent metadata mappings are organized as separate file system metadata structures in the memory.

6. The system of claim 5 wherein separation of the volume metadata mappings of the volume layer and the extent metadata mappings of the extent store layer enables different volumes with different offset ranges to map to a same extent key and a same extent.

7. A method comprising:

providing, by a node of a cluster, a layered file system, the layered file system including a volume layer and an extent store layer configured to provide sequential log-structured layout of data and metadata on solid state drives (SSDs) of one or more storage arrays of the cluster, the data and metadata organized as variable-length extents of one or more logical units (LUNs), the metadata including volume metadata mappings from offset ranges of a LUN to extent keys and extent metadata mappings of the extent keys to storage locations of the extents on the SSDs;

determining whether a data extent is stored on a storage array of the cluster; and

in response to determining that the data extent is stored on the storage array of the cluster, retrieving an extent key for the stored data extent from an extent metadata mapping having a storage location on a SSD for the stored data extent, and returning the extent key for the stored data extent to the volume layer to enable inline de-duplication that obviates writing a duplicate copy of the data extent on the storage array, wherein a flag passed to the extent store layer enables the selective inline de-duplication such that a metadata extent is not de-duplicated.

8. The method of claim 7 further comprising:

cooperating, between the volume layer and the extent store layer, to provide a level of indirection that facilitates the log-structured layout of the extents on the SSDs by the extent store layer.

9. The method of claim 8 further comprising:

relocating the data extent without update to the volume metadata mappings.

10. The method of claim 9 wherein the storage location of the data extent on the SSD is virtualized by the extent key.

11. The method of claim 10 wherein the extent store layer is implemented as one or more processes, and the method further comprises:

performing a cleaning process in the extent store layer without update to the volume metadata mappings.

12. The method of claim 7 wherein the volume metadata mappings and the extent metadata mappings are organized as separate file system metadata structures.

13. The method of claim 12 further comprising:

mapping different volumes with different offset ranges to a same extent key and a same extent.

14. A non-transitory computer readable medium including program instructions for execution on one or more processors, the program instructions when executed operable to:

provide a layered file system, the layered file system including a volume layer and an extent store layer configured to provide sequential log-structured layout of data and metadata on solid state drives (SSDs) of one or more storage arrays of a cluster, the data and metadata organized as variable-length extents of one or more logical units (LUNs), the metadata including volume metadata mappings from offset ranges of a LUN to extent keys and extent metadata mappings of the extent keys to storage locations of the extents on the SSDs;

determine whether a data extent is stored on a storage array of the cluster; and

in response to determination that the data extent is stored on the storage array of the cluster, retrieve an extent key for the stored data extent from an extent metadata mapping having a storage location on a SSD for the stored data extent, and return the extent key for the stored data extent to the volume layer to enable inline de-duplication that obviates writing a duplicate copy of the extent on the storage array, wherein a flag passed to the extent store layer enables the selective inline de-duplication such that a metadata extent is not de-duplicated.

15. The non-transitory computer readable medium of claim 14 wherein the volume layer and the extent store layer cooperate to provide a level of indirection that facilitates the log-structured layout of the extents on the SSD by the extent store layer.

16. The non-transitory computer readable medium of claim 14 wherein the program instructions when executed are further operable to:

relocate the data extent without update to the volume metadata mappings.

17. The non-transitory computer readable medium of claim 16 wherein the storage location of the data extent on the SSD is virtualized by the extent key.

18. The non-transitory computer readable medium of claim 17 wherein the extent store layer is implemented as one or more processes, and a cleaning process in the extent store layer is performed without update to the volume metadata mappings.

19. The non-transitory computer readable medium of claim 14 wherein the volume metadata mappings and the extent metadata mappings are organized as separate file system metadata structures.

20. The non-transitory computer readable medium of claim 19 wherein separation of the volume metadata mappings of the volume layer and the extent metadata mappings of the extent store layer enables different volumes with different offset ranges to map to a same extent key and a same extent.