GLOBAL IN-LINE EXTENT-BASED DEDUPLICATION

• US 20150193170A1
• Filed: 01/08/2014
• Published: 07/09/2015
• Est. Priority Date: 01/08/2014
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Sundaram, Rajesh, Kimmel, Jeffrey S., Lewis, Blake H.
• Granted Patent: US 9,529,546 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0688 Non-volatile semiconductor memory arrays
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

In one embodiment, a layered file system includes a volume layer and an extent store layer configured to provide sequential log-structured layout of data and metadata on solid state drives (SSDs) of one or more storage arrays. The data is organized as variable-length extents of one or more logical units (LUNs). The metadata includes volume metadata mappings from offset ranges of a LUN to extent keys and extent metadata mappings of the extent keys to storage locations of the extents on the SSDs. The extent store layer maintaining the extent metadata mappings determines whether an extent is stored on a storage array, and, in response to determination that the extent is stored on the storage array, returns an extent key for the stored extent to the volume layer to enable global inline de-duplication that obviates writing a duplicate copy of the extent on the storage array.

Claims

1-20. -20. (canceled)

21. A system comprising:

a central processing unit (CPU) of a node of a cluster having a plurality of nodes, each node coupled to one or more storage arrays of solid state drives (SSDs); and

a memory coupled to the CPU and configured to store a layered file system of a storage input/output (I/O) stack, the layered file system including a volume layer and an extent store layer configured to provide sequential log-structured layout of data and metadata on the SSDs, the data organized as variable-length extents of one or more logical units (LUNs) served by the nodes, the metadata including volume metadata mappings from offset ranges of a LUN to extent keys and extent metadata mappings of the extent keys to storage locations of the extents on the SSDs, wherein the extent store layer maintaining the extent metadata mappings is configured to

determine whether an extent is stored on a storage array of the cluster and,

in response to determination that the extent is stored on the storage array of the cluster, return an extent key for the stored extent to the volume layer to enable global inline de-duplication that obviates writing a duplicate copy of the extent on the storage array.

22. The system of claim 21 wherein the volume layer cooperates with the extent store layer to provide a level of indirection that facilitates the log-structured layout of the extent on a SSD by the extent store layer.

23. The system of claim 22 wherein the level of indirection allows relocation of the extent without update to the volume metadata mappings such that the storage location of the extent on the SSD is virtualized by a mapped extent key.

24. The system of claim 23 wherein the extent store layer is implemented as one or more processes, and wherein the virtualized storage location permits a cleaning process to occur in the extent store layer without update to the volume metadata mappings.

25. The system of claim 21 wherein the volume metadata mappings and the extent metadata mappings are organized as separate file system metadata structures in the memory.

26. The system of claim 25 wherein separation of the volume metadata mappings of the volume layer and the extent metadata mappings of the extent store layer enables different volumes with different offset ranges to map to a same extent key and a same extent.

27. A method comprising:

providing, by a node of a cluster, a layered file system, the layered file system including a volume layer and an extent store layer configured to provide sequential log-structured layout of data and metadata on solid state drives (SSDs) of one or more storage arrays of the cluster, the data organized as variable-length extents of one or more logical units (LUNs), the metadata including volume metadata mappings from offset ranges of a LUN to extent keys and extent metadata mappings of the extent keys to storage locations of the extents on the SSDs;

determining, whether an extent is stored on a storage array of the cluster; and

in response to determining that the extent is stored on the storage array of the cluster, returning an extent key for the stored extent to the volume layer to enable global inline de-duplication that obviates writing a duplicate copy of the extent on the storage array.

28. The method of claim 27 further comprising:

cooperating, between the volume layer and the extent store layer, to provide a level of indirection that facilitates the log-structured layout of the extent on a SSD by the extent store layer.

29. The method of claim 28 further comprising:

relocating the extent without update to the volume metadata mappings.

30. The method of claim 29 wherein the storage location of the extent on the SSD is virtualized by a mapped extent key.

31. The method of claim 30 wherein the extent store layer is implemented as one or more processes, and the method further comprises:

performing a cleaning process in the extent store layer without update to the volume metadata mappings.

32. The method of claim 27 wherein the volume metadata mappings and the extent metadata mappings are organized as separate file system metadata structures.

33. The method of claim 32 further comprising:

mapping different volumes with different offset ranges to a same extent key and a same extent.

34. A non-transitory computer readable medium including program instructions for execution on one or more processors, the program instructions when executed operable to:

provide a layered file system, the layered file system including a volume layer and an extent store layer configured to provide sequential log-structured layout of data and metadata on solid state drives (SSDs) of one or more storage arrays of the cluster, the data organized as variable-length extents of one or more logical units (LUNs), the metadata including volume metadata mappings from offset ranges of a LUN to extent keys and extent metadata mappings of the extent keys to storage locations of the extents on the SSDs;

determine, whether an extent is stored on a storage array of the cluster; and

in response to determination that the extent is stored on the storage array of the cluster, return an extent key for the stored extent to the volume layer to enable global inline de-duplication that obviates writing a duplicate copy of the extent on the storage array.

35. The non-transitory computer readable medium of claim 34 wherein the volume layer and the extent store layer cooperate to provide a level of indirection that facilitates the log-structured layout of the extent on a SSD by the extent store layer.

36. The non-transitory computer readable medium of claim 34 wherein the program instructions when executed are further operable to:

relocate the extent without update to the volume metadata mappings.

37. The non-transitory computer readable medium of claim 36 wherein the storage location of the extent on the SSD is virtualized by a mapped extent key.

38. The non-transitory computer readable medium of claim 37 wherein the extent store layer is implemented as one or more processes, and a cleaning process in the extent store layer is performed without update to the volume metadata mappings.

39. The non-transitory computer readable medium of claim 34 wherein the volume metadata mappings and the extent metadata mappings are organized as separate file system metadata structures.

40. The non-transitory computer readable medium of claim 39 wherein separation of the volume metadata mappings of the volume layer and the extent metadata mappings of the extent store layer enables different volumes with different offset ranges to map to a same extent key and a same extent.