System and method for granular deduplication

• US 10,346,045 B2
• Filed: 06/07/2017
• Issued: 07/09/2019
• Est. Priority Date: 02/11/2015
• Status: Active Grant

• Current Assignee: Innovations In Memory LLC
• Original Assignee: Violin Systems LLC
• Inventors: Garg, Amit, Stoakes, Timothy, Ratna, Vikas
• Primary Examiner(s): Verbrugge, Kevin
• Application Number:
  US15/616,373
• Publication Number:
  US 20170269850A1
• Time in Patent Office: 762 Days
• Field of Search: None
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

A method of managing a memory system is described, the memory system having a processor with a client interface, a buffer memory, and a storage memory. A LUN (logical unit) is exposed to the client, the LUN having a memory size, and QoS specification. When the processor receives a command from the client to write data to the LUN, determining if the QoS specification includes deduplication and: if the QoS does not include deduplication, processing the command by storing the data in the storage memory and creating metadata uniquely referencing a storage location of the data that is written; or if the QoS includes deduplication, determine if deduplication is possible while meeting a latency value as specified in the QoS specification and performing deduplication.

Claims

1. A method of managing a memory system, comprising:

providing a memory system with a processor having a client interface, a buffer memory, and a storage memory;

allocating a logical unit (LUN) of a plurality of LUNs to the client, the allocated LUN having a memory size, and a QoS specification;

receiving a command to write data to the allocated LUN;

determining if the QoS specification for the allocated LUN includes deduplication and;

if the QoS specification does not include deduplication, process the command by storing the data in the storage memory and creating metadata uniquely referencing a storage location of the data that is written to the storage memory;

or

if the QoS specification includes deduplication, determine if deduplication is possible while meeting the QoS specification and;

if deduplication is possible, performing deduplication of the data and creating metadata referencing the storage location where the deduplicated data is stored;

or,

if deduplication is not possible, storing the data in a backing LUN associated with the allocated LUN, and

draining the data stored in the backing LUN when deduplication is possible by performing deduplication.

2. The method of claim 1, wherein the step of deduplication includes a step of data compression.

3. The method of claim 1 wherein the QoS specification includes express and implied requirements and express requirements are determined at a memory system-client interface.

4. The method of claim 3, wherein the step of determination if deduplication is possible comprises:

determining that the express requirements of the QoS specification for data to be stored by the allocated LUN can be met by the memory system while performing deduplication.

5. The method of claim 4, wherein the express QoS requirement comprises a throughput.

6. The method of claim 4, wherein the express QoS requirement comprises a latency.

7. The method of claim 6, wherein at least one express QoS requirement has a predetermined time variation.

8. The method of claim 3, wherein the implied requirement includes at least one of:

deduplication latency, or compression latency.

9. The method of claim 1, wherein the allocated LUN where the QoS specification includes deduplication further comprises:

a stub LUN (SL) storing metadata for the deduplicated stored data and the metadata comprises a logical block address (LBA) mapped to a location of the deduplicated stored data in a single instance repository (SIR).

10. The method of claim 9, wherein the metadata of the stub LUN (SL) is encrypted.

11. The method of claim 9, wherein the single instance repository (SIR) is a logical memory extent of the storage memory where data is stored.

12. The method of claim 9, wherein the SIR includes deduplicated data that is either unique data or duplicate data referenced by one or more LUNs of a plurality of allocated LUNS.

13. The method of claim 9, wherein the data stored in the SIR is encrypted.

14. The method of claim 1, wherein the data is erased from the allocated LUN by modifying the metadata of a stub LUN (SL) of the allocated LUN.

15. The method of claim 1, further comprising providing a clone LUN by copying the metadata of a stub LUN (SL) of the allocated LUN to the clone LUN or by copying the metadata of a sub-LUN of the SL to a sub-LUN clone in accordance with a requirement of the QoS specification.

16. The method of claim 1, further comprising providing a snapshot resource area (SRA) in the memory system such that when a requirement of the QoS specification for the allocated LUN includes a snapshot, the method further includes copying metadata for stored data from a stub LUN (SL) of the LUN to the SRA when new data is being stored to the allocated LUN after the snapshot.

17. The method of claim 1, wherein the step of deduplication includes:

computing a hash value of the data;

comparing the hash value with hash values in a hash value repository; and

;

if the hash value exists, adding a reference to the storage location of the data in a single instance repository (SIR) to a stub LUN of the allocated LUN;

or

if the hash value does not exist, adding the hash value to the hash value repository, storing the data in the SIR, and adding a reference to the storage location of the data in the SIR to the stub LUN of the allocated LUN.

18. The method of claim 17, wherein the hash value repository is stored in the SIR.

19. The method of claim 17, wherein the hash value is computed using a cryptographic hash function.

20. An apparatus for managing a memory system, comprising:

storage memory; and

a processor configured to;

allocate a logical unit (LUN) to a client;

receive a command to write data to the allocated LUN;

determine if a quality of service (QoS) specification for the allocated LUN includes deduplication;

if the QoS specification does not include deduplication, process the command by storing the data in the storage memory and creating metadata referencing a storage location of the data in the storage memory; and

if the QoS specification includes deduplication, deduplicating the data and creating metadata referencing the storage location where the deduplicated data is stored.

21. The apparatus of claim 20, wherein the processor is further configured to:

compute a hash value of the data;

compare the hash value with hash values in a hash value repository;

if the hash value exists in the hash value repository, add a storage location of the data in a single instance repository (SIR) to a stub LUN for the allocated LUN; and

if the hash value does not exist, add the hash value to the hash value repository, store the data in the SIR, and add a storage location of the stored data to the stub LUN for the allocated LUN.

22. The apparatus of claim 20, wherein the processor is further configured to:

determine if deduplication is possible while meeting the QoS specification;

if deduplication is possible, performing deduplication of the data and creating the metadata referencing the location where the deduplicated data is stored; and

if deduplication is not possible, storing the data in a backing LUN associated with the allocated LUN.