Deduplicated host cache flush to remote storage

• US 9,830,273 B2
• Filed: 07/30/2015
• Issued: 11/28/2017
• Est. Priority Date: 07/30/2015
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Bk, Girish Kumar, Makkar, Gaurav
• Primary Examiner(s): Dudek, Jr., Edward
• Assistant Examiner(s): RUSSELL, ANDREW D
• Application Number:
  US14/814,053
• Publication Number:
  US 20170031830A1
• Time in Patent Office: 852 Days
• CPC Class Codes: G06F 12/0802 Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
• CPC Class Codes: G06F 12/0804 with main memory updating G06F12/0806 takes precedence
• CPC Class Codes: G06F 12/0868 Data transfer between cache memory and other subsystems, e.g. storage devices or host systems
• CPC Class Codes: G06F 12/0891 using clearing, invalidating or resetting means
• CPC Class Codes: G06F 2212/60 Details of cache memory
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0656 Data buffering arrangements
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

In addition to caching I/O operations at a host, at least some data management can migrate to the host. With host side caching, data sharing or deduplication can be implemented with the cached writes before those writes are supplied to front end storage elements. When a host cache flush to distributed storage trigger is detected, the host deduplicates the cached writes. The host aggregates data based on the deduplication into a “change set file” (i.e., a file that includes the aggregation of unique data from the cached writes). The host supplies the change set file to the distributed storage system. The host then sends commands to the distributed storage system. Each of the commands identifies a part of the change set file to be used for a target of the cached writes.

Claims

1. A method comprising:

caching writes in a persistent cache of a host device, wherein the writes indicate write targets and data units to write;

after detection of a cache flush trigger, determining a first set of the data units that are each unique among the data units;

communicating the first set of data units to a remote, distributed storage system;

generating a command for each of a subset of the cached writes, wherein the command indicates a corresponding one of the first set of data units as a donor and the write target of the cached write of the subset as a recipient, wherein generating the command for each of the subset of the cached writes comprises generating a copy command in accordance with a network file system protocol and indicating in the copy command that a share should be performed instead of a copy if a front end device of the distributed storage system is capable of performing the share; and

communicating the commands to the remote, distributed storage system.

2. The method of claim 1 further comprising aggregating the first set of data units.

3. The method of claim 2, wherein aggregating the first set of data units comprises generating a file with the first set of data units, wherein communicating first set of data units comprises communicating the file.

4. The method of claim 3, wherein generating the copy command for each of the subset of the cached writes comprises generating the copy command to indicate a part of the file as the donor for sharing or copying and the write target of the cached write as the recipient for the sharing or copying.

5. The method of claim 1, wherein communicating the first set of data units comprises communicating, to a front end device of the storage system, a write command that indicates the first set of data units.

6. The method of claim 1 further comprising detecting the cache flush trigger.

7. The method of claim 6, wherein detecting the cache flush trigger comprises detecting at least one of initiation of a data management operation, detecting expiration of a time period, and detection of an explicit cache flush command.

8. The method of claim 1, further comprising restricting each of the cached writes to read access in response to detection of the cache flush trigger until after flush of the cached writes to the remote, distributed storage system completes.

9. The method of claim 1, wherein performing a share operation comprises setting metadata of a write target to refer to the donor.

10. The method of claim 1 further comprising:

determining that a first write target and a second write target of the cached writes are contiguous; and

coalescing the generated commands corresponding to the first write target and the second write target into a single command.

11. The method of claim 1 further comprising:

determining that a first cached write has a first write target that is contiguous with a second write target of a second cached write;

wherein generating a command for each of a subset of the cached writes comprises generating a first command that indicates an offset of the first write target as a recipient offset and a length that encompasses the first write target and the second write target.

12. The method of claim 1 further comprising:

determining that a first cached write of the subset of cached writes increases size of a first write target of the first cached write;

generating a request to update metadata of the first write target to indicate the increase in size and to accommodate the increase in size; and

communicating the request to the remote, distributed storage system.

13. The method of claim 1 further comprising:

generating a request to deallocate a physical location referenced by a logical representation, wherein the logical representation is indicated in a first cached write of the subset of cached writes; and

communicating the request to the remote, distributed storage system,

wherein the command for the first cached write causes the remote, distributed storage system to update the logical representation to refer to a physical location of a first of the first set of data units instead of the deallocated physical location.

14. A non-transitory machine-readable medium comprising program code for efficient flushing of cached writes from a host device to a front end device of a storage system, the program code to:

cache writes in a persistent cache of the host device, wherein the writes indicate write targets and data units to write;

after detection of a cache flush trigger, determine whether any of the data units have a fingerprint that matches an entry in a fingerprint map or a fingerprint of any of the other data units;

update a change set file to include any of the data units that have a fingerprint that does not match an entry in the fingerprint map and that does not match a fingerprint of any of the other data units;

update the fingerprint map to indicate any fingerprint of the data units that was unique among the data units and unique with respect to the fingerprint map;

for each of the write targets, create a mapping between the write target and a part of the change set file that matches the data unit associated with the write target;

communicate any part of the change set file to the storage system that has not yet been communicated to the storage system;

for each of a subset of the cached writes, generate a command that indicates the write target of the cached write as a recipient and that indicates the part of the change set file mapped to the write target as a donor, wherein the program code to generate the command comprises program code to generate a copy command in accordance with a network file system protocol and to indicate in the copy command that a share should be performed instead of a copy if a front end device of the distributed storage system is capable of performing the share; and

communicate the commands to the storage system.

15. The non-transitory machine-readable medium of claim 14, further comprising program code to coalesce contiguous write targets.

16. The non-transitory machine-readable medium of claim 14, further comprising program code to generate a request for the storage system to update metadata of the write targets of the subset of cached writes based, at least in part, on an increase in size of the write target.

17. An apparatus comprising:

a processor;

a persistent cache; and

a machine-readable medium having program code executable by the processor to cause the apparatus to, cache writes in the persistent cache, wherein the writes indicate write targets and data units to write;

after detection of a cache flush trigger, determine a first set of the data units that are each unique among the data units;

communicate the first set of data units to a remote, distributed storage system;

generate a command for each of a subset of the cached writes, wherein the command indicates a corresponding one of the first set of data units as a donor and the write target of the cached write of the subset as a recipient, wherein generating the command for each of the subset of the cached writes comprises generating a copy command in accordance with a network file system protocol and indicating in the copy command that a share should be performed instead of a copy if a front end device of the distributed storage system is capable of performing the share;

communicate the commands to the remote, distributed storage system.

18. The apparatus of claim 17, wherein indication of performing a share comprises an indication of setting metadata of a write target to refer to the donor.

19. The apparatus of claim 17, wherein the machine-readable medium further has program code executable by the processor to cause the apparatus to:

determine those of the subset of cached writes that increase size of the corresponding ones of the writes targets to request the remote, distributed storage system to update metadata of those write targets based, at least in part, on the increased size.