Deduplicating container files

• US 9,569,455 B1
• Filed: 06/28/2013
• Issued: 02/14/2017
• Est. Priority Date: 06/28/2013
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Bono, Jean-Pierre, Davenport, William C., de Forest, Miles A., Armangau, Philippe, Forrester, Walter C., Owen, Karl M., Tummala, Himabindu
• Primary Examiner(s): Trujillo, James
• Assistant Examiner(s): Mina, Fatima
• Application Number:
  US13/930,010
• Time in Patent Office: 1,327 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

A technique forms contents of a deduplication container from a standalone container file system stored on the same data storage array on which the deduplication container is stored. Such a standalone container includes a container file which is formed by standalone blocks that are not shared with any other file within the standalone container outside of snapshots of that file. The standalone blocks store host data and which are hierarchically related to each other. Upon receiving a deduplication command, the storage processor carries out a set of deduplication operations which include creating and storing the deduplication container in the data storage array. The deduplication container includes container files which are formed by deduplication blocks storing the host data and which are hierarchically related to each other.

Claims

1. A method of performing deduplication on a data storage array that stores host data, the method comprising:

initially storing a standalone container on the data storage array, the standalone container including container files which are formed by standalone blocks storing the host data and which are hierarchically related to each other;

receiving a deduplication command which identifies the standalone container; and

in response to the deduplication command, performing a set of deduplication operations to create and store a deduplication container on the data storage array, the deduplication container including container files which are formed by deduplication blocks storing the host data and which are hierarchically related to each other,

wherein each of the standalone container and the deduplication container is a lower deck file system in the data storage array, each container file of the standalone container and the deduplication container representing an upper deck file system for a host file system accessible to a respective host computer of a set of host computers;

wherein the deduplication command which identifies the standalone container is received from a host computer of the set of host computers.

2. A method as in claim 1,

wherein performing the set of deduplication operations includes:

assigning a virtual block map (VBM) pointer to each standalone block that is not marked as a duplicate to form a deduplication block,

for each standalone block marked as a duplicate, i) assigning a VBM pointer to another VBM pointer that points to a standalone block that is not marked as a duplicate and contains the same host data as that standalone block, and ii) freeing that standalone block.

3. A method as in claim 2,

wherein performing the set of deduplication operations further includes:

establishing a replication session between the standalone container and the deduplication container, and

after establishing the replication session, replicating the container file of the standalone container in the deduplication container; and

wherein freeing each standalone block includes;

removing the standalone container from the data storage array.

4. A method as in claim 3,

wherein the standalone container includes a particular container file and a set of snapshots of the particular container file, each of the set of snapshots being identified with an elapsed time since creation;

wherein establishing the replication session includes;

locating an oldest snapshot of the set of snapshots, the oldest snapshot being identified with the longest elapsed time since creation, and

identifying subsequent snapshots in descending order of elapsed time since creation.

5. A method as in claim 4,

wherein replicating the full contents of the standalone container in the deduplication container includes:

replicating the oldest snapshot in the deduplication container;

for each subsequent snapshot, i) forming a difference between that subsequent snapshot and the previous snapshot, ii) replicating the difference in the deduplication container, and iii) adding the replicated difference to the replicated previous snapshot to form a replicated subsequent snapshot; and

wherein removing the standalone container from the data storage array includes;

deleting the set of snapshots from the standalone container.

6. A method as in claim 4,

wherein the standalone container includes an external replication session having a replication configuration between the standalone container and a replicated standalone container on another data storage array;

wherein the method further comprises;

prior to removing the standalone container from the data storage array, i) pausing the external replication session and ii) copying the replication configuration of the external replication session to the deduplication container;

deleting the replication configuration from the standalone container; and

establishing another external replication session between the deduplication container and the replicated standalone container.

7. A method as in claim 4,

wherein the standalone container includes an external replication session having a replication configuration between the standalone container and a replicated standalone container on another data storage array

wherein the standalone container acts as a source file system and the other data storage array as a target file system, the standalone container having a replication configuration between the standalone container and the replicated standalone container on another data storage array;

wherein establishing the replication session includes;

setting up a symmetric replication session involving the external replication session and the deduplication container;

wherein the method further comprises;

prior to removing the standalone container from the data storage array, establishing a failover condition in the standalone container, the failover condition causing the deduplication container to act as the source file system; and

deleting the replication configuration from the standalone container.

8. A method as in claim 1,

wherein the data storage array includes an upper deck file system accessible to a host and a lower deck file system including the standalone container and the deduplication container, the upper deck file system being stored in the form of a container file of the standalone container;

wherein receiving the deduplication command which identifies the standalone container includes;

obtaining a message from the host indicating that the container file of the standalone container is to be converted into a container file in the deduplication container.

9. A method as in claim 8,

wherein the lower deck file system is provisioned blocks of data from a storage pool, the storage pool containing all blocks of storage of the data storage array, a single deduplication container being stored per storage pool;

and

wherein performing the set of deduplication operations to create and store the deduplication container on the data storage array includes;

prior to creating the deduplication container, verifying whether there is another deduplication container stored in the lower deck file system.

21. A method as in claim 1, wherein each of the standalone container and the deduplication container includes an inode table containing inode numbers;

wherein each container file of the standalone container includes an inode having an inode number contained in the inode table of the standalone container;

wherein each container file of the deduplication container includes an inode having an inode number contained in the inode table of the deduplication container; and

wherein the method further comprises, in response to performing the set of deduplication operations, updating the respective inode tables of the standalone container and the deduplication container.

22. A method as in claim 1, wherein each block of each container file of the standalone container is fully owned by that container file and is accessible to only to snapshots of that container file;

and

wherein a block of a container file of the deduplication container is a shared block accessible to another container file of the deduplication container, the other container file not being a snapshot of the container file.

10. A data storage apparatus constructed and arranged to store host data, the data storage system comprising:

a set of storage devices; and

a storage processor, the storage processor including;

memory; and

a set of processors coupled to the memory to form controlling circuitry, the controlling circuitry constructed and arranged to;

initially store a standalone container on the data storage array, the standalone container including container files which are formed by standalone blocks storing the host data and which are hierarchically related to each other;

receive a deduplication command which identifies the standalone container; and

in response to the deduplication command, perform a set of deduplication operations to create and store a deduplication container on the data storage array, the deduplication container including container files which are formed by deduplication blocks storing the host data and which are hierarchically related to each other,

wherein each of the standalone container and the deduplication container is a lower deck file system in the data storage array, each container file of the standalone container and the deduplication container representing an upper deck file system for a host file system accessible to a respective host computer of a set of host computers;

wherein the deduplication command which identifies the standalone container is received from a host computer of the set of host computers.

11. A data storage apparatus as in claim 10,

wherein performing the set of deduplication operations includes:

assigning a virtual block map (VBM) pointer to each standalone block that is not marked as a duplicate to form a deduplication block,

for each standalone block marked as a duplicate, i) assigning a VBM pointer to another VBM pointer that points to a standalone block that is not marked as a duplicate and contains the same host data as that standalone block, and ii) freeing that standalone block.

12. A data storage apparatus as in claim 11,

wherein performing the set of deduplication operations further includes:

establishing a replication session between the standalone container and the deduplication container, and

after establishing the replication session, replicating the container file of the standalone container in the deduplication container; and

wherein freeing each standalone block includes;

removing the standalone container from the data storage array.

13. A data storage apparatus as in claim 12,

wherein the standalone container includes a particular container file and a set of snapshots of the particular container file, each of the set of snapshots being identified with an elapsed time since creation;

wherein establishing the replication session includes;

locating an oldest snapshot of the set of snapshots, the oldest snapshot being identified with the longest elapsed time since creation, and

identifying subsequent snapshots in descending order of elapsed time since creation.

14. A data storage apparatus as in claim 13,

wherein replicating the full contents of the standalone container in the deduplication container includes:

replicating the oldest snapshot in the deduplication container;

for each subsequent snapshot, i) forming a difference between that subsequent snapshot and the previous snapshot, ii) replicating the difference in the deduplication container, and iii) adding the replicated difference to the replicated previous snapshot to form a replicated subsequent snapshot; and

wherein removing the standalone container from the data storage array includes;

deleting the set of snapshots from the standalone container.

15. A data storage apparatus as in claim 13,

wherein the standalone container includes an external replication session having a replication configuration between the standalone container and a replicated standalone container on another data storage array;

wherein the controlling circuitry is further constructed and arranged to;

prior to removing the standalone container from the data storage array, i) pause the external replication session and ii) copy the replication configuration of the external replication session to the deduplication container;

delete the replication configuration from the standalone container; and

establish another external replication session between the deduplication container and the replicated standalone container.

16. A data storage apparatus as in claim 13,

wherein the standalone container includes an external replication session having a replication configuration between the standalone container and a replicated standalone container on another data storage array

wherein the standalone container acts as a source file system and the other data storage array as a target file system, the standalone container having a replication configuration between the standalone container and the replicated standalone container on another data storage array;

wherein establishing the replication session includes;

setting up a symmetric replication session involving the external replication session and the deduplication container;

wherein the method further comprises;

prior to removing the standalone container from the data storage array, establishing a failover condition in the standalone container, the failover condition causing the deduplication container to act as the source file system; and

deleting the replication configuration from the standalone container.

17. A data storage apparatus as in claim 10,

wherein the data storage array includes an upper deck file system accessible to a host and a lower deck file system including the standalone container and the deduplication container, the upper deck file system being stored in the form of a container file of the container file system;

wherein receiving the deduplication command which identifies the standalone container includes;

obtaining a message from the host indicating that the container file of the standalone container is to be converted into a container file of the deduplication container.

18. A computer program product having a non-transitory, computer-readable storage medium which stores instructions which, when executed by a computer, cause the computer to perform a method of performing deduplication on a data storage array that stores host data, the method comprising:

initially storing a standalone container on the data storage array, the standalone container including container files which are formed by standalone blocks storing the host data and which are hierarchically related to each other;

receiving a deduplication command which identifies the standalone container; and

in response to the deduplication command, performing a set of deduplication operations to create and store a deduplication container on the data storage array, the deduplication container including container files which are formed by deduplication blocks storing the host data and which are hierarchically related to each other,

wherein each of the standalone container and the deduplication container is a lower deck file system in the data storage array, each container file of the standalone container and the deduplication container representing an upper deck file system for a host file system accessible to a respective host computer of a of host computers;

wherein the deduplication command which identifies the standalone container is received from a host computer of the set of host computers.

19. A computer program product as in claim 18,

wherein performing the set of deduplication operations includes:

assigning a virtual block map (VBM) pointer to each standalone block that is not marked as a duplicate to form a deduplication block,

for each standalone block marked as a duplicate, i) assigning a VBM pointer to another VBM pointer that points to a standalone block that is not marked as a duplicate and contains the same host data as that standalone block, and ii) freeing that standalone block.

20. A computer program product as in claim 19,

wherein performing the set of deduplication operations further includes:

establishing a replication session between the standalone container and the deduplication container, and

after establishing the replication session, replicating the container file of the standalone container in the deduplication container; and

wherein freeing each standalone block includes;

removing the standalone container from the data storage array.