Policy based sharing of redundant data across storage pools in a deduplicating system

• US 8,495,032 B2
• Filed: 10/01/2008
• Issued: 07/23/2013
• Est. Priority Date: 10/01/2008
• Status: Active Grant

• Current Assignee: Dropbox, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Anglin, Matthew J., Cannon, David M.
• Primary Examiner(s): Rostami, Mohammad S
• Application Number:
  US12/243,743
• Publication Number:
  US 20100082558A1
• Time in Patent Office: 1,756 Days
• Field of Search: None
• US Class Current: 707/692
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

One aspect of the present invention includes enabling data chunks to be shared among different storage pools within a storage management system, according the use of deduplication and storage information kept at the system level, and applied with policy-based rules that define the scope of deduplication. In one embodiment, the parameters of performing deduplication are defined within the policy, particularly which of the plurality of storage pools allow deduplication to which other pools. Accordingly, a data object may be linked to deduplicated data chunks existent within other storage pools, and the transfer of a data object may occur by simply creating references to existing data chunks in other pools provided the policy allows the pool to reference chunks in these other pools. Additionally, a group of storage pools may be defined within the policy to perform a common set of deduplication activities across all pools within the group.

Claims

1. A method in a computer system for implementing policy-based sharing of redundant data across deduplicating storage pools within a storage management system, comprising:

deduplicating a data object as data chunks across more than one of a plurality of storage pools based on a defined policy comprising a table with an entry for each storage pool, wherein an object index tracks a location of the data object, a deduplication index tracks data chunks, the plurality of storage pools are organized in a storage management system, each storage pool is designated as one of a deduplicating pool and a non-deduplicating pool, the table is stored on a storage management system level and is accessible by each of the plurality of storage pools, each entry of the table specifies allowed storage pools from which the storage pool references data chunks, tape storage pools are allowed to access disk storage pools and disk storage pools are not allowed to access tape storage pools, a first deduplicating storage pool stores a first link for a first data chunk of the data object linking to an instance of the first data chunk in an allowed deduplicating storage pool and does not store the first data chunk in response to the instance of the first data chunk existing in a third deduplicating storage pool that the allowed deduplicating storage pool is allowed to access, the first deduplicating storage pool stores the first data chunk of the data object in response to the instance of the first data chunk not existing in the allowed deduplicating storage pool, and a second non-deduplicating pool stores the data object without deduplication;

accessing the first data chunk from the allowed deduplicating storage pool using the first link if the first link is stored in the allowed deduplicating storage pool and accessing the first data chunk from the first storage pool if the first link is not stored in the allowed deduplicating storage pool; and

deleting the location of the data object from the object index and tracking the data chunks of the data object in the deduplication index in response turning off deduplication in the first storage pool.

2. The method as in claim 1, wherein the defined policy allows the table to specify referencing of data chunks between a backup storage pool and an archive storage pool and does not allow the table to specify referencing of data chunks between the backup storage pool and a hierarchical storage management (HSM) storage pool.

3. The method as in claim 1, wherein the defined policy defines a pool group comprising one or more storage pools within the plurality of storage pools.

4. The method as in claim 3, wherein the defined policy is configured to enable references from a defined storage pool to data chunks stored within pools which share group membership with the defined storage pool, and to disable references from the defined storage pool to data chunks stored within pools which do not share group membership with the defined storage pool.

5. The method as in claim 1, further comprising transferring the data object from the first deduplicating storage pool to a target storage pool within the plurality of storage pools wherein, for each data chunk of the data object, the data chunk is transferred only if the data chunk is not stored within either of the target storage pool or another storage pool within the plurality of storage pools which the target storage pool can reference according to the defined policy.

6. The method as in claim 1, wherein the defined policy defines rules for client node groups, wherein each client node group comprises a plurality of storage pools and storage pools of the plurality of storage pools of client nodes within a defined node group and can share data chunks regardless of table entries for the storage pools.

7. The method as in claim 1, wherein the defined policy is derived automatically by the storage management system, and wherein the policy is explicitly overridden by defined policy settings.

8. A system, comprising:

at least one processor; and

at least one memory which stores instructions operable with the at least one processor for implementing policy-based sharing of redundant data across deduplicating storage pools within a storage management system, the instructions being executed for;

deduplicating a data object as data chunks across more than one of a plurality of storage pools based on a defined policy comprising a table with an entry for each storage pool, wherein an object index tracks a location of the data object, a deduplication index tracks data chunks, the plurality of storage pools are organized in a storage management system, each storage pool is designated as one of a deduplicating pool and a non-deduplicating pool, the table is stored on a storage management system level and is accessible by each of the plurality of storage pools, each entry of the table specifies allowed storage pools from which the storage pool references data chunks, tape storage pools are allowed to access disk storage pools and disk storage pools are not allowed to access tape storage pools, a first deduplicating storage pool stores a first link for a first data chunk of the data object linking to an instance of the first data chunk in an allowed deduplicating storage pool and does not store the first data chunk in response to the instance of the first data chunk existing in a third deduplicating storage pool that the allowed deduplicating storage pool is allowed to access, the first deduplicating storage pool stores the first data chunk of the data object in response to the instance of the first data chunk not existing in the allowed deduplicating storage pool, and a second non-deduplicating pool stores the data object without deduplication;

accessing the first data chunk from the allowed deduplicating storage pool using the first link if the first link is stored in the allowed deduplicating storage pool and accessing the first data chunk from the first storage pool if the first link is not stored in the allowed deduplicating storage pool; and

deleting the location of the data object from the object index and tracking the data chunks of the data object in the deduplication index in response turning off deduplication in the first storage pool.

9. The system of claim 8, wherein the defined policy allows the table to specify referencing of data chunks between a backup storage pool and an archive storage pool and does not allow the table to specify referencing of data chunks between the backup storage pool and a hierarchical storage management (HSM) storage pool.

10. The system of claim 8, wherein the defined policy defines a pool group comprising one or more storage pools within the plurality of storage pools.

11. The system of claim 10, wherein the defined policy is configured to enable references from a defined storage pool to data chunks stored within pools which share group membership with the defined storage pool, and to disable references from the defined storage pool to data chunks stored within pools which do not share group membership with the defined storage pool.

12. The system of claim 8, further comprising transferring the data object from the first deduplicating storage pool to a target storage pool within the plurality of storage pools wherein, for each data chunk of the data object, the data chunk is transferred only if the data chunk is not stored within either of the target storage pool or another storage pool within the plurality of storage pools which the target storage pool can reference according to the defined policy.

13. The system of claim 8, wherein the defined policy defines rules for client node groups, wherein each client node group comprises a plurality of storage pools and storage pools of the plurality of storage pools of client nodes within a defined node group and can share data chunks regardless of table entries for the storage pools.

14. The system of claim 8, wherein the defined policy is derived automatically by the storage management system, and wherein the policy is explicitly overridden by defined policy settings.

15. A computer program product comprising a non-transitory computer readable medium storing a computer readable program for implementing policy-based sharing of redundant data across deduplicating storage pools within a storage management system, wherein the computer readable program when executed on a computer causes the computer to:

deduplicate a data object as data chunks across more than one of a plurality of storage pools based on a defined policy comprising a table with an entry for each storage pool, wherein an object index tracks a location of the data object, a deduplication index tracks data chunks, the plurality of storage pools are organized in a storage management system, each storage pool is designated as one of a deduplicating pool and a non-deduplicating pool, the table is stored on a storage management system level and is accessible by each of the plurality of storage pools, each entry of the table specifies allowed storage pools from which the storage pool references data chunks, tape storage pools are allowed to access disk storage pools and disk storage pools are not allowed to access tape storage pools, a first deduplicating storage pool stores a first link for a first data chunk of the data object linking to an instance of the first data chunk in an allowed deduplicating storage pool and does not store the first data chunk in response to the instance of the first data chunk existing in a third deduplicating storage pool that the allowed deduplicating storage pool is allowed to access, the first deduplicating storage pool stores the first data chunk of the data object in response to the instance of the first data chunk not existing in the allowed deduplicating storage pool, and a second non-deduplicating pool stores the data object without deduplication;

accessing the first data chunk from the allowed deduplicating storage pool using the first link if the first link is stored in the allowed deduplicating storage pool and accessing the first data chunk from the first storage pool if the first link is not stored in the allowed deduplicating storage pool; and

deleting the location of the data object from the object index and tracking the data chunks of the data object in the deduplication index in response turning off deduplication in the first storage pool.

16. The computer program product of claim 15, wherein the defined policy allows the table to specify referencing of data chunks between a backup storage pool and an archive storage pool and does not allow the table to specify referencing of data chunks between the backup storage pool and a hierarchical storage management (HSM) storage pool.

17. The computer program product of claim 15, wherein the defined policy defines a pool group comprising one or more storage pools within the plurality of storage pools.

18. The computer program product of claim 17, wherein the defined policy is configured to enable references from a defined storage pool to data chunks stored within pools which share group membership with the defined storage pool, and to disable references from the defined storage pool to data chunks stored within pools which do not share group membership with the defined storage pool.

19. The computer program product of claim 15, further comprising transferring the data object from the first deduplicating storage pool to a target storage pool within the plurality of storage pools wherein, for each data chunk of the data object, the data chunk is transferred only if the data chunk is not stored within either of the target storage pool or another storage pool within the plurality of storage pools which the target storage pool can reference according to the defined policy.

20. The computer program product of claim 15, wherein the defined policy defines rules for client node groups, wherein each client node group comprises a plurality of storage pools and storage pools of the plurality of storage pools of client nodes within a defined node group and can share data chunks regardless of table entries for the storage pools.

21. The computer program product of claim 15, wherein the defined policy is derived automatically by the storage management system, and wherein the policy is explicitly overridden by defined policy settings.