Enhanced reliability in deduplication technology over storage clouds

• US 9,229,819 B2
• Filed: 10/27/2014
• Issued: 01/05/2016
• Est. Priority Date: 04/25/2012
• Status: Expired due to Fees

• Current Assignee: AirBNB Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Patil, Sandeep R., Ramanathan, Sri, Shiraguppi, Riyazahamad M., Sodhiya, Prashant, Trevathan, Matthew B.
• Primary Examiner(s): THAI, HANH B
• Application Number:
  US14/524,442
• Publication Number:
  US 20150046410A1
• Time in Patent Office: 435 Days
• Field of Search: 707 37-638, 707690-692, 707/770
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 16/24578 using ranking

Abstract

Methods and systems for enhancing reliability in deduplication over storage clouds are provided. A method includes: determining a weight for each of a plurality of duplicate files based on parameters associated with a respective storage device of each of the plurality of duplicate files; and designating one of the plurality of duplicate files as a master copy based on the determined weight.

Claims

1. A method of file deduplication implemented in a computer infrastructure comprising a combination of hardware and software, the method comprising:

performing, by a computer processor, a file deduplication process comprising;

determining, by the computer processor, a weight for each of a plurality of duplicate files, wherein the weight is based on;

(i) parameters associated with a respective storage device of each of the plurality of duplicate files and (ii) a respective weighting factor associated with each one of the parameters; and

obtaining numerical values for the each one of the parameters and the respective weighting factors; and

designating, by the computer processor, one of the plurality of duplicate files as a master copy based on the determined weight.

2. The method of claim 1, wherein the parameters comprise static parameters and dynamic parameters.

3. The method of claim 1, wherein the parameters comprise at least one of vendor weight, mean time between failure, read performance, write performance, availability, and durability of a respective said storage device.

4. The method of claim 1, wherein the parameters comprise at least one of head flying height, remapped sector, error counts, spin up time, temperature, and data throughput of a respective said storage device.

5. The method of claim 1, wherein the weight of a particular one of the plurality of files is determined according to:

6. The method of claim 1, further comprising replacing remaining ones of the plurality of duplicate files, other than the master copy, with respective pointers pointing to the master copy.

7. The method of claim 1, further comprising performing the determining and the designating as part of a post deduplication process in a cloud environment.

8. The method of claim 1, further comprising performing the determining and the designating as part of an inline deduplication process in a cloud environment.

9. The method of claim 1, wherein a service provider at least one of creates, maintains, deploys and supports the computer infrastructure.

10. The method of claim 1, wherein steps of claim 1 are provided by a service provider on a subscription, advertising, and/or fee basis.

11. A system comprising:

one or more computer processors;

one or more computer readable hardware storage device;

program instructions stored on the one or more computer readable hardware storage device for execution by at least one of the one or more processors, the program instructions comprising;

program instructions to identify duplicate files stored at different storage devices;

program instructions to determine a weight for each one of the duplicate files based on;

(i) parameters associated with the storage devices and (ii) weighting factors defined for the parameters; and

program instructions to designate one of the duplicate files as a master copy based on the determined weights.

12. The system of claim 11, wherein the parameters are related to at least one of reliability, health, and user preference of the storage devices on which the duplicate files are stored.

13. The system of claim 11, wherein:

the parameters include at least one of vendor weight, mean time between failure, read performance, write performance, availability, and durability of the storage devices; and

the parameters include at least one of head flying height, remapped sector, error counts, spin up time, temperature, and data throughput of the storage devices.

14. A computer program product comprising:

one or more computer readable hardware storage device and program instructions stored on the one or more computer readable hardware storage device, the program instructions comprising;

program instructions to determine a hash value for each of a plurality of files;

program instructions to determine a set of duplicate files based on the hash values; and

program instructions to deduplicate the set of duplicate files, wherein the deduplicating comprises;

determining a weight for each one of the duplicate files, wherein the weight is based on parameters associated with storage devices;

designating a master copy of the set based on the weight of each one of the duplicate files; and

nominating remaining files in the set, other than the master copy, for deletion.

15. The computer program product of claim 14, wherein the parameters comprise static parameters that are related to at least one of reliability, health, and user preference of the storage devices.

16. The computer program product of claim 14, wherein the storage devices are different nodes in a cloud environment.