Storing data in a distributed file system

• US 10,169,363 B2
• Filed: 07/02/2015
• Issued: 01/01/2019
• Est. Priority Date: 09/04/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Ioannou, Nikolas, Koltsidas, Ioannis, Kurmus, Anil, Pletka, Roman A., Sorniotti, Alessandro, Weigold, Thomas D
• Primary Examiner(s): Hoang, Son T
• Application Number:
  US14/790,351
• Publication Number:
  US 20160070715A1
• Time in Patent Office: 1,279 Days
• Field of Search: None
• CPC Class Codes: G06F 16/152 using file content signatures, e.g. hash values
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/182 Distributed file systems

Abstract

A device for storing data in a distributed file system, the distributed file system including a plurality of deduplication storage devices, includes a determination unit configured to determine a characteristic of first data to be stored in the distributed file system; an identification unit configured to identify one of the deduplication storage devices of the distributed file system as deduplication storage device for the first data based on the characteristic of the first data; and a storing unit configured to store the first data in the identified deduplication storage device such that the first data and second data being redundant to the first data are deduplicatable within the identified deduplication storage device.

Claims

1. A hardware device for storing data in a distributed file system, the distributed file system including a plurality of nonvolatile deduplication storage devices, the device comprising:

at least one memory storing computer-executable instructions; and

at least one processor configured to access the at least one memory and execute the computer-executable instructions to;

receive a request to store first data in the distributed file system;

determine a characteristic of the first data;

identify one of the plurality of nonvolatile deduplication storage devices of the distributed file system as a deduplication storage device for the first data based on the determined characteristic of the first data matching a characteristic of second data stored in the identified deduplication device;

determine that the first data is redundant to the second data based on the determined characteristic of the first data matching the characteristic of the second data; and

in response at least in part to determining that the first data is redundant to the second data, store the first data in the identified deduplication storage device such that the first data and the second data are redundantly stored in the identified deduplication storage device and are deduplicatable within the identified deduplication storage device after the first data is redundantly stored.

2. The device of claim 1, wherein the at least one processor is further configured to execute the computer-executable instructions to compare metadata of the first data and metadata of the second data.

3. The device of claim 2, wherein each of the metadata includes one or more of a digest and a fingerprint of one or more of the first data and the second data.

4. The device of claim 2, wherein, when a result of the comparison is negative, the at least one processor is configured to execute the computer-executable instructions to identify any deduplication storage device of the plurality of nonvolatile deduplication storage devices as the deduplication storage device for the first data.

5. The device of claim 2, wherein, when a result of the comparison is positive, the at least one processor is configured to execute the computer-executable instructions to identify the deduplication storage device storing the second data.

6. The device of claim 1, wherein the at least one processor is configured to determine the characteristic of the first data by executing the computer-executable instructions to compute a hash of content of the first data using a hash function, wherein the hash is the characteristic of the first data.

7. The device of claim 6, wherein the hash function is a generic non-cryptographic hash function.

8. The device of claim 1, wherein the at least one processor is configured to identify the deduplication storage device based on load balancing.

9. The device of claim 1, wherein the at least one processor is configured to store the first data in the identified deduplication storage device by executing the computer-executable instructions to perform a write request to the identified deduplication storage device.

10. The device of claim 9, wherein, when the write request fails, the at least one processor is configured to execute the computer-executable instructions to perform a write request to a random deduplication storage device.

11. A computerized system, comprising:

a distributed file system including a plurality of deduplication storage devices; and

the hardware device configured to store data in the distributed file system as claimed in claim 1.

12. The computerized system of claim 11, wherein each of the plurality of deduplication storage devices is configured to carry out a storage-internal deduplication.

13. The computerized system of claim 11, wherein the device for storing data in the distributed file system is arranged at the layer of the distributed file system.

14. A method, implemented by one or more computing systems, for storing data in a distributed file system, the distributed file system including a plurality of nonvolatile deduplication storage devices, the method comprising:

receiving a request to store first data in the distributed file system;

determining a characteristic of the first data;

identifying one of the plurality of nonvolatile deduplication storage devices of the distributed file system as a deduplication storage device for the first data based on the determined characteristic of the first data matching a characteristic of second data stored in the identified deduplication device;

determining that the first data is redundant to the second data based on the determined characteristic of the first data matching the characteristic of the second data; and

in response at least in part to determining that the first data is redundant to the second data, storing the first data in the identified deduplication storage device such that the first data and the second data are redundantly stored in the identified deduplication storage device and are deduplicatable within the identified deduplication storage device after the first data is stored.

15. The method of claim 14, further comprising comparing metadata of the first data and metadata of the second data.

16. The method of claim 15, wherein each of the metadata includes one or more of a digest and a fingerprint of one or more of the first data and the second data.

17. The method of claim 15, wherein, when a result of the comparison is negative, identifying one of the plurality of nonvolatile deduplication storage devices of the distributed file system as the deduplication storage device for the first data comprises identifying any deduplication storage device of the plurality of deduplication storage devices as the deduplication storage device for the first data.

18. The method of claim 15, wherein, when a result of the comparison is positive, identifying one of the plurality of nonvolatile deduplication storage devices of the distributed file system as the deduplication storage device for the first data comprises identifying the deduplication storage device storing the second data.

19. The method of claim 14, wherein determining the characteristic of the first data comprises computing a hash of content of the first data using a generic non-cryptographic hash function, wherein the hash is the characteristic of the first data.

20. A nontransitory computer readable storage medium having computer readable instructions stored thereon that, when executed by a computer, implement a method for storing data in a distributed file system, the distributed file system including a plurality of nonvolatile deduplication storage devices, the method comprising:

receiving a request to store first data in the distributed file system;

determining a characteristic of the first data;

identifying one of the plurality of nonvolatile deduplication storage devices of the distributed file system as deduplication storage device for the first data based on the determined characteristic of the first data matching a characteristic of second data stored in the identified deduplication device;

determining that the first data is redundant to the second data based on the determined characteristic of the first data matching the characteristic of the second data; and

in response at least in part to determining that the first data is redundant to the second data, storing the first data in the identified deduplication storage device such that the first data and the second data are redundantly stored in the identified deduplication storage device and are deduplicatable within the identified deduplication storage device after the first data is stored.