Dynamically adjusting the number of replicas of a file according to the probability that the file will be accessed within a distributed file system

• US 9,348,707 B2
• Filed: 12/18/2013
• Issued: 05/24/2016
• Est. Priority Date: 12/18/2013
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Maurice, Lisa B., Kashyap, Sujatha, Robichaux, Joseph H.
• Primary Examiner(s): Choi, Yuk Ting
• Application Number:
  US14/132,289
• Publication Number:
  US 20150169612A1
• Time in Patent Office: 888 Days
• Field of Search: 707/610
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1492 by run-time replication performed by the application software
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/184 implemented as replicated file system
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

In a distributed file system with a number of replicas of a file set to two or more replicas, a computer sets a timer to track a time since a last access to the file, wherein the replicas of the file are distributed across two or more nodes within the distributed file system. Responsive to an access to the file prior to the timer reaching a first timer window threshold, the computer resets the timer. Responsive to the timer matching a first timer window threshold, the computer automatically reduces a number of replicas of the file within the distributed file system, wherein the probability that the file will be accessed prior to the first timer window threshold is greater than the probability that the file will be accessed after the first timer window threshold.

Claims

1. A system for dynamically managing a number of replicas of a file in a distributed file system, comprising:

one or more processors;

a memory coupled to at least one of the processors;

a set of computer program instructions stored in the memory and executed by at least one of the processors in order to perform actions of;

in a distributed file system with a number of replicas of a file set to two or more replicas, setting a timer to track a time since a last access to the file, wherein the replicas of the file are distributed across two or more nodes within the distributed file system;

responsive to an access to the file prior to the timer reaching a first timer window threshold, resetting the timer; and

responsive to the timer matching a first timer window threshold, automatically reducing a number of replicas of the file within the distributed file system, wherein the probability that the file will be accessed prior to the first timer window threshold is greater than the probability that the file will be accessed after the first timer window threshold.

2. The system according to claim 1, wherein the distributed file system is a hadoop distributed file system.

3. The system according to claim 1, wherein the processors perform additional actions comprising:

responsive to the timer matching a second timer window threshold, automatically reducing the number of the replicas of the file; and

responsive to the number of replicas of the file reaching a low threshold, automatically moving the file from the distributed file system to a permanent storage tape.

4. The system according to claim 1, wherein the processors perform additional actions comprising:

responsive to the timer matching the first timer window threshold, resetting the timer to count to a second timer window threshold; and

responsive to receiving a write access prior to the timer reaching the second timer window threshold, increasing the number of replicas of the file and resetting the timer to count to the first timer window threshold.

5. The system according to claim 1, wherein the processors perform additional actions comprising:

responsive to the timer matching the first timer window threshold, resetting the timer to count to a second timer window threshold;

responsive to the timer matching the second timer window threshold, resetting the timer to count to a third timer window threshold; and

responsive to receiving a write access prior to the timer reaching the third timer window threshold, increasing the number of replicas of the file by two and resetting the timer to count to the first timer window threshold.

6. The system according to claim 1, wherein the processors perform additional actions comprising:

responsive to the timer matching the first timer window threshold, resetting the timer to count to a second timer window threshold;

responsive to the timer matching the second timer window threshold, resetting the timer to count to a third timer window threshold; and

responsive to receiving a read access prior to the timer reaching the third timer window threshold, increasing the number of replicas of the file and resetting the timer to count to the second timer window threshold.

7. The system according to claim 1, wherein the processors perform additional actions comprising:

reading a plurality of log entries for each of a plurality of accesses to a plurality of files within the distributed file system over a window time period;

identifying a first time by which a first threshold percentage of the log entries identifying accesses to the plurality of files occur; and

setting the first timer window threshold to the first time.

8. A computer program product for dynamically managing a number of replicas of a file in a distributed file system, the computer program product comprising a non-transitory computer readable storage medium having program code embodied therewith, the program code executable by a computer system to:

in a distributed file system with a number of replicas of a file set to two or more replicas, set, by the computer system, a timer to track a time since a last access to the file, wherein the replicas of the file are distributed across two or more nodes within the distributed file system;

responsive to an access to the file prior to the timer reaching a first timer window threshold, reset, by the computer system, the timer; and

responsive to the timer matching a first timer window threshold, automatically reduce, by the computer system, a number of replicas of the file within the distributed file system, wherein the probability that the file will be accessed prior to the first timer window threshold is greater than the probability that the file will be accessed after the first timer window threshold.

9. The computer program product according to claim 8, further comprising the program code executable by the computer system to:

responsive to the timer matching a second timer window threshold, automatically reduce, by the computer system, the number of the replicas of the file; and

responsive to the number of replicas of the file reaching a low threshold, automatically move, by the computer system, the file from the distributed file system to a permanent storage tape.

10. The computer program product according to claim 8, further comprising the program code executable by the computer system to:

responsive to the timer matching the first timer window threshold, reset, by the computer system, the timer to count to a second timer window threshold; and

responsive to receiving a write access prior to the timer reaching the second timer window threshold, increase, by the computer system, the number of replicas of the file and resetting the timer to count to the first timer window threshold.

11. The computer program product according to claim 8, further comprising the program code executable by the computer system to:

responsive to the timer matching the first timer window threshold, reset, by the computer system, the timer to count to a second timer window threshold;

responsive to the timer matching the second timer window threshold, reset, by the computer system, the timer to count to a third timer window threshold; and

responsive to receiving a write access prior to the timer reaching the third timer window threshold, increase, by the computer system, the number of replicas of the file by two and resetting the timer to count to the first timer window threshold.

12. The computer program product according to claim 8, further comprising the program code executable by the computer system to:

responsive to the timer matching the first timer window threshold, reset, by the computer system, the timer to count to a second timer window threshold;

responsive to the timer matching the second timer window threshold, reset, by the computer system, the timer to count to a third timer window threshold; and

responsive to receiving a read access prior to the timer reaching the third timer window threshold, increase, by the computer system, the number of replicas of the file and resetting the timer to count to the second timer window threshold.

13. The computer program product according to claim 8, further comprising the program code executable by the computer system to:

read, by the computer system, a plurality of log entries for each of a plurality of accesses to a plurality of files within the distributed file system over a window time period;

identify, by the computer system, a first time by which a first threshold percentage of the log entries identifying accesses to the plurality of files occur; and

set, by the computer system, the first timer window threshold to the first time.