Space management for a hierarchical set of file systems

• US 10,592,479 B2
• Filed: 01/03/2017
• Issued: 03/17/2020
• Est. Priority Date: 01/03/2017
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Bhagwat, Deepavali M., Eshel, Marc, Hildebrand, Dean, Naik, Manoj P., Sawdon, Wayne A., Schmuck, Frank B., Tewari, Renu
• Primary Examiner(s): Morrison, Jay A
• Application Number:
  US15/397,663
• Publication Number:
  US 20180189319A1
• Time in Patent Office: 1,169 Days
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 11/1448 Management of the data involved in backup or backup restore
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/185 Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof details of archiving G06F16/11
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data

Abstract

One embodiment provides a method for storage management in a hierarchical file system. The method includes tracking directories for sub-file systems of the hierarchical set of file systems and maintaining a corresponding directory in a top-file system and a number of symbolic pointers in the corresponding directory which each point from the corresponding directory to a sub-file system where any portion of the tracked directory resides. If available space in a first sub-file system is below a threshold, the system identifies a second sub-file system with available space above the threshold, wherein the first sub-file system contains a first portion of the directory. If a second portion of the directory does not exist in the second sub-file system, the system creates the second portion of the directory in the second sub-file system and creates a link from the first portion of the directory to the second portion of the directory.

Claims

1. A method for storage management in a hierarchical set of file systems, the method comprising:

creating a structure for the hierarchical set of file systems including a top-file system and sub-file systems that each include files and directories;

tracking directories for the sub-file systems of the hierarchical set of file systems;

maintaining a directory while providing the directory is capable of residing in more than one sub-file system;

for each directory in the hierarchical set of file systems;

maintaining a corresponding directory in the top-file system; and

maintaining a number of symbolic pointers in the corresponding directory which each point from the corresponding directory to a sub-file system where any portion of the tracked directory resides; and

upon a request by a particular sub-file system for more inodes without any inodes available, revoking inode numbers from at least one other sub-file system having unused inodes;

wherein per-dataset failure isolation is limited by placement of files across multiple sub-file systems, and each directory in the sub-file systems is named using an inode number of a directory in the top file system.

2. The method of claim 1, further comprising:

creating a sub-file system of the hierarchical set of file systems based on applying a directory creation policy;

wherein;

maintaining comprises;

if available space in a first sub-file system is below a threshold, identifying a second sub-file system with available space above the threshold, wherein the first sub-file system contains a first portion of the directory; and

a namespace in the top file system and pointers to each of the multiple sub-file systems include a name of a directory in a sub-file system that is its inode number in the top file system portion.

3. The method of claim 2, further comprising:

if a second portion of the directory exists in the second sub-file system, creating a file in the second portion of the directory; and

parallelizing file system operations by issuing the file system operations on any number of sub-file systems simultaneously.

4. The method of claim 3, further comprising:

if a second portion of the directory does not exist in the second sub-file system;

creating the second portion of the directory in the second sub-file system;

creating the file in the second portion of the directory; and

creating a symbolic link from a corresponding directory in the top-file system to the second sub-file system, wherein the corresponding directory relates to the first portion of the directory.

5. The method of claim 2, further comprising:

if a second portion of the directory exists in the second sub-file system, moving a file into the second portion of the directory.

6. The method of claim 5, further comprising:

if a second portion of the directory does not exist in the second sub-file system;

creating the second portion of the directory in the second sub-file system;

moving the file into the second portion of the directory; and

creating a symbolic link from a corresponding directory in the top-file system to the second sub-file system, wherein the corresponding directory relates to the first portion of the directory.

7. The method of claim 2, further comprising:

if a second portion of the directory exists in the second sub-file system, creating a pointer for a file which points from the first portion of the directory to the second portion of the directory, wherein an extended part of a file resides in the second portion of the directory.

8. The method of claim 7, further comprising:

if a second portion of the directory does not exist in the second sub-file system;

creating the second portion of the directory in the second sub-file system;

creating a link from the first portion of the directory to the second portion of the directory; and

creating a pointer for the file which points from the first portion of the directory to the second portion of the directory, wherein an extended part of the file resides in the second portion of the directory.

9. The method of claim 2, wherein:

identifying comprises;

checking available space in sub-file systems that contain a portion of the directory; and

then checking available space in sub-file systems that do not contain a portion of the directory;

the sub-file systems are mapped across storage building blocks and include elastic file servers and storage devices; and

create, move and search operations performed on the sub-file systems are parallelized.

10. A computer program product for storage management in a hierarchical set of file systems, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:

create, by the processor, a structure for the hierarchical set of file systems including a top-file system and sub-file systems that each include files and directories;

track, by the processor, directories for the sub-file systems of the hierarchical set of file systems;

maintain, by the processor, a directory while providing the directory is capable of residing in more than one sub-file system;

for each directory in the hierarchical set of file systems;

maintain, by the processor, a corresponding directory in the top-file system; and

maintain, by the processor, a number of symbolic pointers in the corresponding directory which each point from the corresponding directory to a sub-file system where any portion of the tracked directory resides; and

upon a request by a particular sub-file system for more inodes without any inodes available, revoke, by the processor, inode numbers from at least one other sub-file system having unused inodes;

wherein per-dataset failure isolation is limited by placement of files across multiple sub-file systems, and each directory in the sub-file systems is named using an inode number of a directory in the top file system.

11. The computer program product of claim 10, further comprising program instructions executable by the processor to cause the processor to:

create, by the processor, a sub-file system of the hierarchical set of file systems based on applying a directory creation policy; and

if available space in a first sub-file system is below a threshold, identify, by the processor a second sub-file system with available space above the threshold, wherein the first sub-file system contains a first portion of the directory;

wherein a namespace in the top file system and pointers to each of the multiple sub-file systems include a name of a directory in a sub-file system that is its inode number in the top file system portion.

12. The computer program product of claim 11, further comprising program instructions executable by the processor to cause the processor to:

if a second portion of the directory exists in the second sub-file system, create, by the processor, a file in the second portion of the directory; and

parallelize, by the processor, file system operations by issuing the file system operations on any number of sub-file systems simultaneously.

13. The computer program product of claim 12, further comprising program instructions executable by the processor to cause the processor to:

if a second portion of the directory does not exist in the second sub-file system;

create, by the processor, the second portion of the directory in the second sub-file system;

create, by the processor, the file in the second portion of the directory; and

create, by the processor, a pointer for the file which points from the first portion of the directory to the second portion of the directory, wherein an extended part of the file resides in the second portion of the directory.

14. The computer program product of claim 11, further comprising program instructions executable by the processor to cause the processor to:

if a second portion of the directory exists in the second sub-file system, move, by the processor, a file into the second portion of the directory.

15. The computer program product of claim 14, further comprising program instructions executable by the processor to cause the processor to s:

if a second portion of the directory does not exist in the second sub-file system;

create, by the processor, the second portion of the directory in the second sub-file system;

move, by the processor, the file into the second portion of the directory; and

create, by the processor, a pointer for the file which points from the first portion of the directory to the second portion of the directory, wherein an extended part of the file resides in the second portion of the directory.

16. The computer program product of claim 11, further comprising program instructions executable by the processor to cause the processor to:

if a second portion of the directory exists in the second sub-file system;

create, by the processor, a pointer for a file which points from the first portion of the directory to the second portion of the directory, wherein an extended part of the file resides in the second portion of the directory.

17. The computer program product of claim 16, further comprising program instructions executable by the processor to cause the processor to:

if a second portion of the directory does not exist in the second sub-file system;

create, by the processor, the second portion of the directory in the second sub-file system; and

create, by the processor, a pointer for the file which points from the first portion of the directory to the second portion of the directory, wherein an extended part of the file resides in the second portion of the directory.

18. The computer program product of claim 11, further comprising program instructions executable by the processor to cause the processor to:

check, by the processor, available space in sub-file systems that contain a portion of the directory; and

then check, by the processor, available space in sub-file systems that do not contain a portion of the directory;

wherein;

the sub-file systems are mapped across storage building blocks and include elastic file servers and storage devices; and

create, move and search operations performed on the sub-file systems are parallelized.

19. An apparatus comprising:

a memory storing instructions; and

one or more processors executing the instructions to;

create a structure for a hierarchical set of file systems including a top-file system and sub-file systems that each include files and directories;

track directories for the sub-file systems of the hierarchical set of file systems;

maintain a directory while providing the directory is capable of residing in more than one sub-file system; and

upon a request by a particular sub-file system for more inodes without any inodes available, revoke inode numbers from at least one other sub-file system having unused inodes;

wherein per-dataset failure isolation is limited by placement of files across multiple sub-file systems, and each directory in the sub-file systems is named using an inode number of a directory in the top file system.

20. The apparatus of claim 19, wherein for each directory in the hierarchical set of file systems the one or more processors further execute instructions to:

create a sub-file system of the hierarchical set of file systems based on applying a directory creation policy;

maintain a corresponding directory in a top-file system;

maintain a number of symbolic pointers in the corresponding directory which each point from the corresponding directory to a sub-file system where any portion of the tracked directory resides; and

parallelize file system operations by issuing the file system operations on any number of sub-file systems simultaneously;

wherein;

the sub-file systems are mapped across storage building blocks and include elastic file servers and storage devices;

create, move and search operations performed on the sub-file systems are parallelized; and

a namespace in the top file system and pointers to each of the multiple sub-file systems include a name of a directory in a sub-file system that is its inode number in the top file system portion.