Cluster file system comprising data mover modules having associated quota manager for managing back-end user quotas

• US 10,078,639 B1
• Filed: 09/29/2014
• Issued: 09/18/2018
• Est. Priority Date: 09/29/2014
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Faibish, Sorin, Teymouri, Sassan, Bent, John M., Pedone, Jr., James M.
• Primary Examiner(s): Perveen, Rehana
• Assistant Examiner(s): Tran, Loc
• Application Number:
  US14/499,429
• Time in Patent Office: 1,450 Days
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/185 Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof details of archiving G06F16/11

Abstract

A cluster file system comprises a front-end file system, a back-end file system, data mover modules arranged between the front-end and back-end file systems, and a quota manager associated with at least a given one of data mover modules. The data mover modules are configured to control archiving of data between the front-end file system and the back-end file system for respective users based at least in part on respective user quotas established by the quota manager and identifying corresponding portions of the back-end file system available to the users. The front-end file system may comprise archive directories configured for respective ones of the users, with the data mover modules being configured to archive a given file from the front-end file system in the back-end file system responsive to a link to that file being stored in a corresponding one of the archive directories of the front-end file system.

Claims

1. An apparatus comprising:

at least one processing device comprising a processor coupled to a memory;

the at least one processing device implementing a front-end file system comprising a plurality of archive directories configured for respective ones of a plurality of users;

the at least one processing device implementing a back-end file system;

the at least one processing device implementing a plurality of data mover modules arranged between the front-end and back-end file systems; and

the at least one processing device implementing a quota manager associated with at least a given one of the plurality of data mover modules;

wherein the data mover modules are configured to control archiving of data between the front-end file system and the back-end file system for respective ones of the users based at least in part on respective user quotas established by the quota manager and identifying corresponding portions of the back-end file system available to said users; and

wherein the data mover modules are configured to archive a given file from the front-end file system in the back-end file system responsive to a link to that file being stored in a corresponding one of the archive directories of the front-end file system;

the data mover modules thereby being configured to support selective archiving of files from the front-end file system in the back-end file system responsive to respective links to those files being stored in corresponding ones of the archive directories, while automatically preventing archiving of other files for which such links are not stored in corresponding ones of the archive directories, so as to conserve storage space in the back-end file system and facilitate conformance with the user quotas;

the respective links to the selectively archived files thereby being stored in at least one of the archive directories;

wherein the quota manager is further configured to establish archive directory quotas for respective ones of the archive directories and to proactively recommend changes to the archive directory quotas based on current sizes of the archive directories relative to the established archive directory quotas.

2. The apparatus of claim 1 wherein archiving of temporary files from the front-end file system in the back-end file system is prevented by preventing storage of links to the temporary files in the archive directories.

3. The apparatus of claim 1 wherein said archive directories include the respective archive directory quotas.

4. The apparatus of claim 1 wherein the quota manager has an associated policy engine providing one or more policies for use by the quota manager in establishing the user quotas identifying corresponding portions of the back-end file system available to said users.

5. The apparatus of claim 1 wherein the quota manager is implemented internally to a given one of the data mover modules.

6. The apparatus of claim 1 wherein the data mover modules enforce the respective user quotas for their corresponding users.

7. The apparatus of claim 1 further comprising a plurality of back-end file systems implemented by the at least one processing device with the data mover modules being configured to control archiving of data between the front-end file system and each of the back-end file systems responsive to user quotas established by the quota manager for each of the back-end file systems.

8. The apparatus of claim 1 wherein the front-end file system and back-end file system are associated with respective different sets of one or more hierarchical storage tiers for use in hierarchical storage management.

9. The apparatus of claim 1 further comprising a plurality of compute nodes coupled to the front-end file system and wherein the data mover modules are implemented at least in part utilizing software running on respective ones of the compute nodes.

10. The apparatus of claim 1 wherein a given one of the data mover modules comprises a plurality of server nodes that are synchronized with one another utilizing a message passing interface (MPI) protocol.

11. The apparatus of claim 1 wherein the data mover modules are implemented at least in part utilizing a virtual layer of a parallel log-structured file system (PLFS) and wherein the virtual layer comprises PLFS middleware implemented by the at least one processing device and configured to migrate files between the front-end and back-end file systems.

12. The apparatus of claim 1 wherein the front-end file system comprises a hierarchical storage management application programming interface and wherein the data mover modules are configured to control movement of data between the front-end file system and the back-end file system responsive to information received via the hierarchical storage management application programming interface.

13. A processing platform comprising the apparatus of claim 1.

16. The apparatus of claim 1 wherein the apparatus is implemented in a cluster file system coupled to a computer system in an information processing system, the computer system comprising a plurality of compute nodes.

14. A method comprising:

configuring a plurality of data mover modules for arrangement between a front-end file system and a back-end file system;

configuring a quota manager associated with at least a given one of the plurality of data mover modules;

establishing via the quota manager a plurality of user quotas identifying corresponding portions of the back-end file system available to respective users; and

controlling archiving of data between the front-end file system and the back-end file system via the data mover modules based at least in part on the established user quotas;

wherein the front-end file system comprises a plurality of archive directories configured for respective ones of said users and controlling archiving further comprises archiving a given file from the front-end file system in the back-end file system via one or more of the data mover modules responsive to a link to that file being stored in a corresponding one of the archive directories of the front-end file system;

the data mover modules thereby being configured to support selective archiving of files from the front-end file system in the back-end file system responsive to respective links to those files being stored in corresponding ones of the archive directories, while automatically preventing archiving of other files for which such links are not stored in corresponding ones of the archive directories, so as to conserve storage space in the back-end file system and facilitate conformance with the user quotas;

the respective links to the selectively archived files thereby being stored in at least one of the archive directories;

wherein the quota manager is further configured to establish archive directory quotas for respective ones of the archive directories and to proactively recommend changes to the archive directory quotas based on current sizes of the archive directories relative to the established archive directory quotas; and

wherein the configuring, establishing and controlling are implemented using at least one processing device comprising a processor coupled to a memory.

17. The method of claim 14 wherein a given one of the data mover modules comprises a plurality of server nodes that are synchronized with one another utilizing a message passing interface (MPI) protocol.

18. The method of claim 14 wherein the data mover modules are implemented at least in part utilizing a virtual layer of a parallel log-structured file system (PLFS) and wherein the virtual layer comprises PLFS middleware configured to migrate files between the front-end and back-end file systems.

19. The method of claim 14 wherein the front-end file system comprises a hierarchical storage management application programming interface and wherein the data mover modules are configured to control movement of data between the front-end file system and the back-end file system responsive to information received via the hierarchical storage management application programming interface.

15. An article of manufacture comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device comprising a processor coupled to a memory causes the at least one processing device:

to configure a plurality of data mover modules for arrangement between a front-end file system and a back-end file system;

to configure a quota manager associated with at least a given one of the plurality of data mover modules;

to establish via the quota manager a plurality of user quotas identifying corresponding portions of the back-end file system available to respective users; and

to control archiving of data between the front-end file system and the back-end file system via the data mover modules based at least in part on the established user quotas;

wherein the front-end file system comprises a plurality of archive directories configured for respective ones of said users and controlling archiving further comprises archiving a given file from the front-end file system in the back-end file system via one or more of the data mover modules responsive to a link to that file being stored in a corresponding one of the archive directories of the front-end file system;

the data mover modules thereby being configured to support selective archiving of files from the front-end file system in the back-end file system responsive to respective links to those files being stored in corresponding ones of the archive directories, while automatically preventing archiving of other files for which such links are not stored in corresponding ones of the archive directories, so as to conserve storage space in the back-end file system and facilitate conformance with the user quotas;

the respective links to the selectively archived files thereby being stored in at least one of the archive directories;

wherein the quota manager is further configured to establish archive directory quotas for respective ones of the archive directories and to proactively recommend changes to the archive directory quotas based on current sizes of the archive directories relative to the established archive directory quotas.

20. The article of manufacture of claim 15 wherein the front-end file system comprises a hierarchical storage management application programming interface and wherein the data mover modules are configured to control movement of data between the front-end file system and the back-end file system responsive to information received via the hierarchical storage management application programming interface.