Switch-Aware Parallel File System

• US 20110296422A1
• Filed: 05/27/2010
• Published: 12/01/2011
• Est. Priority Date: 05/27/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Hildebrand, Dean, Tewari, Renu
• Granted Patent: US 8,701,113 B2
• Time in Patent Office: Days
• US Class Current: 718/102
• CPC Class Codes: G06F 16/1824 implemented using Network-attached Storage [NAS] architecture distributed or networked storage systems G06F3/067; protocols for distributed storage of data in a network H04L67/1097

Abstract

Embodiments of the invention related to a switch-aware parallel file system. A computing cluster is partitioned into a plurality of computing cluster building blocks comprising a parallel file system. Each computing cluster building block comprises a file system client, a storage module, a building block metadata module, and a building block network switch. The building block metadata module tracks a storage location of data allocated by the storage module within the computing cluster building block. The computing cluster further comprises a file system metadata module that tracks which of the plurality of computing cluster building blocks data is allocated among within the parallel file system. The computing cluster further comprises a file system network switch to provide the parallel file system with access to each of the plurality of computing cluster building blocks and the file system metadata module. At least one additional computing cluster building block is added to the computing cluster, if resource utilization of the computing cluster exceeds a pre-determined threshold.

Claims

1. A file system, comprising:

a plurality of computing cluster building blocks comprising a parallel file system, each of the plurality of computing cluster building blocks comprising;

a file system client, the file system client serves a file system application request,

a storage module coupled to the file system client, the storage module serves a data request for the file system client, and

a building block metadata module coupled to the storage module, the building block metadata module tracks a storage location of data allocated by the storage module within the computing cluster building block; and

a file system metadata module coupled to the plurality of computing cluster building blocks, the file system metadata module tracks which of the plurality of computing cluster building blocks data is allocated among within the parallel file system.

2. The system of claim 1, comprising:

a file system network switch coupled to each of the plurality of computing cluster building blocks and the file system metadata module, the file system network switch provides the parallel file system with access to;

each of the plurality of computing cluster building blocks, and

the file system metadata module.

3. The system of claim 2, wherein each of the plurality of computing cluster building blocks comprises a building block network switch, the building block network switch:

is coupled to the file system client, the storage module, and the building block metadata module within the computing cluster building block, and

provides the parallel file system with access to the file system client, the storage module, and the building block metadata module within a computing cluster building block.

4. The system of claim 3, wherein the file system client for each of the plurality of computing cluster building blocks has access to data allocated among each of the plurality of computing cluster building blocks.

5. The system of claim 1, comprising:

a job scheduler coupled to each of the plurality of computing cluster building blocks, the job scheduler spawns a file system job on at least one of the plurality of computing cluster building blocks.

6. The system of claim 5, wherein prior to spawning the job, the job scheduler schedules the file system job to be spawned.

7. The system of claim 5, wherein if there is pre-existing data for the file system job in the parallel file system, the job scheduler spawns the file system job on the at least one of the plurality of computing cluster building blocks where data for the file system job is located.

8. The system of claim 5, wherein if there is no pre-existing data for the file system job in the parallel file system, the job scheduler spawns the file system job on the at least one of the plurality of parallel computing cluster building blocks with the most resources being available.

9. The system of claim 1, wherein the building block metadata module creates a data object for data to be written to, the data object being created in response to a data request by the file system client.

10. The system of claim 9, wherein the file system client:

writes data to the data object by striping the data across at least one storage module in the same computing cluster building block as the file system client, and

reads the data from the data object by accessing the at least one storage module.

11. A method, comprising:

partitioning a computing cluster into a plurality of computing cluster building blocks, wherein;

the plurality of computing cluster building blocks comprises a parallel file system, and

each computing cluster building block comprises a file system client, a storage module, a building block metadata module, and a building block network switch;

providing the parallel file system with access to each of the plurality computing cluster building blocks;

tracking which of the plurality of computing cluster building blocks data is allocated among within the parallel file system; and

tracking a storage location of data allocated within each of the computing cluster building blocks.

12. The method of claim 11, wherein providing the parallel file system with access to each of the plurality of computing cluster building blocks comprises:

providing the parallel file system with access to the file system client, the storage module, and the metadata module within each of the plurality of computing cluster building blocks.

13. The method of claim 12, wherein providing the parallel file system with access to each of the plurality of parallel computing cluster building blocks comprises:

providing a file system client within each of the plurality of computing cluster building blocks with access to data allocated among each of the plurality of computing cluster building blocks.

14. The method of claim 11, comprising:

creating a data object for data to be written to within a computing cluster building block, the data object being created in response to a data request by a file system client within the computing cluster building block.

15. The method of claim 11, comprising:

scheduling a file system job to be spawned on at least one of the plurality of computing cluster building blocks; and

at a point-in-time the file system job was scheduled to be spawned, spawning the file system job on at least one of the plurality of computing cluster building blocks.

16. The method of claim 15, wherein if there is pre-existing data for the file system job in the parallel file system, the file system job is spawned on the at least one of the plurality of computing cluster building blocks where data for the file system job is located.

17. The method of claim 15, wherein if there is no pre-existing data for the file system job in the parallel file system, the file system job is spawned on the at least one of the plurality of computing cluster building blocks with the most resources being available.

18. The method of claim 11, comprising:

if resource utilization of the computing cluster exceeds a pre-determined threshold, adding at least one additional computing cluster building block to the computing cluster.

19. A computer program product, comprising:

a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising;

computer readable program code to provide a parallel file system with access to each of a plurality of computing cluster building blocks in a computing cluster, wherein;

the plurality of computing cluster building blocks comprises the parallel file system, and

each computing cluster building block comprises a file system client, a storage module, a building block metadata module, and a building block network switch;

computer readable program code to create a data object for data to be written to in response to a data request by a file system client within one of the plurality of computing cluster building block, the data object being created in a computing cluster building block where the file system client is located;

computer readable program code to track which of the plurality of computing cluster building blocks data is allocated among within the parallel file system;

computer readable program code to track a storage location of data allocated within each of the plurality of computing cluster building blocks; and

computer readable program code to provide a file system client for each of the plurality of computing cluster building blocks with access to data allocated among each of the plurality of computing cluster building blocks.

20. The computer program product of claim 19, comprising:

computer readable program code to spawn a file system job on at least one of the plurality of computing cluster building blocks, wherein;

if there is pre-existing data for the file system job in the file system, the file system job is spawned on the at least one of the plurality of parallel computing cluster building blocks where data for the file system job is located, and

if there is no pre-existing data for the file system job in the file system, the file system job is spawned on the at least one of the plurality of parallel computing cluster building blocks the most resources being available.