Optimizing a File System for Different Types of Applications in a Compute Cluster Using Dynamic Block Size Granularity

• US 20110258378A1
• Filed: 04/14/2010
• Published: 10/20/2011
• Est. Priority Date: 04/14/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Shah, Mansi Ajit, Pucha, Himabindu, Pandey, Prashant, Tewari, Renu, Sarkar, Prasenjit, Ananthanarayanan, Rajagopal, Gupta, Karan
• Granted Patent: US 9,021,229 B2
• Time in Patent Office: Days
• US Class Current: 711/114
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

Embodiments of the invention relate to optimizing a file system for different types of applications in a compute cluster using dynamic block size granularity. An exemplary embodiment includes reserving a predetermined number of storage allocation regions for each node in a cluster, wherein each storage allocation region comprises a set of contiguous data blocks on a single storage disk of locally attached storage; using a contiguous set of data blocks on a single storage disk of locally attached storage as a file system'"'"'s block allocation granularity for striping data to be stored in the file system for a compute operation in the cluster; and using a single data block of a shared storage subsystem as the file system'"'"'s block allocation granularity for striping data to be stored in the file system for a data access operation in the cluster.

Claims

1. A method for optimizing a file system for different types of applications in a compute cluster using dynamic block size granularity, comprising:

reserving a predetermined number of storage allocation regions for each node in a cluster, wherein each storage allocation region comprises a set of contiguous data blocks on a single storage disk of locally attached storage;

using a contiguous set of data blocks on a single storage disk of locally attached storage as a file system'"'"'s block allocation granularity for striping data to be stored in the file system for a compute operation in the cluster; and

using a single data block of a shared storage subsystem as the file system'"'"'s block allocation granularity for striping data to be stored in the file system for a data access operation in the cluster.

2. The method of claim 1, further comprising:

allocating data for a compute operation to at least one of the reserved storage allocation regions.

3. The method of claim 1, further comprising:

when an aggregate number of the reserved storage allocation regions for a node in the cluster is less than the predetermined number, reserving at least one additional storage allocation region until the aggregate number of reserved storage allocations regions for the node is equal to the predetermined number.

4. The method of claim 1, further comprising:

sending a compute operation to a node in the cluster with data for the compute operation allocated to the node'"'"'s locally attached storage.

5. The method of claim 1, further comprising:

replicating each contiguous set of data blocks allocated to a node'"'"'s locally attached storage to locally attached storage of a second node in the cluster.

6. The method of claim 1, further comprising:

allocating data for a data access operation to the shared storage subsystem.

7. The method of claim 1, further comprising:

tracking a location of each data block allocated to the shared storage subsystem and to locally attached storage in the file system.

8. The method of claim 1, wherein the data access operation is selected from the group consisting of a bookkeeping operation, a data transfer operation, a cache management operation, and a pre-fetching operation.

9. The method of claim 6, further comprising:

if a node with a compute operation has a failure, restarting the compute operation on a second node where data associated with the compute operation is replicated.

10. The method of claim 1, wherein the cluster hosts computing services for a remote client.

11. A system for optimizing a file system for different types of applications in a compute cluster using dynamic block size granularity, comprising:

a compute cluster, the compute cluster comprising;

a plurality of nodes, wherein each of the plurality of node comprises locally attached storage,

a shared storage subsystem coupled to each of the plurality of nodes, and

a file system manager coupled to the shared storage subsystem and each of the plurality of nodes, wherein the file system manager;

reserves a predetermined number of storage allocation regions for each of the plurality of nodes in the compute cluster, wherein each storage allocation region comprises a set of contiguous data blocks on a single storage disk of locally attached storage,

uses a contiguous set of data blocks on a single storage disk of locally attached storage as a file system'"'"'s block allocation granularity for striping data to be stored in the file system for a compute operation in the compute cluster, and

uses a single data block of a shared storage subsystem as the file system'"'"'s block allocation granularity for striping data to be stored in the file system for a data access operation in the compute cluster.

12. The system of claim 11, wherein the file system manager sends a compute operation to a node in the compute cluster with data for the compute operation allocated to the node'"'"'s locally attached storage.

13. The system of claim 11, wherein the file system manager reserves at least one additional storage allocation region until an aggregate number of the reserved storage allocations regions for a node is equal to the predetermined number when the aggregate number of the reserved storage allocation regions for the node is less than the predetermined number.

14. The system of claim 11, wherein the file system manager:

replicates each contiguous set of data blocks allocated to a node'"'"'s locally attached storage to locally attached storage of a second node in the compute cluster, and

restarts a compute operation on a second node where data associated with the compute operation is replicated if a node with the compute operation has a failure.

15. A computer program product for optimizing a file system for different types of applications in a compute cluster using dynamic block size granularity, comprising:

a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising;

computer readable program code configured to reserve a predetermined number of storage allocation regions for each node in a cluster, wherein each storage allocation region comprises a set of contiguous data blocks on a single storage disk of locally attached storage,

computer readable program code configured to use a contiguous set of data blocks on a single storage disk of locally attached storage as a file system'"'"'s block allocation granularity for striping data to be stored in the file system for a compute operation in the cluster, and

computer readable program code configured to use a single data block of a shared storage subsystem as the file system'"'"'s block allocation granularity for striping data to be stored in the file system for a data access operation in the cluster.

16. The computer program product of claim 15, wherein the computer readable program code further comprises:

computer readable program code configured to send a compute operation to a node in the cluster with data for the compute operation allocated to the node'"'"'s locally attached storage.

17. The computer program product of claim 15, wherein the computer readable program code further comprises:

computer readable program code configured to reserve at least one additional storage allocation region until an aggregate number of the reserved storage allocations regions for a node is equal to the predetermined number when the aggregate number of the reserved storage allocation regions for the node is less than the predetermined number.

18. The computer program product of claim 15, wherein the computer readable program code further comprises:

computer readable program code configured to track a location of each data block allocated to the shared storage subsystem and to locally attached storage in the file system.

19. The computer program product of claim 15, wherein the computer readable program code further comprises:

computer readable program code configured to replicate each contiguous set of data blocks allocated to a node'"'"'s locally attached storage to locally attached storage of a second node in the cluster.

20. The computer program product of claim 19, wherein the computer readable program code further comprises:

computer readable program code configured to restart a compute operation on a second node where data associated with the compute operation is replicated if the node with the compute operation has a failure.