Optimizing a file system for different types of applications in a compute cluster using dynamic block size granularity

• US 9,021,229 B2
• Filed: 04/14/2010
• Issued: 04/28/2015
• Est. Priority Date: 04/14/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Ananthanarayanan, Rajagopal, Gupta, Karan, Pandey, Prashant, Pucha, Himabindu, Sarkar, Prasenjit, Shah, Mansi Ajit, Tewari, Renu
• Primary Examiner(s): Dudek, Jr., Edward
• Assistant Examiner(s): Verderamo, III, Ralph A
• Application Number:
  US12/760,476
• Publication Number:
  US 20110258378A1
• Time in Patent Office: 1,840 Days
• Field of Search: None
• US Class Current: 711/171
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

Embodiments of the invention relate to optimizing a file system for different types of applications in a compute cluster using dynamic block size granularity. An exemplary embodiment includes reserving a predetermined number of storage allocation regions for each node in a cluster, wherein each storage allocation region comprises a set of contiguous data blocks on a single storage disk of locally attached storage; using a contiguous set of data blocks on a single storage disk of locally attached storage as a file system'"'"'s block allocation granularity for striping data to be stored in the file system for a compute operation in the cluster; and using a single data block of a shared storage subsystem as the file system'"'"'s block allocation granularity for striping data to be stored in the file system for a data access operation in the cluster.

Claims

1. A method of using a dynamic storage data block size granularity in a computing cluster with a clustered file system for serving applications that initiate data access operations and applications that initiate compute operations, the computing cluster having a shared storage system and compute servers with locally attached storage, the method comprising:

prefetching a predetermined number of storage allocation regions for each compute server in said cluster, each storage allocation region being a set of contiguous data blocks on a single storage disk of locally attached storage to a compute server, and further prefetching at least one additional storage allocation region until the aggregate number of reserved storage allocations regions for a compute server is equal to the predetermined number if an aggregate number of the reserved storage allocation regions for the compute server in said cluster is less than the predetermined number;

allocating data for a compute operation to at least one of the reserved storage allocation regions, said allocating includes using a contiguous set of data blocks on a single storage disk of locally attached storage as the clustered file system'"'"'s block allocation granularity for striping data to be stored by said clustered file system for a compute operation in the cluster;

allocating data for a data access operation to said shared storage system, said allocating data for a data access operation includes using a single data block of said shared storage system as the clustered file system'"'"'s block allocation granularity for striping data to be stored in the clustered file system for a data access operation in the cluster;

sending a compute operation to be served by said clustered file system to a compute server having data for the compute operation allocated to compute server'"'"'s locally attached storage such that the compute operation and the data for the compute operation are co-located on the compute server.

2. The method of claim 1, further comprising replicating each contiguous set of data blocks allocated to a compute server'"'"'s locally attached storage to a second compute server'"'"'s locally attached storage in said cluster.

3. The method of claim 1, further comprising tracking a location of each data block stored by said file system to said storage system and to locally attached storage of the compute servers.

4. The method of claim 1, wherein the data access operation is a file system operation selected from the group consisting of a bookkeeping operation, a data transfer operation, a cache management operation, and a pre-fetching operation.

5. The method of claim 1, wherein said cluster hosts computing services for one or more remote clients in a cloud computing environment.

6. The method of claim 5, wherein the applications that initiate compute operations include data analytics applications.

7. The method of claim 1, wherein said block size granularity of a single block is a size in the range of 256 KB and 512 KB.

8. The method of claim 1, further comprising:

restarting the compute operation on a second compute server where data associated with the compute operation is replicated from the compute server if the compute server serving the compute operation has a failure.

20. The computer program product of claim 7, wherein said manager restarts the compute operation on a second compute server where data associated with the compute operation is replicated from the compute server if the compute server serving the compute operation has a failure.

9. A computing cluster using a dynamic storage block size granularity for serving applications that initiate data access operations and applications that initiate compute operations, said cluster comprising:

a shared storage system;

compute servers that serve file system compute operations and having locally attached; and

a file system manager of said file system, said manager;

prefetches a predetermined number of storage allocation regions for each compute server in said cluster, each storage allocation region being a set of contiguous data blocks on a single storage disk of locally attached storage to a compute server, and further prefetches at least one additional storage allocation region until the aggregate number of reserved storage allocations regions for a compute server is equal to the predetermined number if an aggregate number of the reserved storage allocation regions for the compute server in said cluster is less than the predetermined number,

allocates data for a compute operation to at least one of the reserved storage allocation regions, said allocating includes using a contiguous set of data blocks on a single storage disk of locally attached storage as said file system'"'"'s block allocation granularity for striping data to be stored by said file system for a compute operation in the cluster,

allocates data for a data access operation to said shared storage system, said allocating data for a data access operation includes using a single data block of said shared storage system as the file system'"'"'s block allocation granularity for striping data to be stored in the file system for a data access operation in the cluster, and

sends a compute operation to be served by said file system to a compute server having data for the compute operation allocated to compute server'"'"'s locally attached storage such that the compute operation and the data for the compute operation are co-located on the compute server.

10. The computing cluster of claim 9, wherein said manager:

replicates each contiguous set of data blocks allocated from a compute server'"'"'s locally attached storage to a second compute server'"'"'s locally attached storage in said cluster.

11. The computing cluster of claim 9, wherein said cluster hosts computing services for one or more remote clients in a cloud computing environment.

12. The computing cluster of claim 11, wherein the applications that initiate compute operations include data analytics applications.

13. The computing cluster of claim 9, wherein said block size granularity of a single block is a size in the range of 256 KB and 512 KB.

14. The computing cluster of claim 9, wherein said manager restarts the compute operation on a second compute server where data associated with the compute operation is replicated from the compute server if the compute server serving the compute operation has a failure.

15. A computer program product of using a dynamic storage data block size granularity in a computing cluster with a clustered file system for serving applications that initiate data access operations and applications that initiate compute operations, the computing cluster having a shared storage system and compute servers with locally attached storage, said program product comprising a non-transitory computer readable storage medium having computer readable program code embodied therewith, said program code being executable by a computer to:

prefetch a predetermined number of storage allocation regions for each compute server in said cluster, each storage allocation region being a set of contiguous data blocks on a single storage disk of locally attached storage to a compute server, and further prefetches at least one additional storage allocation region until the aggregate number of reserved storage allocations regions for a compute server is equal to the predetermined number if an aggregate number of the reserved storage allocation regions for the compute server in said cluster is less than the predetermined number;

allocate data for a compute operation to at least one of the reserved storage allocation regions, said allocating includes using a contiguous set of data blocks on a single storage disk of locally attached storage as said clustered file system'"'"'s block allocation granularity for striping data to be stored by said clustered file system for a compute operation in the cluster;

allocate data for a data access operation to said shared storage system, said allocating data for a data access operation includes using a single data block of said shared storage system as the clustered file system'"'"'s block allocation granularity for striping data to be stored in the clustered file system for a data access operation in the cluster, and

sends a compute operation to be served by said file system to a compute server having data for the compute operation allocated to compute server'"'"'s locally attached storage such that the compute operation and the data for the compute operation are co-located on the compute server.

16. The computer program product of claim 15, said program code being further executable by a computer to track a location of each data block stored by said storage system and to locally attached storage of compute servers.

17. The computer program product of claim 15, said program code being further executable by a computer to replicate each contiguous set of data blocks allocated from a compute server'"'"'s locally attached storage to a second compute server'"'"'s locally attached storage in said cluster.

18. The computer program product of claim 15, wherein said cluster hosts computing services for one or more remote clients in a cloud computing environment and the applications that initiate compute operations include data analytics applications.

19. The computer program product of claim 15, wherein said block size granularity of a single block is a size in the range of 256 KB and 512 KB.