Transactional processing for clustered file systems

• US 8,275,815 B2
• Filed: 08/25/2008
• Issued: 09/25/2012
• Est. Priority Date: 08/25/2008
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Aronovich, Lior, Toaff, Yair, Paz, Gil, Asher, Ron
• Primary Examiner(s): Casanova, Jorge A
• Application Number:
  US12/197,953
• Publication Number:
  US 20100049718A1
• Time in Patent Office: 1,492 Days
• Field of Search: 707/826
• US Class Current: 707/826
• CPC Class Codes: G06F 12/0802 Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
• CPC Class Codes: G06F 12/0815 Cache consistency protocols
• CPC Class Codes: G06F 12/0828 with concurrent directory accessing, i.e. handling multiple concurrent coherency transactions
• CPC Class Codes: G06F 12/084 with a shared cache
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/16 File or folder operations, e.g. details of user interfaces specifically adapted to file systems
• CPC Class Codes: G06F 16/172 Caching, prefetching or hoarding of files
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 16/1865 Transactional file systems
• CPC Class Codes: G06F 16/24552 Database cache management
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 2212/604 Details relating to cache allocation

Abstract

Systems and methods for transactional processing within a clustered file system wherein user defined transactions operate on data segments of the file system data. The users are provided within an interface for using a transactional mechanism, namely services for opening, writing and rolling-back transactions. A distributed shared memory technology is utilized to facilitate efficient and coherent cache management within the clustered file system based on the granularity of data segments (rather than files).

Claims

1. A method comprising:

providing a clustered file system (CFS) residing on a cluster of nodes for accessing a shared storage of file system data; and

providing a local cache memory on each node to reduce file system access to the shared storage and for processing modifications to the file system data;

providing a list of data segments modified within an ongoing transaction; and

providing a distributed shared memory (DSM) agent on each node wherein;

the DSM agents collectively manage access permissions to the entire space of file system data on a data segment basis;

the DSM agents utilize the distributed cache memories of the CFS as a virtual shared cache to provide transaction based modifications for user defined transactions on a data segment basis and to provide transaction based modifications of CFS defined transactions on the data segment basis;

each node comprises a local transaction journal for storing committed transactions generated by users on that node;

a rollback procedure scans the list to identify the location of the latest contents of a data segment prior to the transaction;

for each modified data segment, the rollback procedure identifies the location of the latest contents prior to the transaction by;

if the data segment was marked as modified in the cache at the time it was inserted into the list, then the latest contents of this data segment appears only in the journal;

otherwise, if the data segment was not marked as modified in the cache at the time it was inserted into the list, then the latest contents of this data segment appears in its final location in the shared storage;

the type of each data segment is recorded in the list during insertion of the data segment into the list;

and wherein;

all data segments in the list whose latest contents appear in their final location are discarded from the cache;

for all the other data segments in the list, their latest contents are restored from the journal into the cache, by scanning the journal from its ending to its beginning and considers the first occurrences of these data segments in the journal, and then setting their modification indication to true; and

the procedure releases the exclusive permissions on all the data segments involved in the cancelled transaction.

2. The method of claim 1, including:

the DSM agents determine the latest contents of file system data to maintain coherency between the distributed cache memories of the CFS.

3. The method of claim 2, wherein:

in response to a user request to a local node, user operations are applied to data segments in the associated local cache memory, including reading requested data segments to the local cache memory and modifying data segments within the local cache memory, in accordance with permissions granted by the DSM agents.

4. The method of claim 3, wherein:

users performing read only operations are allowed to access the file system data concurrently, while the operations of users that require access for modification of a same data segment are serialized.

5. The method of claim 1, including:

each node having a journal for storing committed transactions generated by users on that node.

6. The method of claim 5, including:

a commit operation wherein the modified data segments are written from the local cache memory to the transaction journal of the local node.

7. The method of claim 5, including:

a checkpoint operation wherein the modified and committed data segments are written from the local cache memory to a final location in the shared storage.

8. The method of claim 5, including:

a roll-back operation, for cancellation of a current transaction, wherein data segments are written from the transaction journal to the associated local cache memory to restore the local cache memory to its state prior to the transaction.

9. The method of claim 5, including:

a recovery operation, wherein upon failure of one or more nodes, data segments are written from the transaction journals of the failed nodes to their final locations in the shared storage, for recovering file system data.

10. The method of claim 9, including:

the recovery procedure scans concurrently the transaction journals of the one or more failed nodes, beginning with the latest complete transaction in each journal, and following a descending order of the recentness of the transactions.

11. The method of claim 10, including:

only the most recent occurrence of each data segment is considered and, for each such data segment the procedure determines if it should be copied to its final location in shared storage by validating that ownership of the data segment is not associated with any of the remaining operational nodes.

12. The method of claim 11, including:

if the validation is true, ownership of such data segments are assigned to the local DSM agent.

13. The method of claim 12, including:

if the validation is true, and if a transaction identifier of the respective data segment contents in its final location is smaller in order than a transaction identifier of the respective data segment contents in the transaction journal, then the data segment is copied from the transaction journal to its final location in the shared storage.

14. The method of claim 1, wherein:

each transaction acquires exclusive access permissions on all data segments it modifies.

15. The method of claim 14, wherein:

a release of the exclusive access permissions accompanies termination of the transaction.

16. The method of claim 1, wherein:

the DSM agents provide a global ordering of the recentness of transactions and data segment contents across the cluster.

17. The method of claim 16, wherein:

the global ordering is based on a termination time for each transaction.

18. The method of claim 16, including:

the global ordering is based on assignment of a transaction identifier which is unique, with respect to all other transaction identifiers existing in the CFS, at the time of a commit operation of a transaction.

19. The method of claim 18, including:

a release of access permissions on the data segments of the transaction accompanies the assignment of the transaction identifier.

20. The method of claim 18, including:

the transaction identifier is included in file system metadata.

21. The method of claim 1, wherein:

each node has a CFS agent for maintaining a local set of data segments in the local cache memory and associated local parameters which include an access permission and ownership by the local DSM agent.

22. The method of claim 21, wherein:

each CFS agent performs checkpoint operations only on modified and committed data segments that the local DSM agent owns.

23. The method of claim 22, wherein:

concurrent checkpoints are performed by CFS agents on different nodes.

24. The method of claim 22, wherein:

the modified and committed data segments whose current owner is the local DSM agent are a subset of the data segments included in a respective transaction journal of the CFS agent and include;

data segments that should be written by the CFS agent to their final location in the shared storage within a checkpoint operation, and data segments that should be written on behalf of the CFS agent after its failure and within a recovery operation by another recovering CFS agent.

25. The method of claim 21, wherein:

the DSM agent that owns a data segment insures that the latest contents of that data segment is written to its final location in the shared storage, prior to transferring ownership to another DSM agent.

26. The method of claim 21, wherein:

if a CFS agent is required to evacuate a modified data segment that is currently owned by the local DSM agent from its local cache memory, the CFS agent first writes the data segment to its final location in the shared storage.

27. The method of claim 1, including:

during a checkpoint operation, users that require access for modification of a data segment being written within the checkpoint operation, are provided with a shadow data segment, whose contents is identical to that of the original data segment used by the checkpoint operation.

28. The method of claim 27, including:

when the checkpoint operation completes, the shadow data segment replaces in local cache memory the original data segment used by the checkpoint operation.

29. The method of claim 1, including:

in a procedure for allocating a new data segment, wherein an associated cache data segment and metadata data segment are loaded into the cache memory and modified in the process of allocating the data segment, the cache data segment and metadata data segments being added to a list of data segments modified within the associated transaction, and the DSM agents then assigning an exclusive permission on the newly allocated data segment.

30. The method of claim 29, including:

in a procedure for de-allocating an already allocated data segment, the procedure insuring that there is an active exclusive permission on the de-allocated disk data segment.

31. The method of claim 30, including:

the procedure removing the de-allocated data segment from the list of data segments modified within the associated transaction and inserting into the list metadata data segments that were modified in the process of de-allocating the data segment.

32. The method of claim 1, including:

in a procedure for marking a retrieved data segment as modified, the procedure inserting the data segment into a list of data segments modified in the associated transaction, accompanied with an indication of whether this data segment was marked as modified before this operation.

33. A non-transitory computer program product for managing access in a clustered file system (CFS) to a shared storage of file system data, each node of the CFS having a local cache memory, the computer program product comprising:

a computer usable medium having computer usable program code embodied therewith, the computer usable code comprising;

computer usable program code configured to provide a distributed shared memory (DSM) agent on each node;

computer usable program code configured to enable the DSM agents to collectively manage access permissions to the entire space of file system data on a data segment basis using the distributed cache memories of the CFS as a virtual shared cache;

computer usable program code configured to provide a list of data segments modified within an ongoing transaction; and

computer usable program code configured to provide transaction based modifications for user defined transactions on a data segment basis and to provide transaction based modifications for CFS defined transactions on the data segment basis, wherein;

each node comprises a local transaction journal for storing committed transactions generated by users on that node;

a rollback procedure scans the list to identify the location of the latest contents of a data segment prior to the transaction; and

for each modified data segment, the rollback procedure identifies the location of the latest contents prior to the transaction by;

if the data segment was marked as modified in the cache at the time it was inserted into the list, then the latest contents of this data segment appears only in the journal;

otherwise, if the data segment was not marked as modified in the cache at the time it was inserted into the list, then the latest contents of this data segment appears in its final location in the shared storage;

the type of each data segment is recorded in the list during insertion of the data segment into the list;

and wherein;

all data segments in the list whose latest contents appear in their final location are discarded from the cache;

for all the other data segments in the list, their latest contents are restored from the journal into the cache, by scanning the journal from its ending to its beginning and consider the first occurrences of these data segments in the journal, and then setting their modification indication to true; and

the procedure releases the exclusive permissions on all the data segments involved in the cancelled transaction.

34. The computer program product of claim 33, further comprising:

computer usable program code configured to enable the DSM agents determine the latest contents of file system data to maintain coherency between the distributed cache memories of the CFS.

35. The computer program product of claim 34, further comprising:

computer usable program code configured to provide, in response to a user request to a local node, applying user operations to data segments in the associated local cache memory, including reading requested data segments to the local cache memory and modifying data segments within the local cache memory, in accordance with permissions granted by the DSM agents.

36. The computer program product of claim 35, further comprising:

computer usable program code configured to enable users that perform read only operations access to the file system data concurrently, while the operations of users that require access for modification of a same data segment are serialized.

37. The computer program product of claim 33, further comprising:

computer usable program code configured to enable each agent to access a local journal for storing committed transactions generated by users on that node.

38. The computer program product of claim 37, further comprising:

computer usable program code configured to provide a commit operation wherein the modified data segments are written from the local cache memory to the transaction journal of the local node.

39. The computer program product of claim 37, further comprising:

computer usable program code configured to provide a checkpoint operation wherein the modified and committed data segments are written from the local cache memory to a final location in the shared storage.

40. The computer program product of claim 37, further comprising:

computer usable program code configured to provide a roll-back operation, for cancellation of a current transaction, wherein data segments are written from the transaction journal to the associated local cache memory to restore the local cache memory to its state prior to the transaction.

41. The computer program product of claim 37, further comprising:

computer usable program code configured to provide a recovery operation, wherein upon failure of one or more nodes, data segments are written from the transaction journals of the failed nodes to their final locations in the shared storage, for recovering file system data.

42. The computer program product of claim 37, further comprising:

computer usable program code configured to enable for each transaction, exclusive access permissions to be acquired on all data segments it modifies.

43. The computer program product of claim 33, further comprising:

computer usable program code configured to enable the DSM agents to provide a global ordering of the recentness of transactions and data segment contents across the cluster.

44. The computer program product of claim 33, further comprising:

computer usable program code configured to provide each node with a CFS agent for maintaining a local set of data segments in the local cache memory and associated local parameters which include an access permission and ownership by one of the DSM agents.

45. The computer program product of claim 44, further comprising:

computer usable program code configured to enable the CFS agent to perform checkpoint operations only on modified and committed data segments that the local DSM agent owns.

46. The computer program product of claim 45, further comprising:

computer usable program code configured to enable concurrent checkpoints to be performed by CFS agents on different nodes.

47. The computer program product of claim 44, further comprising:

computer usable program code configured to enable the DSM agent that owns a data segment to insure that the latest contents of that data segment is written to its final location in the shared storage, prior to transferring ownership to another DSM agent.

48. The computer program product of claim 44, further comprising:

computer usable program code configured to, if a CFS agent is required to evacuate a modified data segment that is currently owned by the local DSM agent from its local cache memory, the CFS agent first writes the data segment to its final location in the shared storage.

49. The computer program product of claim 33, further comprising:

computer usable program code configured to enable, during a checkpoint operation, that users requiring access for modification of a data segment being written within the checkpoint operation, be provided with a shadow data segment, whose contents is identical to that of the original data segment used by the checkpoint operation.

50. The computer program product of claim 49, further comprising:

computer usable program code configured to provide, when the checkpoint operation completes, replacing in local cache memory the original data segment used by the checkpoint operation with the shadow data segment.

51. The computer program product of claim 33, further comprising:

computer usable program code configured to provide a list of data segments modified within an ongoing transaction.

52. The computer program product of claim 51, further comprising:

computer usable program code configured to enable a rollback procedure to scan the list to identify the location of the latest contents of a data segment prior to the transaction.

53. The computer product program of claim 51, further comprising:

computer usable program code configured to enable a recovery procedure to scan concurrently the transaction journals of one or more failed nodes, beginning with the latest complete transaction in each journal, and following a descending order of the recentness of the transactions.

54. The computer product program of claim 33, further comprising:

computer usable program code configured to provide, in a procedure for allocating a new data segment, wherein an associated cache data segment and metadata data segment are loaded into the cache memory and modified in the process of allocating the data segment, the cache data segment and metadata data segments being added to a list of data segments modified within the associated transaction, and the DSM agents then assigning an exclusive permission on the newly allocated data segment.

55. The computer product program of claim 33, further comprising:

computer usable program code configured to provide, in a procedure for de-allocating an already allocated data segment, the procedure insuring that there is an active exclusive permission on the de-allocated disk data segment.

56. A system comprising:

a clustered file system (CFS) including a cluster of nodes forming a computer cluster, each node having a processor and a local cache memory coupled to the processor, and each node being in communication with a shared storage of file system data;

wherein the processor and the memory are configured to perform a method comprising;

providing a list of data segments modified within an ongoing transaction; and

providing a distributed shared memory (DSM) agent on each node;

wherein;

the DSM agents collectively manage access permissions to the entire space of file system data on a data segment basis;

the DSM agents utilize the distributed cache memories of the CFS as a virtual shared cache to provide transaction based modifications for user defined transactions on a data segment basis and to provide transaction based modifications for CFS defined transactions on the data segment basis;

each node comprises a local transaction journal for storing committed transactions generated by users on that node;

a rollback procedure scans the list to identify the location of the latest contents of a data segment prior to the transaction;

for each modified data segment, the rollback procedure identifies the location of the latest contents prior to the transaction by;

if the data segment was marked as modified in the cache at the time it was inserted into the list, then the latest contents of this data segment appears only in the journal;

otherwise, if the data segment was not marked as modified in the cache at the time it was inserted into the list, then the latest contents of this data segment appears in its final location in the shared storage;

the type of each data segment is recorded in the list during insertion of the data segment into the list;

and wherein;

all data segments in the list whose latest contents appear in their final location are discarded from the cache;

for all the other data segments in the list, their latest contents are restored from the journal into the cache by scanning the journal from its ending to its beginning and consider the first occurrences of these data segments in the journal, and then setting their modification indication to true; and

the procedure releases the exclusive permissions on all the data segments involved in the cancelled transaction.