Data management application programming interface failure recovery in a parallel file system

• US 7,024,582 B2
• Filed: 06/25/2001
• Issued: 04/04/2006
• Est. Priority Date: 06/26/2000
• Status: Expired due to Fees

• Current Assignee: Daedalus Blue LLC
• Original Assignee: International Business Machines Corporation
• Inventors: Schmuck, Frank, Loy, Irit, Marberg, John, Haskin, Roger, Wyllie, James, Yehudai, Zvi, Shumeli, Boaz
• Primary Examiner(s): Beausoliel, Robert
• Assistant Examiner(s): GUYTON, PHILIP A
• Application Number:
  US09/887,533
• Publication Number:
  US 20020124013A1
• Time in Patent Office: 1,744 Days
• Field of Search: 714/4, 714/9, 714/10, 714/11, 714/42, 714/43
• US Class Current: 714/4.4
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/1858 Parallel file systems, i.e. file systems supporting multiple processors
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: Y10S 707/99931 Database or file accessing

Abstract

In a cluster of computing nodes having shared access to one or more volumes of data storage using a parallel file system, a method for managing the data storage includes initiating a session of a data management application on a session node selected from among the nodes in the cluster. The session node receives an event message in a session queue for processing by the data management application, responsive to a request submitted to the parallel file system by a source node among the nodes in the cluster to perform a file operation on a file in the data storage. Following a failure at the session node, the session queue is reconstructed so that processing of the event message by the data management application can continue after recovery from the failure, and the request can be fulfilled at the source node.

Claims

1. In a cluster of computing nodes having shared access to one or more volumes of data storage using a parallel file system, a method for managing the data storage, comprising:

initiating a session of a data management application on a session node selected from among the nodes in the cluster;

receiving an event message in a session queue for processing by the data management application at the session node, responsive to a request submitted to the parallel file system by a user application on a source node among the nodes in the cluster to perform a file operation on a file in the data storage; and

following a failure at the session node, reconstructing the session queue so that processing of the event message by the data management application can continue after recovery from the failure,

wherein reconstructing the session queue comprises selecting a new session node from among the nodes in the cluster, and assuming the data management session on the new session node, whereupon the session queue is reconstructed on the new session node, and

wherein assuming the data management session comprises assuming the session on the same session node that was used before the failure.

2. A method according to claim 1, wherein the failure at the session node comprises a file system failure at the session node.

3. A method according to claim 2, wherein reconstructing the session queue comprises selecting a new session node from among the nodes on which the file system failure has not occurred, and moving the data management session to the new session node.

4. A method according to claim 1, wherein the failure comprises a file system failure at the session node, which is followed by file system recovery, and wherein assuming the session comprises invoking any function call of a data management application programming interface (DMAPI) at the session node after the recovery, whereby reconstruction of the session queue is triggered implicitly.

5. A method according to claim 1, wherein assuming the session comprises explicitly invoking a session creation function call of a data management application programming interface (DMAPI).

6. A method according to claim 1, wherein the failure comprises a file system failure at the session node, which is followed by file system recovery, and wherein assuming the session comprises invoking any function call of a data management application programming interface (DMAPI) at the session node after the recovery, whereby reconstruction of the session queue is triggered implicitly.

7. A method according to claim 1, and comprising storing information regarding the session and events before the failure at one or more additional nodes among the nodes in the cluster, wherein reconstructing the session queue comprises using the information stored at the one or more additional nodes to reconstruct the queue.

8. A method according to claim 1, and comprising selecting one of the nodes to serve as a session manager node, and assuming the session by sending a message to the session manager node, causing the session manager node to distribute information regarding the session among the nodes in the cluster so that the data management application can continue after the recovery.

9. A method according to claim 1, wherein initiating the session comprises initiating the session in accordance with a data management application programming interface (DMAPI) of the parallel file system, and wherein processing the event message comprises processing the request using the DMAPI.

10. A method according to claim 1, and comprising:

sending a response to the event message from the data management application on the session node to the source node following the recovery from the failure; and

performing the file operation requested by the source node subject to the response from the data management application.

11. A method according to claim 10, wherein receiving the event message comprises receiving the message responsive to submission of the request by a file operation thread of a user application running on the source node, and blocking the thread until the response is received from the session node after the recovery from the failure.

12. A method according to claim 11, wherein reconstructing the session queue comprises sending a message from the session node to all of the nodes, so as to prompt the file operation thread on the source node to submit a new event message to the session node, whereby the event is placed in the reconstructed queue responsive to the new message.

13. A method according to claim 1, and comprising processing the event message in the reconstructed queue, and responsive to the event message, reacquiring a data management access right needed to handle the request.

14. A method according to claim 1, wherein receiving the event message comprises receiving multiple event messages from multiple source nodes in the cluster, and wherein reconstructing the session queue comprises collecting information regarding the session and events from the multiple source nodes.

15. A method according to claim 1, wherein initiating the session of the data management application comprises initiating a data migration application, so as to free storage space on at least one of the volumes of data storage.

16. A method according to claim 1, and comprising, following the failure, when the source node has not received a response to the event message within a predetermined lapse of time, failing the request submitted at the source node to the parallel file system.

17. In a cluster of computing nodes having shared access to one or more volumes of data storage using a parallel file system, a method for managing the data storage, comprising:

initiating a session of a data management application on a session node selected from among the nodes in the cluster;

receiving an event message in a session queue for processing by the data management application at the session node, responsive to a request submitted to the parallel file system by a user application on a source node among the nodes in the cluster to perform a file operation on a file in the data storage;

following a failure at the session node, reconstructing the session queue so that processing of the event message by the data management application can continue after recovery from the failure;

sending a response to the event message from the data management application on the session node to the source node following the recovery from the failure; and

performing the file operation requested by the source node subject to the response from the data management application,

wherein receiving the event message comprises receiving the message responsive to submission of the request by a file operation thread of a user application running on the source node, and blocking the thread until the response is received from the session node after the recovery from the failure, and

wherein reconstructing the session queue comprises sending a message from the session node to all of the nodes, so as to prompt the file operation thread on the source node to submit a new event message to the session node, whereby the event is placed in the reconstructed queue responsive to the new message, and

wherein prompting the file operation thread comprises instructing the file operation thread to submit the new event message with respect to an event that is defined as a synchronous event.

18. A method according to claim 17, wherein an event that is defined as an asynchronous event that was in the session queue prior to the failure is not placed in the reconstructed queue.

19. A method according to claim 17, wherein receiving the event message comprises receiving an event identifier, which is assigned to the event at the source node, and wherein the event placed in the reconstructed queue has the same event identifier as was assigned before the failure.

20. Computing apparatus, comprising:

one or more volumes of data storage, arranged to store data; and

a plurality of computing nodes, linked to access the volumes of data storage using a parallel file system, and arranged so as to enable a data management application to initiate a data management session on a session node selected among the nodes in the cluster, so that when a request is submitted to the parallel file system by a user application on a source node among the nodes in the cluster to perform a file operation on a file in the data storage, an event message is received at the session node responsive to the request, for processing by the data management application, and so that following a failure at the session node, the session queue is reconstructed so that processing of the event message by the data management application can continue after recovery from the failure,

wherein the nodes are arranged so that following the failure, a new session node is selected from among the nodes on which the failure has not occurred, and the data management session is assumed on the new session node, whereupon the session queue is reconstructed on the new session node, and

wherein the session is assumed on the same session node that was used before the failure.

21. Apparatus according to claim 20, wherein the failure at the session node comprises a file system failure at the session node.

22. Apparatus according to claim 21, wherein the nodes are arranged so that following the file system failure, a new session node is selected from among the nodes on which the file system failure has not occurred, and the data management session is moved to the new session node.

23. Apparatus according to claim 20, wherein the session is assumed on a different node from the session node used before the failure.

24. Apparatus according to claim 20, wherein the session is assumed by explicitly invoking a session creation function call of a data management application programming interface (DMAPI).

25. Apparatus according to claim 20, wherein the failure comprises a file system failure at the session node, which is followed by file system recovery, and wherein the session is assumed by invoking any function call of a data management application programming interface (DMAPI) at the session node after the recovery, whereby reconstruction of the session queue is triggered implicitly.

26. Apparatus according to claim 20, wherein the nodes are arranged so that information regarding the session and events is stored before the failure at one or more additional nodes among the nodes in the cluster, whereby the session queue is reconstructed using the information stored at the one or more additional nodes.

27. Apparatus according to claim 20, wherein one of the nodes is selected to serve as a session manager node, and wherein to assume the session, a message is sent to the session manager node, causing the session manager node to distribute information regarding the session among the nodes in the cluster so that the data management application can continue after the recovery.

28. Apparatus according to claim 20, wherein the session is initiated in accordance with a data management application programming interface (DMAPI) of the parallel file system, and wherein the event message is processed using the DMAPI.

29. Apparatus according to claim 20, wherein the nodes are arranged so that a response to the event message is sent from the data management application on the session node to the source node following the recovery from the failure, whereupon the file operation requested by the source node is carried out subject to the response from the data management application.

30. Apparatus according to claim 29, wherein the event message is received responsive to submission of the request by a file operation thread of a user application running on the source node, and the thread is blocked until the response is received from the session node after the recovery from the failure.

31. Apparatus according to claim 30, wherein to reconstruct the session queue, a message is sent from the session node to all of the nodes, so that the file operation thread on the source node is prompted to submit a new event message to the session node, whereby the event is placed in the reconstructed queue responsive to the new message.

32. Apparatus according to claim 20, wherein after reconstructing the session queue, the data management application reacquires a data management access right needed to handle the request.

33. Apparatus according to claim 20, wherein the nodes are arranged so that the session node receives multiple event messages from multiple source nodes in the cluster, and so that in reconstructing the session queue, information regarding the session and events is collected from the multiple source nodes.

34. Apparatus according to claim 20, wherein the data management application comprises a data migration application, for freeing storage space on at least one of the volumes of data storage.

35. Apparatus according to claim 20, wherein following the failure, when the source node has not received a response to the event message within a predetermined lapse of time, the request submitted at the source node to the parallel file system is failed.

36. Computing apparatus, comprising:

one or more volumes of data storage, arranged to store data; and

a plurality of computing nodes, linked to access the volumes of data storage using a parallel file system, and arranged so as to enable a data management application to initiate a data management session on a session node selected among the nodes in the cluster, so that when a request is submitted to the parallel file system by a user application on a source node among the nodes in the cluster to perform a file operation on a file in the data storage, an event message is received at the session node responsive to the request, for processing by the data management application, and so that following a failure at the session node, the session queue is reconstructed so that processing of the event message by the data management application can continue after recovery from the failure,

wherein the nodes are arranged so that a response to the event message is sent from the data management application on the session node to the source node following the recovery from the failure, whereupon the file operation requested by the source node is carried out subject to the response from the data management application, and

wherein the event message is received responsive to submission of the request by a file operation thread of a user application running on the source node, and the thread is blocked until the response is received from the session node after the recovery from the failure, and

wherein to reconstruct the session queue, a message is sent from the session node to all of the nodes, so that the file operation thread on the source node is prompted to submit a new event message to the session node, whereby the event is placed in the reconstructed queue responsive to the new message, and

wherein the file operation thread is prompted to submit the new event message with respect to an event that is defined as a synchronous event.

37. Apparatus according to claim 36, wherein an event that is defined as an asynchronous event that was in the session queue prior to the failure is not placed in the reconstructed queue.

38. Apparatus according to claim 36, wherein the event message contains an event identifier, which is assigned to the event at the source node, and wherein the event placed in the reconstructed queue has the same event identifier as was assigned before the failure.

39. A computer software product for use in a cluster of computing nodes having shared access to one or more volumes of data storage using a parallel file system, the product comprising a computer-readable medium in which program instructions are stored, which instructions, when read by the computing nodes, cause a session of a data management application to be initiated on a session node selected among the nodes in the cluster, such that when a user application on a source node among the nodes in the cluster submits a request to the parallel file system to perform a file operation on a file in the data storage, an event message is received at the session node, for processing by the data management application, and such that following a failure at the session node, the session queue is reconstructed so that processing of the event message by the data management application can continue after recovery from the failure,

wherein following the failure, the instructions cause a new session node to be selected from among the nodes on which the failure has not occurred, whereupon the data management session is assumed on the new session node, and the session queue is reconstructed on the new session node, and

wherein the session is assumed on the same session node that was used before the failure.

40. A product according to claim 39, wherein the failure at the session node comprises a file system failure at the session node.

41. A product according to claim 40, wherein following the file system failure, the instructions cause a new session node to be selected from among the nodes on which the file system failure has not occurred, whereupon the data management session is moved to the new session node.

42. A product according to claim 39, wherein the session is assumed on a different node from the session node used before the failure.

43. A product according to claim 39, wherein the session is assumed by explicitly invoking a session creation function call of a data management application programming interface (DMAPI).

44. A product according to claim 39, wherein the failure comprises a file system failure at the session node, which is followed by file system recovery, and wherein the session is assumed by invoking any function call of a data management application programming interface (DMAPI) at the session node after the recovery, whereby reconstruction of the session queue is triggered implicitly.

45. A product according to claim 39, wherein the instructions cause information regarding the session and events to be stored before the failure at one or more additional nodes among the nodes in the cluster, whereby the session queue is reconstructed using the information stored at the one or more additional nodes.

46. A product according to claim 39, wherein one of the nodes is selected to serve as a session manager node, and wherein to assume the session, a message is sent to the session manager node, causing the session manager node to distribute information regarding the session among the nodes in the cluster so that the data management application can continue after the recovery.

47. A product according to claim 39, wherein the product comprises a data management application programming interface (DMAPI) of the parallel file system, and wherein the event message is processed using the DMAPI.

48. A product according to claim 39, wherein the instructions cause a response to the event message to be sent from the data management application on the session node to the source node following the recovery from the failure, whereupon the file operation requested by the source node is carried out subject to the response from the data management application.

49. A product according to claim 48, wherein the event message is received responsive to submission of the request by a file operation thread of a user application running on the source node, and the thread is blocked until the response is received from the session node after the recovery from the failure.

50. A product according to claim 49, wherein to reconstruct the session queue, a message is sent from the session node to all of the nodes, so that the file operation thread on the source node is prompted to submit a new event message to the session node, whereby the event is placed in the reconstructed queue responsive to the new message.

51. A product according to claim 39, wherein after reconstructing the session queue, the data management application reacquires a data management access right needed to handle the request.

52. A product according to claim 39, wherein the instructions cause the session node to receive multiple event messages from multiple source nodes in the cluster, and to reconstruct the session queue by collecting information regarding the session and events from the multiple source nodes.

53. A product according to claim 39, wherein the data management application comprises a data migration application, for freeing storage space on at least one of the volumes of data storage.

54. A product according to claim 39, wherein following the failure, when the source node has not received a response to the event message within a predetermined lapse of time, the request submitted at the source node to the parallel file system is failed.

55. A computer software product for use in a cluster of computing nodes having shared access to one or more volumes of data storage using a parallel file system, the product comprising a computer-readable medium in which program instructions are stored, which instructions, when read by the computing nodes, cause a session of a data management application to be initiated on a session node selected among the nodes in the cluster, such that when a user application on a source node among the nodes in the cluster submits a request to the parallel file system to perform a file operation on a file in the data storage, an event message is received at the session node, for processing by the data management application, and such that following a failure at the session node, the session queue is reconstructed so that processing of the event message by the data management application can continue after recovery from the failure,

wherein the instructions cause a response to the event message to be sent from the data management application on the session node to the source node following the recovery from the failure, whereupon the file operation requested by the source node is carried out subject to the response from the data management application, and

wherein the event message is received responsive to submission of the request by a file operation thread of a user application running on the source node, and the thread is blocked until the response is received from the session node after the recovery from the failure, and

wherein to reconstruct the session queue, a message is sent from the session node to all of the nodes, so that the file operation thread on the source node is prompted to submit a new event message to the session node, whereby the event is placed in the reconstructed queue responsive to the new message, and

wherein the file operation thread is prompted to submit the new event message with respect to an event that is defined as a synchronous event.

56. A product according to claim 55, wherein an event that is defined as an asynchronous event that was in the session queue prior to the failure is not placed in the reconstructed queue.

57. A product according to claim 55, wherein the event message contains an event identifier, which is assigned to the event at the source node, and wherein the event placed in the reconstructed queue has the same event identifier as was assigned before the failure.