Group communication system with flexible member model

• US 6,493,716 B1
• Filed: 01/19/2000
• Issued: 12/10/2002
• Est. Priority Date: 01/19/2000
• Status: Expired due to Term

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Goft, Gera, Factor, Michael, Azagury, Alain Charles, Pinter, Shlomit, Yeger-Lotem, Esther
• Primary Examiner(s): Mizrahi, Diane D.
• Application Number:
  US09/487,474
• Time in Patent Office: 1,056 Days
• Field of Search: 707/8, 707/10, 707/103.R, 707/104.1, 709/227, 709/466, 709/220, 709/204, 709/205, 370/352, 370/400, 370/401
• US Class Current: 1/1
• CPC Class Codes: G06F 2209/505 Clust
• CPC Class Codes: G06F 9/546 Message passing systems or structures, e.g. queues

Abstract

A group communication system (GCS), for use within a group of clustered computing nodes, provides a flexible group membership model. Flexible group members are defined generally as physical computing entities (such as computing nodes or other devices) or logical computing entities that take part in a distributed application. Such logical entities may include substantially any combination of processes, threads and callback functions. Logical entities that are members in the group may also be objects, such as files, that at least at some times during their existence have no active processes or threads. A single member may include multiple simultaneous processes.

Claims

1. A method for distributing messages among a group of member computing nodes, which are mutually-linked in a distributed computing system and take part in a distributed software application, the method comprising conveying a sequence of messages to all of the member nodes in the group in accordance with a communication protocol such that the messages are delivered in a uniform order to all of the member nodes, while on at least one of the member nodes there is no process involved in running the distributed software application for at least some time while the distributed software application is running on the group of member nodes.

2. A method according to claim 1, wherein the at least one of the member nodes that has no process holds a data structure used in the distributed software application.

3. A method according to claim 1, wherein the at least one of the member nodes that has no process comprises a device used in the distributed software application.

4. A method according to claim 1, wherein on another one of the member nodes there is a plurality of processes involved in running the distributed software application.

5. A method according to claim 1, wherein conveying the sequence of messages in accordance with the communication protocol comprises delivering the messages using a group communication system.

6. A method according to claim 1, and comprising periodically checking to determine whether any of the member nodes is unable to perform a part in the application, including the at least one of the member nodes that has no process.

7. A method according to claim 6, wherein periodically checking comprises defining a respective liveness function for each of the member nodes to determine whether the member nodes are functioning and able to participate in the application, such that different, respective liveness functions are defined for two or more of the member nodes.

8. A method according to claim 1, wherein conveying the sequence of messages comprises invoking callback functions for use by the member nodes in processing the messages.

9. A method according to claim 8, wherein conveying the sequence of messages comprises, at least for messages in the sequence of one or more specified types, waiting for the callback function invoked by a first message in the sequence to return before delivering a second, subsequent message, whereby the uniform order is maintained.

10. A method according to claim 1, and comprising:

assigning a unique, respective token to each of the member nodes;

receiving a message, in accordance with the protocol, sent by one of the member nodes, the message including the respective token; and

processing the message responsive to the token.

11. A method for distributing messages among a group of member computing nodes, which are mutually-linked in a distributed computing system and take part in a distributed software application, the method comprising conveying a sequence of messages to all of the member nodes in the group in accordance with a communication protocol such that the messages are delivered in a uniform order to all of the member nodes, while on at least one of the member nodes there is a plurality of processes involved in running the distributed software application.

12. A method according to claim 11, wherein conveying the sequence of messages in accordance with the communication protocol comprises delivering the messages using a group communication system.

13. A method according to claim 11, and comprising defining a respective liveness function for each of the member nodes to determine whether the member nodes are functioning and able to participate in the application, and periodically using the liveness function to determine whether any of the member nodes is unable to perform a part in the application, wherein the liveness function invokes a response by one of the plurality of processes on the at least one of the member nodes.

14. A method according to claim 11, wherein conveying the sequence of messages comprises invoking callback functions for use by the member nodes in processing the messages.

15. A method according to claim 14, wherein conveying the sequence of messages comprises, at least for messages in the sequence of one or more specified types, waiting for the callback function invoked by a first message in the sequence to return before delivering a second, subsequent message, whereby the uniform order is maintained.

16. A method according to claim 11, and comprising:

assigning a unique, respective token to each of the member nodes;

receiving one or more messages, in accordance with the protocol, sent by one of the member nodes, the messages including the respective token; and

processing the messages responsive to the token.

17. A method according to claim 16, wherein receiving the one or more messages comprises receiving respective first and second messages from first and second processes among the plurality of processes running on the at least one of the member nodes, both the first and second messages including the same token.

18. A method for transferring messages among a group of member computing nodes, which are mutually-linked in a distributed computing system and which communicate in accordance with a communication protocol such that a sequence of messages is delivered to all of the member nodes in the group in an order that is uniform among all of the member nodes, the method comprising:

assigning a unique, respective token to each of the member nodes;

receiving a message, in accordance with the protocol, sent by one of the member nodes, the message including the respective token assigned to the one of the member nodes sending the message; and

processing the message responsive to the token.

19. A method according to claim 18, wherein processing the message comprises identifying the one of the member nodes sending the message based on the token.

20. A method according to claim 18, wherein the one of the member nodes sending the message has a plurality of objects involved in running a distributed software application, and wherein receiving the message sent by the member entity comprises receiving respective messages from two or more of the plurality of objects, wherein the respective messages include the same token.

21. A method according to claim 18, wherein the member nodes take part in a distributed application running on the system.

22. A method according to claim 18, wherein receiving the message in accordance with the protocol comprises receiving the message using a group communication system.

23. A method according to claim 18, wherein receiving the message comprises invoking a callback function associated with the member node and receiving the message responsive to the callback function.

24. In a distributed computing system, in which a group of mutually-linked member computing nodes communicate in accordance with a communication protocol such that a sequence of messages is delivered to all of the member nodes in the group in an order that is uniform among all of the member nodes, a method for processing the messages received by the member computing nodes, comprising:

defining for each node among the member nodes at least one callback function to be invoked when messages of a predetermined type are handed to the node, such that different callback functions are defined for different nodes in the group of member nodes;

receiving a message of the predetermined type; and

invoking the callback function to process the message.

25. A method according to claim 24, wherein defining the at least one callback function comprises defining a plurality of different callback functions for a corresponding plurality of different message types.

26. A method according to claim 24, wherein receiving the message of the predetermined type comprises receiving the message from a group communication system.

27. A method according to claim 24, wherein the member nodes take part in a distributed application running on the system.

28. A method according to claim 27, wherein defining the at least one callback function comprises defining a liveness check function for each of the member nodes to determine whether the member nodes are functioning and able to participate in the application, and wherein receiving the message of the predetermined type comprises periodically receiving a liveness check message sent by one of the member nodes to ascertain whether the other member nodes are functioning and able to participate in the application, and wherein invoking the callback function comprises invoking the liveness check function responsive to the liveness check message.

29. A method according to claim 24, wherein invoking the callback function comprises aborting delivery of the message if the callback function does not return within a specified time.

30. A method according to claim 24, wherein invoking the callback function comprises waiting for the callback function invoked by a first message in the sequence to return before invoking a callback function to handle a second, subsequent message, whereby the uniform order is maintained.

31. A method according to claim 24, wherein invoking the callback function comprises acquiring a permission required for the callback.

32. A method according to claim 24, wherein defining the at least one callback function comprises defining a callback such that when messages of the predetermined type are received, they are passed to a designated process run by the node.

33. In a distributed computing system, in which a group of mutually-linked member computing nodes take part in a distributed computing application, a method for checking whether the member computing nodes are functioning and able to participate in the distributed computing application, the method comprising:

defining a respective liveness function for each member node that joins the group, to determine whether the member nodes are functioning and able to participate in the application, such that at least two of the member nodes have different liveness functions; and

periodically invoking the liveness functions of the member nodes so as to determine whether any of the nodes is unable to participate in the application.

34. A method according to claim 33, wherein invoking the liveness function comprises conveying a liveness check request from a group communication system.

35. A method according to claim 33, wherein defining the liveness function comprises defining a liveness function for at least one of the member nodes substantially without reference to execution of any process by the at least one of the member nodes.

36. A method according to claim 33, wherein defining the liveness function comprises, for at least one of the member nodes, defining a message that is sent to an object on the member node when the function is invoked.

37. A method according to claim 36, wherein periodically invoking the liveness functions comprises sending the message and determining whether the object responds within a predetermined period of time.

38. A method according to claim 33, wherein defining the liveness function comprises, for at least one of the member nodes, defining a function that checks the existence of a process with a specified identification.

39. A method according to claim 33, wherein defining the liveness function comprises, for at least one of the member nodes, defining a function that checks whether a queue held by the member node is full.

40. In a distributed computing system, in which a group of mutually-linked member computing nodes take part in a distributed computing application, a method for checking whether a given node among the member computing nodes is functioning and able to participate in the distributed computing application, the method comprising periodically invoking a liveness function to be carried out by the given node so as to determine whether the given node is unable to participate in the application, without reference to execution of any process by the given node.

41. A method according to claim 40, wherein invoking the liveness function comprises conveying a liveness check request from a group communication system.

42. A method according to claim 40, wherein invoking the liveness function comprises, for at least one of the member nodes, sending a message to an object on the member node and determining whether the object responds within a predetermined period of time.

43. A method according to claim 40, wherein invoking the liveness function comprises, for at least one of the member nodes, checking whether a queue held by the member node is full.

44. Distributed computing apparatus, comprising:

a computer network; and

a group of computer nodes, arranged to take part in a distributed software application and mutually-linked by the network in accordance with a communication protocol such that when a sequence of messages is transmitted in the group, the messages are delivered to all of the nodes in a uniform order, while on at least one of the nodes there is no process involved in running the distributed software application for at least some time while the distributed software application is running on the group of nodes.

45. Apparatus according to claim 44, wherein the at least one of the member nodes that has no process holds a data structure used in the distributed software application.

46. Apparatus according to claim 44, wherein the at least one of the member nodes that has no process comprises a device used in the distributed software application.

47. Apparatus according to claim 44, wherein the communication protocol comprises a group communication system.

48. Distributed computing apparatus, comprising:

a computer network; and

a group of computer nodes, arranged to take part in a distributed software application and mutually-linked by the network in accordance with a communication protocol such that when a sequence of messages is transmitted in the group, the messages are delivered to all of the nodes in a uniform order, while on at least one of the nodes there is a plurality of processes involved in running the distributed software application.

49. Distributed computing apparatus, comprising:

a computer network; and

a group of computer nodes, mutually-linked by the network using a group communication system such that when a sequence of messages is transmitted in the group, the messages are delivered to all of the nodes in a uniform order,

wherein a unique token is respectively assigned to each of the nodes, such that whenever a message is passed from any one of the nodes to the group communication system, the message includes the respective unique token.

50. Apparatus according to claim 49, wherein at least one of the member nodes has a plurality of objects involved in running a distributed software application, and wherein two or more of the plurality of objects are adapted to convey messages that include the same token to the group communication system.

51. Distributed computing apparatus, comprising:

a computer network; and

a group of computer nodes, mutually-linked by the network in accordance with a communication protocol such that when a sequence of messages is transmitted in the group, the messages are delivered to all of the nodes in a uniform order, and such that for each of the nodes, at least one callback function is defined, to be invoked when messages of a predetermined type are handed to the nodes, such that different callback functions are defined for different nodes in the group.

52. Apparatus according to claim 51, wherein a plurality of different callback functions are defined for a corresponding plurality of different message types.

53. Distributed computing apparatus, comprising:

a computer network; and

a group of computer nodes, mutually-linked by the network and configured to take part in a distributed computing application running on the apparatus,

wherein a respective liveness function is defined for each node that joins the group to determine whether the nodes are functioning and able to participate in the application, such that at least two of the member nodes having different liveness functions, which liveness functions are invocable so as to determine whether any of the nodes is unable to participate in the application.

54. Apparatus according to claim 53, wherein the liveness function is invoked by a group communication system.

55. Distributed computing apparatus, comprising:

a computer network; and

a group of computer nodes, mutually-linked by the network and configured to take part in a distributed computing application running on the apparatus,

wherein a respective liveness function is defined for at least one of the nodes for checking whether the at least one of the nodes is functioning and able to participate in the distributed computing application, and wherein the liveness function is invoked so as to determine whether the at least one of the nodes is unable to participate in the application, without reference to execution of any process by the at least one of the nodes.

56. A computer software product for distributing messages among a group of member computing nodes, which are mutually-linked in a distributed computing system and are arranged to take part in a distributed software application, the product comprising a computer-readable medium having program instructions stored therein, which instructions, when read by one of the member nodes, cause the one of the member nodes to convey a sequence of messages to all of the member nodes in the group in accordance with a communication protocol such that the messages are delivered in a uniform order to all of the member nodes, while on at least one of the member nodes there is no process involved in running the distributed software application for at least some time while the distributed software application is running on the group of member nodes.

57. A product according to claim 56, wherein the product comprises group communication system middleware.

58. A computer software product for distributing messages among a group of member computing nodes, which are mutually-linked in a distributed god computing system and are arranged to take part in a distributed software application, the product comprising a computer-readable medium having program instructions stored therein, which instructions, when read by one of the member nodes, cause the one of the member nodes to convey a sequence of messages to all of the member nodes in the group in accordance with a communication protocol such that the messages are delivered in a uniform order to all of the member nodes, while on at least one of the member nodes there is a plurality of processes involved in running the distributed software application.

59. A computer software product for transferring messages among a group of member computing nodes, which are mutually-linked in a distributed computing system and which communicate in accordance with a communication protocol such that a sequence of messages is delivered to all of the member nodes in the group in an order that is uniform among all of the member nodes, the product comprising a computer-readable medium having program instructions stored therein, which instructions, when read by the member nodes, cause the member nodes to assign respective unique tokens to the member nodes, such that messages including the respective tokens are sent by the member nodes in accordance with the protocol and are processed responsive to the tokens.

60. A computer software product for use in a distributed computing system in which a group of mutually-linked member computing nodes communicate in accordance with a communication protocol such that a sequence of messages is delivered to all of the member nodes in the group in an order that is uniform among all of the member nodes, the product comprising a computer-readable medium having program instructions stored therein, which instructions, when read by the member nodes, cause the member nodes to define for each of the member nodes at least one callback function, to be invoked when messages of a predetermined type are handed to the nodes, such that when a message of the predetermined type is received by one of the member nodes, the callback function is invoked to process the message, and such that different callback functions are defined for different nodes in the group.

61. A computer software product for use in a distributed computing system in which a group of mutually-linked member computing nodes take part in a distributed computing application, the product comprising a computer-readable medium having program instructions stored therein, which instructions, when read by the member nodes, cause the nodes to define a respective liveness function for each node that joins the group to determine whether the nodes are functioning and able to participate in the application, such that at least two of the member nodes have different liveness functions, and which further cause the member nodes to periodically invoke the liveness functions so as to determine whether any of the member nodes is unable to participate in the application.

62. A computer software product for use in a distributed computing system in which a group of mutually-linked member computing nodes take part in a distributed computing application, the product comprising a computer-readable medium having program instructions stored therein, which instructions, when read by the member nodes, cause the nodes to periodically invoke a liveness function to be carried out by a given one of the member nodes so as to determine whether the given one of the member nodes is functioning and able to participate in the distributed computing application,

wherein the liveness function is carried out by the given one of the member nodes without reference to execution of any process by the given one of the member nodes.