Selective message loss handling in a cluster of replicated servers

• US 8,738,959 B2
• Filed: 11/15/2011
• Issued: 05/27/2014
• Est. Priority Date: 11/15/2011
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Harpaz, Avraham, Naaman, Nir, Zach, Idan
• Primary Examiner(s): Wilson, Yolanda L
• Application Number:
  US13/296,250
• Publication Number:
  US 20130124909A1
• Time in Patent Office: 924 Days
• Field of Search: 714/4.1, 714/15, 714/13
• US Class Current: 714/4.1
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership
• CPC Class Codes: G06F 11/1443 Transmit or communication errors
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: H04L 67/1034 Reaction to server failures by a load balancer

Abstract

A computer-implemented method, a computerized system and a product for providing a cluster of replicated servers. The method performed by a computerized server in a cluster of servers, wherein the cluster of servers are executing replicated instances of an application, wherein the replicated instances are configured to perform the same processing of the same input, comprising: detecting a message loss in the server; electively determining a responsive action to the message loss; and notifying the cluster of servers of the responsive action determined by the server, whereby other servers of the cluster of servers are able to mimic operation of the server by simulating the responsive action.

Claims

1. A computer-implemented method performed by a computerized server in a cluster of servers, wherein the cluster of servers are executing replicated instances of an application, wherein the replicated instances are configured to perform the same processing of the same input, comprising:

detecting a message loss in the server;

selectively determining a responsive action to the message loss; and

notifying the cluster of servers of the responsive action determined by the server, whereby other servers of the cluster of servers are able to mimic operation of the server by simulating the responsive action.

2. The computer-implemented method of claim 1, wherein the message loss comprises failure to receive one or more messages by the server;

and wherein at least one of the other servers of the cluster of servers received the one or more messages.

3. The computer-implemented method of claim 1, further comprising selectively or non-selectively attempting to recover one or more lost messages;

wherein non-selective recovery is performed in response to said detecting; and

wherein selective recovery is performed based on a responsive action.

4. The computer-implemented method of claim 1, wherein the responsive action comprises dropping one or more lost messages from being processed by the application, whereby in response to said notifying, the cluster of servers are able to avoid processing the one or more lost messages.

5. The computer-implemented method of claim 4, wherein said selectively determining the responsive action further comprises determining a message type to be ignored, wherein in response to receiving new messages of the message type, the server does not provide the new messages to application for processing.

6. The computer-implemented method of claim 1, wherein:

said selectively determining comprises selecting whether or not to stop the application being executed by the server;

in response to a determination not to stop the application, determining the responsive action to the message loss.

7. The computer-implemented method of claim 1, wherein said determining the responsive action is performed by the application.

8. The computer-implemented method of claim 7, wherein said determining the responsive action comprises:

providing the application with a state of the cluster; and

wherein the application determining the responsive action based on the state of the cluster.

9. The computer-implemented method of claim 8, wherein the state of the cluster comprises a number of servers that are executing the replicated application.

10. The computer-implemented method of claim 7, wherein said determining the responsive action comprises:

providing the application with information selected from the group consisting of;

number of lost messages;

estimated age of lost messages;

content of next available one or more messages; and

transport information relating to one or more lost messages.

11. The computer-implemented method of claim 7, wherein said determining the responsive action comprises:

providing the application with an indication relating to priority of one or more lost messages, wherein the indication is determined based on two or more messages, wherein the two or more messages comprise at least one message preceding the one or more lost messages and at least one message succeeding the one or more lost messages.

12. The computer-implemented method of claim 11, further comprising determining the indication relating to priority by:

extracting from a succeeding message a first ordinal distance since last message having a priority;

extracting from a preceding message a second ordinal distance since last message having the priority; and

determining whether one or more of the lost messages having the priority based on the difference between the first ordinal distance and the second ordinal distance.

13. The computer-implemented method of claim 1, wherein said detecting, said selectively determining, and said notifying are performed a plurality of times, wherein at least two different responsive actions are determined with respect to different message losses.

14. The computer-implemented method of claim 13,

wherein in response to detecting message loss of a first message having a first priority, determining to drop the first message;

wherein in response to detecting message loss of a second message having a second priority, determining not to drop the second message; and

wherein the first priority is lower than the second priority.

15. The computer-implemented method of claim 1, wherein the notification to the cluster is either an implicit or explicit notification.

16. The computer-implemented method of claim 1, wherein said selectively determining is based on metadata information comprised by messages that were received, wherein the metadata information of a message is useful for determining a property associated with one or more of lost messages.

17. A computerized system comprising:

a cluster of computerized servers;

wherein each server of the cluster comprises a processor, a memory unit and a communication module for communicating over a computerized network;

wherein each server of the cluster is adapted to execute a replicated instance of an application, wherein the servers are operative to receive the same input and perform the same processing, wherein the servers are substantial replicas of one another;

wherein a server of the cluster comprising a processor operative to;

detect a message loss in the server;

selectively determine a responsive action to the message loss; and

notify the cluster of servers of the responsive action determined by the server, whereby other servers of the cluster of servers are able to mimic operation of the server by simulating the responsive action.

18. The computerized system of claim 17, wherein message loss is detected upon inability of the server to receive a message, notwithstanding other servers receiving or failing to receive the message.

19. The computerized system of claim 17, wherein the responsive action comprises dropping one or more lost messages from being processed by the application, whereby in response to the notification, the cluster of servers are able to avoid processing the one or more lost messages.

20. The computerized system of claim 17, wherein the processor is further configured to execute the application, and wherein the application is operative to perform the selectively determine the responsive action to the message loss.

21. The computerized system of claim 20, wherein the processor is further operative to determine a state of the cluster;

and wherein the application is operative to determine the responsive action based on the state of the cluster.

22. The computerized system of claim 20, wherein the processor is further operative to determine an indication of priority of one or more lost messages, wherein the indication is determined based on two or more messages, wherein the two or more messages comprise at least one message preceding the one or more lost messages and at least one message succeeding the one or more lost messages.

23. The computerized system of claim 20, wherein the application is operative to determine to drop a message having a relatively low priority and not to drop a message having a relatively high priority.

24. The computerized system of claim 17, wherein the cluster is configured to receive messages from a plurality of sources, wherein the plurality of sources comprise sources having different reliability, wherein the server is adapted to perform different message recovery procedures with respect to message loss from sources having different reliability.

25. A computer program product comprising:

a non-transitory computer readable medium having stored thereon a set of instructions to be performed by a computerized server in a cluster of servers, wherein the cluster of servers are executing replicated instances of an application, wherein the replicated instances are configured to perform the same processing of the same input;

wherein said set of instructions comprises;

a first program instruction for detecting a message loss in the server;

a second program instruction for selectively determining a responsive action to the message loss; and

a third program for notifying the cluster of servers of responsive action determined by the server, whereby other servers of the cluster of servers are able to mimic operation of the server by simulating the responsive action.