Method and system for detecting a dead server

• US 7,395,328 B2
• Filed: 12/27/2004
• Issued: 07/01/2008
• Est. Priority Date: 06/23/2000
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Mavrinac, Erik, Ronneburg, Frank
• Primary Examiner(s): Jacobs; Lashonda T
• Application Number:
  US11/023,200
• Publication Number:
  US 20050114352A1
• Time in Patent Office: 1,282 Days
• Field of Search: 709/223, 709/224, 709/225, 709/226, 709/203, 709/217
• US Class Current: 709/223
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0772 Means for error signaling, e.g. using interrupts, exception flags, dedicated error registers
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: H04L 41/0659 by isolating or reconfiguring faulty entities
• CPC Class Codes: H04L 43/0817 by checking functioning
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

Method and system for detecting a dead server in a multi-server environment. A virtual ring structure is used in which each server in a server pool is only required to monitor the status of two other servers in the server pool. Thus, a server need only transmit ping signals to two other servers (its buddies) in the server pool at any given time. Because each server maintains the status of only two other servers at any given time, the size of the server pool is not limited by the ability of each server to send and process ping signals. The two servers which are monitored by any given server in the server pool are referred to as the “buddy A” server and the “buddy B” server. When the monitoring server determines that one of its buddy servers is down, the monitoring server reports the status of the down server to a SQL server that maintains a server table. The server table maintains a list of each “live” server and the buddy servers assigned to that server. Down servers are removed from the server table. When a server determines that one of its buddies is down, the report to the SQL server results in a buddy reassignment. The buddies of the down server are made buddies of one another and the virtual server ring is once more intact. The SQL server then knows not to route any client to the down server.

Claims

1. A system for removing an inoperative computer from a pool of computers, comprising:

a first computer associated with a first monitored computer and a second monitored computer, the first computer being operative to monitor the status of the first monitored computer and the second monitored computer;

wherein being operative to monitor the status of the first monitored computer and the second monitored computer comprises being operative to initiate transmission of a first signal to the first monitored computer and to initiate transmission of a second signal to the second monitored computer and to await receipt of a first responsive signal from the first monitored computer and to await receipt of a second responsive signal from the second monitored computer;

a computer list operative to maintain the association between the first computer and the first monitored computer and to maintain the association between the first computer and the second monitored computer;

wherein the first computer is further operative to send a first monitored computer inoperative signal to the computer list, in response to a determination that the first monitored computer is inoperative; and

wherein the computer list is further operative to disassociate the first computer from the first monitored computer and to associate the first computer with a third monitored computer in response to the receipt of the first monitored computer inoperative signal.

2. The system recited in claim 1, wherein the first computer is further operative to send a second monitored computer inoperative signal to the computer list, in response to a determination that the second monitored computer is inoperative and wherein the computer list is further operative to disassociate the first computer from the second monitored computer and to associate the first computer with a fourth monitored computer in response to the receipt of the second monitored computer inoperative signal.

3. The system recited in claim 1, wherein the first signal comprises an ICMP ping signal and wherein the second signal comprises an ICMP ping signal.

4. The system recited in claim 1, wherein the computer list maintains a list of active computers comprising the first computer, the first monitored computer and the second monitored computer, and wherein, in response to the first monitored computer inoperative signal, the computer list is further operative to remove the first monitored computer from the list of active computers.

5. The system recited in claim 4, further comprising a client connected to the computer list and wherein the list of active computers is used to connect the client to one of the computers in the list of active computers.

6. A computer-implemented method for creating a virtual computer ring, the method comprising:

storing an entry in a computer table, wherein the computer table identifies a plurality of computers in a computer pool;

wherein each entry comprises a computer identification of a computer in the computer pool, a first monitored computer, and a second monitored computer and;

causing at least one of the plurality of computers to monitor its first monitored computer and its second monitored computer to determine whether one of its monitored computers is inoperative;

wherein monitoring the status of the first monitored computer and the second monitored computer comprises being operative to initiate transmission of a first signal to the first monitored computer and to initiate transmission of a second signal to the second monitored computer and to await receipt of a first responsive signal from the first monitored computer and to await receipt of a second responsive signal from the second monitored computer;

a computer list operative to maintain the association between the one of the plurality of computers and the first monitored computer and to maintain the association between the one of the plurality of computers and the second monitored computer;

wherein the one of the plurality of computers is further operative to send a first monitored computer inoperative signal to the computer list, in response to a determination that the first monitored computer is inoperative; and

wherein the computer list is further operative to disassociate the one of the plurality of computers from the first monitored computer and to associate the one of the plurality of computers with a third monitored computer in response to the receipt of the first monitored computer inoperative signal.

7. The method of claim 6, further comprising:

adding a new computer to the virtual computer ring by randomly choosing one of the plurality of computers and the randomly chosen computer'"'"'s first monitored computer;

reassigning the new computer as the randomly chosen computer'"'"'s first monitored computer; and

reassigning the new computer as either the first monitored computer or second monitored computer of the randomly chosen computer'"'"'s first monitored computer.

8. The method of claim 6, further comprising:

when the monitoring computer determines that one of its monitored computers is inoperative, reporting the identity of the inoperative computer to the computer table.

9. The method of claim 8, further comprising:

causing a routing computer, wherein the routing computer is responsible for routing a client to the virtual computer ring, to no longer route the client to the inoperative computer.

10. The method recited in claim 8, further comprising:

reassigning the inoperative computer'"'"'s monitored computers to be associated with one another.

11. The method of claim 9, wherein causing the routing computer to no longer route the client to the inoperative computer comprises removing the inoperative computer from the computer table.

12. The method of claim 6, further comprising storing a monitored computer table at each of the plurality of computers, wherein the monitored computer table comprises the computer'"'"'s first monitored computer and the computer'"'"'s second monitored computer.

13. The method of claim 6, wherein sending a first signal by at least one of a plurality of computers to its first monitored computer and sending a second signal to its second monitored computer to determine whether one of its monitored computers is inoperative comprises sending an ICMP ping signal to the first monitored computer and to the second monitored computer.

14. The method of claim 6, further comprising:

if one of the plurality of computers in the computer pool shuts down normally, then causing the normal shutdown computer to report its identity to the computer table and removing the normally shutdown computer from the computer table.

15. The method of claim 14, further comprising:

reassigning the normally shutdown computer'"'"'s first monitored computer and second monitored computer to monitor one another such that the first monitored computer monitors the second monitored computer rather than monitoring the normally shutdown computer and the second monitored computer monitors the first monitored computer rather than monitoring the normally shutdown computer.

16. A computer-implemented method for monitoring the status of a plurality of computers in a pool of computers, the method comprising:

assigning each of the plurality of computers a first monitored computer and a second monitored computer within the computer pool;

causing each of the plurality of computers to monitor the status of its first monitored computer and its second monitored computer by initiating transmission of a first signal to the first monitored computer and initiating transmission of a second signal to the second monitored computer and waiting to receive a first responsive signal from the first monitored computer and waiting to receive a second responsive signal from the second monitored computer;

if one of the plurality of computers determines that one of its monitored computers is inoperative, then causing a monitoring computer to notify a central computer list that stores associations between computers that one of its monitored computers is inoperative;

a computer list operative to maintain the association between the one of the plurality of computers and the first monitored computer and to maintain the association between the one of the plurality of computers and the second monitored computer;

wherein the one of the plurality of computers is further operative to send a first monitored computer inoperative signal to the computer list, in response to a determination that the first monitored computer is inoperative; and

wherein the computer list is further operative to disassociate the one of the plurality of computers from the first monitored computer and to associate the one of the plurality of computers with a third monitored computer in response to the receipt of the first monitored computer inoperative signal.

17. The method of claim 16, further comprising:

removing the inoperative monitored computer from the central computer list when notification is received that the monitored computer is inoperative.

18. The method of claim 17, further comprising reassigning the inoperative computer'"'"'s other monitored computer to be associated with the monitoring computer.

19. A computer-readable medium having stored thereon computer-executable instructions which, when executed by a computer, cause the computer to:

assign each of the plurality of computers a first monitored computer and a second monitored computer within the computer pool;

cause each of the plurality of computers to monitor the status of its first monitored computer and its second monitored computer by initiating transmission of a first signal to the first monitored computer and initiating transmission of a second signal to the second monitored computer and waiting to receive a first responsive signal from the first monitored computer and waiting to receive a second responsive signal from the second monitored computer;

if one of the plurality of computers determines that one of its monitored computers is inoperative, then cause a monitoring computer to notify a central computer list that stores associations between computers that one of its monitored computers is inoperative;

cause the central computer list to remove the inoperative monitored computer from the central computer list and reassign the inoperative computer'"'"'s other monitored computer to be associated with the monitoring computer when notification is received that the monitored computer is inoperative;

cause the central computer list to connect a client connected to the central computer list to one of the plurality of operative computers in the computer pool;

if one of the plurality of computers shuts down normally, then cause the computer to report its identity to the central computer table and remove said computer from the computer table and cause the normally shutdown computer'"'"'s first monitored computer and second monitored computer to monitor one another; and

;

If a new computer is added to the plurality of computers in the computer pool, cause the central computer list to randomly choose one of the plurality of computers and the randomly chosen computer'"'"'s first monitored computer, reassign the new computer as the randomly chosen computer'"'"'s first monitored computer, and reassign the new computer as either the first monitored computer or second monitored computer of the randomly chosen computer'"'"'s first monitored computer.