Fast cluster failure detection

• US 8,266,474 B2
• Filed: 03/04/2010
• Issued: 09/11/2012
• Est. Priority Date: 12/30/2009
• Status: Active Grant

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Goel, Garima
• Primary Examiner(s): Baderman, Scott
• Assistant Examiner(s): Mehrmanesh, Elmira
• Application Number:
  US12/717,909
• Publication Number:
  US 20110219263A1
• Time in Patent Office: 922 Days
• Field of Search: 714/4.2, 714/43, 714/4.1, 714/4.3
• US Class Current: 714/4.2
• CPC Class Codes: G06F 11/07 Responding to the occurrence of a fault, e.g. fault tolerance
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: H04L 43/0811 by checking connectivity

Abstract

A method and system for fast failure detection in a distributed computer system. The method includes executing a distributed computer system having a plurality of clusters comprising at least a first cluster, a second cluster and the third cluster, and initializing failure detection by creating a connected cluster list in each of the plurality of clusters, wherein for each one of the plurality of clusters, a respective connected cluster list describes others of the plurality of clusters said each one is communicatively connected with. A status update message is sent upon changes in connectivity between the plurality of clusters, and generating an updated connected cluster list in each of the plurality of clusters in accordance with the status update message. The method then determines whether the change in connectivity results from a cluster failure by examining the updated connected cluster list in each of the plurality of clusters.

Claims

1. A method for fast failure detection in a distributed computer system, comprising:

executing a distributed computer system having a plurality of clusters comprising at least a first cluster, a second cluster and the third cluster;

initializing failure detection by creating a connected cluster list in each of the plurality of clusters, wherein for each one of the plurality of clusters, a respective connected cluster list describes others of the plurality of clusters said each one is communicatively connected with;

sending a status update message upon a change in connectivity between the plurality of clusters;

generating an updated connected cluster list in each of the plurality of clusters in accordance with the status update message; and

determining whether the change in connectivity is a result of a cluster failure by examining the updated connected cluster list in each of the plurality of clusters;

wherein upon receiving a loss of communication status update message from the second cluster, the third cluster removes the first cluster from a connected cluster list of the second cluster, and wherein the third cluster checks a connected cluster list of the third cluster to determine whether the third cluster is connected to another cluster to which the first cluster is also connected.

2. The method of claim 1, further comprising determining whether the change in connectivity is a result of a communications link failure by examining the updated connected cluster list in each of the plurality of clusters.

3. The method of claim 1, wherein the status update message comprises a name of a cluster with which a connection was established or broken.

4. The method of claim 1, wherein each of the plurality of clusters maintains a respective connected cluster list created by others of the plurality of clusters.

5. The method of claim 1, wherein at the time of establishing a communication link, each of the plurality of clusters updates and exchanges their respective connected cluster list with others of the plurality of clusters.

6. The method of claim 1, wherein the third cluster declares a failure of the first cluster and updates its respective connected cluster list accordingly, when the third cluster is not connected to another cluster to which the first cluster is also connected.

7. The method of claim 1, wherein the third cluster recognizes the health of the first cluster and updates its respective connected cluster list accordingly, when the third cluster is connected to another cluster to which the first cluster is also connected.

8. A computer readable storage medium having stored thereon, computer executable instructions that, if executed by a computer system cause the computer system to perform a method comprising:

executing a distributed computer system having a plurality of clusters comprising at least a first cluster, a second cluster and the third cluster;

initializing failure detection by creating a connected cluster list in each of the plurality of clusters, wherein for each one of the plurality of clusters, a respective connected cluster list describes others of the plurality of clusters said each one is communicatively connected with;

sending a status update message upon a change in connectivity between the plurality of clusters;

generating an updated connected cluster list in each of the plurality of clusters in accordance with the status update message; and

determining whether the change in connectivity is a result of a cluster failure by examining the updated connected cluster list in each of the plurality of clusters;

wherein upon receiving a loss of communication status update message from the second cluster, the third cluster removes the first cluster from a connected cluster list of the second cluster, and wherein the third cluster checks a connected cluster list of the third cluster to determine whether the third cluster is connected to another cluster to which the first cluster is also connected.

9. The computer readable storage medium of claim 8, further comprising determining whether the change in connectivity is a result of a communications link failure by examining the updated connected cluster list in each of the plurality of clusters.

10. The computer readable storage medium of claim 8, wherein each of the plurality of clusters maintains a respective connected cluster list created by others of the plurality of clusters.

11. The computer readable storage medium of claim 8, wherein at the time of establishing a communication link, each of the plurality of clusters updates and exchanges their respective connected cluster list with others of the plurality of clusters.

12. The computer readable storage medium of claim 8, wherein the third cluster declares a failure of the first cluster and updates its respective connected cluster list accordingly, when the third cluster is not connected to another cluster to which the first cluster is also connected.

13. The computer readable storage medium of claim 8, wherein the third cluster recognizes the health of the first cluster and updates its respective connected cluster list accordingly, when the third cluster is connected to another cluster to which the first cluster is also connected.

14. A server computer system, comprising:

a computer system having a plurality of clusters comprising at least a first cluster, a second cluster and a third cluster;

a processor coupled to a computer readable storage media and executing computer readable code which causes the computer system to implement a failure detection agent that functions by;

initializing failure detection by creating a connected cluster list, wherein for each one of the plurality of clusters, a respective connected cluster list describes others of the plurality of clusters said each one is communicatively connected with;

sending a status update message upon a change in connectivity between the plurality of clusters;

generating an updated connected cluster list in in accordance with the status update message; and

determining whether the change in connectivity is a result of a cluster failure by examining the updated connected cluster list for each of the plurality of clusters;

wherein upon receiving a loss of communication status update message from the second cluster, the third cluster removes the first cluster from a connected cluster list of the second cluster, and wherein the third cluster checks a connected cluster list of the third cluster to determine whether the third cluster is connected to another cluster to which the first cluster is also connected.

15. The server computer system of claim 14, further comprising determining whether the change in connectivity is a result of a communications link failure by examining the updated connected cluster list for each of the plurality of clusters.

16. The server computer system of claim 14, wherein the status update message comprises a name of a cluster with which a connection was established or broken.

17. The server computer system of claim 14, wherein the server computer system maintains a respective connected cluster list created by each of the plurality of clusters.

18. The server computer system of claim 14, wherein at the time of establishing a communication link, the server computer system updates and exchanges its respective connected cluster list with others of the plurality of clusters.