Distributed database management system with node failure detection

• US 9,501,363 B1
• Filed: 03/17/2014
• Issued: 11/22/2016
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: NuoDB, Inc. (Dassault Systemes SA)
• Original Assignee: NuoDB, Inc. (Dassault Systemes SA)
• Inventors: Ottavio, Daniel P.
• Primary Examiner(s): Manoskey, Joseph D
• Application Number:
  US14/215,372
• Time in Patent Office: 981 Days
• Field of Search: 714/4.1, 714/4.2, 714/4.4, 714/25, 714/33, 714/37, 714/40, 714/47.1
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16

Abstract

A node failure detector for use in a distributed database that is accessed through a plurality of interconnected transactional and archival nodes. Each node is selected as an informer node that tests communications with each other node. Each informer node generates a list of suspicious nodes that is resident in one node designated as a leader node. The leader node analyzes the data from all of the informer nodes to designate each node that should be designated for removal with appropriate failover procedures.

Claims

1. In a distributed database management processing system including a plurality of nodes and means at each node for establishing a communications path with every other node in the system and each node includes means for transmitting a ping message and for transmitting a ping acknowledgement message in response to a received ping message, a method for detecting failures of individual nodes in the system comprising the steps of:

A) designating one node as a leader node for analyzing information,

B) at each node as an I-node node, transmitting a ping message to each other node as a receiving node, monitoring at the I-node the corresponding communications path for a valid response from each receiving node, and responding to an invalid response by designating the corresponding receiving node as a suspicious node,

C) generating a message for transmittal to the leader node with an identification of the I-node and the suspicious node,

D) responding to the message in the leader node by identifying other instances for which communications problems have been recorded with the identified suspicious node, determining a number of I-nodes included in the other instances, and where fewer than a majority of the I-nodes are included in other instances, sending an acknowledgement message to all the I-nodes, and

E) selectively designating suspicious nodes as failed where the majority of the I-nodes identify the suspicious nodes in a generated message or the majority of the I-nodes identify the suspicious nodes in a response to the acknowledgment message.

2. The method as recited in claim 1 wherein the leader node at the time of designation not be a suspicious node and upon a leader node becoming suspicious, transferring the designation of a leader node to another node.

3. The method as recited in claim 1 wherein upon a transmittal of a message to the leader node by an I-node, the I-node upon a failure to receive an acknowledgment by the existing leader node selects another node that is not suspicious.

4. The method as recited in claim 1 wherein said selective designation of suspicious nodes as failed determines if a selected suspicious node is suspicious to only the selected I-node.

5. The method as recited in claim 4 wherein the node with the highest node identification is designated for being disabled if the selected suspicious node is suspicious to only the selected I-node.

6. The method as recited in claim 4 wherein all remaining suspicious nodes associated with that I-node are designated for being disabled if the selected suspicious node is suspicious to more than the selected I-node.

7. In a distributed database management processing system including a plurality of nodes and means at each node for establishing a communications path with every other node in the system and each node includes means for transmitting a ping message and for transmitting a ping acknowledgement message in response to a received ping message, means for detecting failures of individual nodes in the system comprising:

A) means for designating one node as a leader node for analyzing information,

B) at each node as an I-node node, means for transmitting a ping message to each other node as a receiving node, means for monitoring at the I-node the corresponding communications path for a valid response from each receiving node, and means for responding to an invalid response by designating the corresponding receiving node as a suspicious node,

C) means for generating a message for transmittal to the leader node with an identification of the I-node and the suspicious node,

D) means for responding to the message in the leader node by identifying other instances for which communications problems have been recorded with the identified suspicious node, means for determining a number of I-nodes included in the other instances, and means for sending an acknowledgement message to all the I-nodes where fewer than a majority of the I-nodes are included in other instances, and

E) means for selectively designating suspicious nodes as failed where the majority of the I-nodes identify the suspicious nodes in a generated message or the majority of the I-nodes identify the suspicious nodes in a response to the acknowledgment message.

8. The system as recited in claim 7 wherein the leader node at the time of designation is not a suspicious node and means responsive to a leader node becoming suspicious, for transferring the designation of a leader node to another node.

9. The system as recited in claim 7 wherein upon a transmittal of a message to the leader node by an I-node and including means responsive to the failure of an I-node to receive an acknowledgment by the existing leader node for selecting another node that is not suspicious.

10. The system as recited in claim 7 wherein said means for selectively designating suspicious nodes as failed includes means for determining if a selected suspicious node is suspicious to only the selected I-node.

11. The system as recited in claim 10 wherein the node including means responsive to the highest node identification for designating the node for being disabled if the selected suspicious node is suspicious to only the selected I-node.

12. The system as recited in claim 10 additional comprising means for designating all remaining suspicious nodes associated with that I-node for being disabled if the selected suspicious node is suspicious to more than the selected I-node.

13. A distributed database management processing system comprising:

a plurality of nodes comprising a leader node and one or more I-nodes, each node of the plurality of nodes comprising a network interface coupled to a database system network and a processing system coupled to the network interface, each I-node of the one or more I-nodes being configured to;

transmit a ping message to each other node of the plurality of nodes;

monitor the database system network for a valid response from each other node of the plurality of nodes;

respond to an invalid response from a responding node by designating the responding node as a suspicious node; and

generate a message for transmittal to the leader node, the message including an identification of the I-node and the suspicious node;

the leader node being configured to;

respond to the message by identifying other invalid responses from the suspicious node, determining a number of the one or more I-nodes that received invalid responses from the suspicious node, and where fewer than a majority of the one or more I-nodes received invalid responses from the suspicious node, sending an acknowledgement message to all of the one or more I-nodes; and

selectively designate suspicious nodes as failed where the majority of the one or more I-nodes identify the suspicious nodes in a generated message or the majority of the one or more I-nodes identify the suspicious nodes in a response to the acknowledgment message.

14. The system as recited in claim 13 wherein the I-node that generated the message is further configured to select another node of the plurality of nodes as a new leader node upon a failure to receive an acknowledgment from the leader node, the new leader node being a node of the plurality of nodes that is not a suspicious node.

15. A distributed database management processing system comprising:

a plurality of nodes comprising a leader node and one or more I-nodes, each node of the plurality of nodes comprising a network interface coupled to a database system network and a processing system coupled to the network interface, each I-node of the one or more I-nodes being configured to;

transmit a ping message to each other node of the plurality of nodes;

monitor the database system network for a valid response from each other node of the plurality of nodes;

respond to an invalid response from a responding node by designating the responding node as a suspicious node; and

generate a message for transmittal to the leader node, the message including an identification of the I-node and the suspicious node;

the leader node being configured to;

respond to the message by identifying that the suspicious node responded with an invalid response to only the I-node that generated the message;

selectively designate the suspicious node as failed where the suspicious node has a higher node identification than the I-node that generated the message; and

selectively designate the I-node that generated the message as failed where the I-node that generated the message has a higher node identification than the suspicious node.

16. The system as recited in claim 15 wherein the I-node that generated the message is further configured to select another node of the plurality of nodes as a new leader node upon a failure to receive an acknowledgment from the leader node, the new leader node being a node of the plurality of nodes that is not a suspicious node.