Data node fencing in a distributed file system

• US 9,753,954 B2
• Filed: 09/11/2013
• Issued: 09/05/2017
• Est. Priority Date: 09/14/2012
• Status: Active Grant

• Current Assignee: Cloudera Incorporated
• Original Assignee: Cloudera Incorporated
• Inventors: Lipcon, Todd, Myers, Aaron T., Collins, Eli
• Primary Examiner(s): Coby, Frantz
• Application Number:
  US14/024,585
• Publication Number:
  US 20140081927A1
• Time in Patent Office: 1,455 Days
• Field of Search: 707793, 707822, 707825, 707827, 714 411, 709203, 709213, 709217, 709220, 709223-226, 709245-246
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence
• CPC Class Codes: G06F 16/1824 implemented using Network-attached Storage [NAS] architecture distributed or networked storage systems G06F3/067; protocols for distributed storage of data in a network H04L67/1097
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: H04L 41/0836 to enhance reliability, e.g. reduce downtime

Abstract

Systems and methods for data node fencing in a distributed file system to prevent data inconsistencies and corruptions are disclosed. An embodiment includes implementing a protocol whereby data nodes detect a failover and determine an active name node based on transaction identifiers associated with transaction requests. The data nodes also provide to the active name node block location information and an acknowledgment. The embodiment further includes a protocol whereby a name node refrains from issuing invalidation requests to the data nodes until the name node receives acknowledgments from all data nodes that are functional.

Claims

1. A method for maintaining data correctness in a Hadoop™

based distributed cluster during a failover, in which an original name node is switched to a backup name node due to failure of the original name node, the distributed cluster having a plurality of data nodes and one or more processors, the method being performed by the one or more processors and comprising;

on the backup name node;

assuming an active role to become a new active name node, upon detecting that the original name node has failed;

flagging all of the plurality of data nodes as untrusted;

for each data node among the plurality of data nodes;

queuing, instead of issuing, commands intended for a data node until the data node is flagged as trusted, and

upon receiving an acknowledgement from the data node acknowledging the assumption of the active role of the backup name node, flagging the data node as trusted; and

on a respective data node;

receiving a first command with a first transaction number from a first name node;

receiving a second command with a second transaction number from a second name node, wherein the second transaction number is greater than the first transaction number; and

sending an acknowledgment of an active role to the second name node.

2. The method of claim 1, further comprising:

sending a message to the data node, wherein the message includes a most recent transaction identifier known to the backup name node assuming the active role.

3. The method of claim 1, wherein commands on any block with replicated data on untrusted data nodes are queued.

4. The method of claim 1, further comprising receiving a data report in addition to the acknowledgment of the active role from the data node.

5. The method of claim 4, wherein the data report includes information regarding location of replicated data stored in the data node.

6. The method of claim 4, wherein each data report includes a list of pending deletions.

7. A Hadoop™

based distributed cluster comprising an original name node, a backup name node, and a distributed file system having a plurality of data nodes,

wherein one or more processors of the backup name node are configured to perform;

assuming an active role to become a new active name node, upon detecting that the original name node has failed;

flagging all of the plurality of data nodes as untrusted;

for each data node among the plurality of data nodes;

queuing, instead of issuing, commands intended for a data node until the data node is flagged as trusted; and

upon receiving an acknowledgement from the data node acknowledging the assumption of the active role of the backup name node, flagging the data node as trusted, and

wherein one or more processors of a respective data node are configured to perform;

receiving a first command with a first transaction number from a first name node;

receiving a second command with a second transaction number from a second name node, wherein the second transaction number is greater than the first transaction number; and

sending an acknowledgment of an active role to the second name node.

9. The cluster of claim 7, wherein the data nodes are configured to ignore commands from other name nodes that issue commands having a transaction identifier lower than a transaction identifier associated with a command issued by the backup name node.

8. A machine-readable storage medium having stored thereon instructions which, when executed by one or more processors, configure the processors to performs a method in a Hadoop™

based distributed cluster comprising a plurality of name nodes and a plurality of data nodes and having a distributed file system, the method comprising;

on the backup name node;

assuming an active role to become a new active name node, upon detecting that the original name node has failed;

flagging all of the plurality of data nodes as untrusted;

for each data node among the plurality of data nodes;

queuing, instead of issuing, commands intended for a data node until the data node is flagged as trusted, and

upon receiving an acknowledgement from the data node acknowledging the assumption of the active role of the backup name node, flagging the data node as trusted; and

on a respective data node;

receiving a first command with a first transaction number from a first name node;

receiving a second command with a second transaction number from a second name node, wherein the second transaction number is greater than the first transaction number; and

sending an acknowledgment of an active role to the second name node.