Distributed file system using consensus nodes

• US 9,361,311 B2
• Filed: 08/29/2013
• Issued: 06/07/2016
• Est. Priority Date: 01/12/2005
• Status: Active Grant

• Current Assignee: Cirata Incorporated
• Original Assignee: WANdisco, Inc.
• Inventors: Shvachko, Kostantin V, Sundar, Jagane, Parkin, Micahel, Aahlad, Yeturu
• Primary Examiner(s): Truong, Cam-Y
• Application Number:
  US14/013,948
• Publication Number:
  US 20150067002A1
• Time in Patent Office: 1,013 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems

Abstract

A cluster of nodes in a distributed file system may include; at least two namenodes, each coupled to a plurality of data nodes and each configured to store a state of a namespace of the cluster and each being configured to respond to a request from a client while other(s) of the namenodes are responding to other requests from other clients; and a coordination engine coupled to each of the namenodes. The coordination engine may be configured to receive proposals from the namenodes to change the state of the namespace by replicating, deleting and/or adding data blocks stored in the data nodes and to generate, in response, an ordered set of agreements that specifies an order in which the namenodes are to change the state of the namespace. The namenodes are configured to delay making changes thereto until after the ordered set of agreements is received from the coordination engine.

Claims

1. A cluster of nodes comprising computing devices configured to implement a distributed file system, comprising:

a plurality of data nodes, each data node configured to store data blocks of client files;

at least two active namenodes, each active namenode coupled to the plurality of data nodes and said each active namenode configured to store a state of a same namespace of the cluster, the namespace including at least a location within said each data node of the plurality of data nodes of each data block of the client files across the cluster, said each active namenode of the at least two active namenodes being configured to respond to any request from any client of the distributed file system to generate a new data block or enable a new data block to be stored on any of the data nodes while at least one other of the active namenodes is responding to any other request from the same or any other client to generate another new data block or enable another new data block to be stored on any of the data nodes, only one of the active namenodes being configured to periodically issue a block replicator heartbeat message and to respond to requests from any client to replicate or delete data blocks from the data nodes;

persistent storage memory, coupled to the at least two active namenodes, configured to store at least one journal containing updates to the namespace of the cluster; and

a coordination engine coupled to said each active namenode of the active namenodes, the coordination engine being configured to receive proposals from the active namenodes to change the state of the namespace by at least one of replicating, deleting and adding data blocks in at least one of the plurality of data nodes and to generate, in response, an ordered set of agreements that specifies an order in which the active namenodes are to change the state of the namespace, wherein the active namenodes are configured to delay making changes to the state of the namespace and updating the journal in the persistent storage memory until the ordered set of agreements is received from the coordination engine,

wherein failure of the active namenode configured to enable replication and deletion of data blocks to periodically issue the block replicator heartbeat triggers an election process to elect a new active namenode that is to be solely configured to enable replication and deletion of data blocks.

2. The cluster of nodes of claim 1, wherein each active namenode is configured to generate a proposal responsive to a client issuing a request to at least one of replicating and deleting the data blocks in the data nodes.

3. The cluster of claim 1, wherein the coordination engine is further configured to assign a unique global sequence number (GSN) to each agreement, the GSN specifying an order in which the active namenodes are to apply the changes to the state of the namespace.

4. The cluster of claim 3, wherein the active namenodes are further configured to apply the changes to the state of the namespace in the order specified by the GSN of the received agreements.

5. The cluster of claim 1, further comprising local storage coupled to each active namenode of the plurality of active namenodes, the local storage being configured to store an image of the namespace and entries specifying updates to the stored image.

6. The cluster of claim 1, wherein the election process is configured to be initiated by any of the active namenodes.

7. The cluster of claim 1, wherein each active namenode of the active namenodes is configured to enable data blocks stored on one of the plurality of data nodes to be replicated to a predetermined other one or other ones of the plurality of data nodes.

8. The cluster of claim 7, wherein at least one of the data nodes, upon receipt of data blocks, is configured to send the data blocks to another of the plurality of data nodes identified by one of the active namenodes.

9. The cluster of claim 1, wherein each data node of the plurality of data nodes is configured to periodically issue a heartbeat message to the active namenodes, indicating a current status of the data node.

10. The cluster of claim 9, wherein the active namenodes are further configured to consider a data node from which the periodic heartbeat has not been timely received to have failed.

11. The cluster of claim 1, wherein one of the active namenodes is further configured to cause data blocks stored in a failed data node to be replicated to another selected one of the plurality of data nodes.

12. The cluster of claim 1, wherein each active namenode of the active namenodes is configured to assign a unique block identifier (ID) to data blocks from within a sub-range of a larger range of block ID numbers that is separate from other respective sub-ranges of the larger range of block IDs from which other ones of the active namenodes assign block ID numbers.

13. The cluster of claim 1, further comprising a communication timeout configured such that, if an active namenode having received a client request has not responded thereto at an expiry of the communication timeout, the client request is made to another of the active namenodes.

14. The cluster of claim 1, wherein all of the active namenodes in the cluster are configured to actively support client requests simultaneously.

15. The cluster of claim 1 wherein, upon failure of one of the active namenodes, client requests previously made to the failed namenode are made to an other one of the active namenodes.

16. The cluster of claim 15 wherein, prior to making client requests to the other active namenode, the client is configured to wait until the other active namenode updates its namespace.

17. The cluster of claim 1, wherein each active namenode of the active namenodes, to prevent stale reads, are configured to submit a coordinated read proposal to the coordination engine when a request is received to read a file that is determined to satisfy selected file name patterns and to enable the file to be read only upon receiving an agreement corresponding to the coordinated read proposal.

18. The cluster of claim 1, further configured to enable a new active namenode to be brought into the cluster, wherein the new active namenode is configured to update the state of its namespace by at least one of:

loading a checkpoint comprising a prior image of the namespace, and

accepting agreements to its namespace from the coordination engine.

19. The cluster of claim 1, wherein the proposals to change the state of the namespace comprise at least one of replicating, deleting and adding data blocks in at least one of the plurality of data nodes.

21. The computing device of claim 19, wherein the proposals to change the state of the namespace comprise at least one of replicating, deleting and adding data blocks in at least one of the plurality of data nodes.

20. A computing system configured as a part of a distributed file system, comprising:

a first persistent storage memory and a second persistent storage;

a first active namenode coupled to a plurality of data nodes, each data node of the plurality of data nodes is configured to store data blocks of client files, the first active namenode being configured to store a state of a same namespace of a cluster of computing nodes and to store updates to the namespace in the first persistent storage memory, the namespace including at least a location within said each data node of the plurality of data nodes of each data block of the client files across the cluster, the first active namenode being configured to be coupled to a second active namenode that is also coupled to the plurality of data nodes and that is also configured to store the state of the namespace and to store updates to the namespace in the second persistent storage, the first active namenode being further configured to respond to any request from any client of the distributed file system to generate a new data block or enable a new data block to be stored on any of the data nodes while the second active namenode is responding to any other request from the same or any other client to generate another new data block or enable another new data block to be stored on any of the data nodes, only one of the first and second active namenodes being configured to periodically issue a block replicator heartbeat message and to respond to requests from any client to replicate or delete data blocks from the data nodes; and

a first coordination engine coupled to the first active namenode, the coordination engine being configured to receive proposals from the first active namenode to change the state of the namespace and to generate, in response, an ordered set of agreements that specifies an order in which the first and the second active namenodes are to change the state of the namespace, wherein the first active namenode is configured to delay making changes to the state of the namespace until the ordered set of agreements is received from the coordination engine, and

wherein failure of the active namenode configured to respond to requests to replicate or delete data blocks to periodically issue the block replicator heartbeat triggers an election process to elect a new active namenode that is to be solely configured to enable replication and deletion of data blocks.