Distributed file system using consensus nodes
First Claim
1. A computer-implemented method of implementing a distributed file system comprising an entire cluster comprising a plurality of data nodes configured to store data blocks of client files, the method, comprising:
- coupling at least two active namenodes to a plurality of data nodes, each active namenode of the active namenodes being configured to store a state of a same namespace of the entire cluster, the namespace including at least a location within each data node of the plurality of data nodes of each data block of the client files across the entire cluster, said each active namenode of the at least two active namenodes being configured to respond to any request from any client of the distributed file system to generate a new data block or enable a new data block to be stored on any of the data nodes while at least one other of the active namenodes is responding to any other request from the same or any other client to generate another new data block or enable another new data block to be stored on any of the data nodes, only one of the active namenodes being configured to respond to requests from any client to replicate or delete data blocks from the data nodes;
receiving, by using a computer system, proposals from the active namenodes to change the state of the namespace, andgenerating, in response to receiving the proposals, an ordered set of agreements that specifies an order in which the active namenodes are to change the state of the namespace and assigning a unique global sequence number (GSN) to each agreement of the ordered set of agreements, the GSN specifying an order in which the active namenodes are to apply changes to the state of the namespace, such that the active namenodes delay making the changes to the state of the namespace until the active namenodes receive the ordered set of agreements and apply the changes to the state of the namespace in the order specified by the GSN of each received agreement of the received agreements.
3 Assignments
0 Petitions
Accused Products
Abstract
A cluster of nodes in a distributed file system may include; at least two namenodes, each coupled to a plurality of data nodes and each configured to store a state of a namespace of the cluster and each being configured to respond to a request from a client while other(s) of the namenodes are responding to other requests from other clients; and a coordination engine coupled to each of the namenodes. The coordination engine may be configured to receive proposals from the namenodes to change the state of the namespace by replicating, deleting and/or adding data blocks stored in the data nodes and to generate, in response, an ordered set of agreements that specifies an order in which the namenodes are to change the state of the namespace. The namenodes are configured to delay making changes thereto until after the ordered set of agreements is received from the coordination engine.
-
Citations
20 Claims
-
1. A computer-implemented method of implementing a distributed file system comprising an entire cluster comprising a plurality of data nodes configured to store data blocks of client files, the method, comprising:
-
coupling at least two active namenodes to a plurality of data nodes, each active namenode of the active namenodes being configured to store a state of a same namespace of the entire cluster, the namespace including at least a location within each data node of the plurality of data nodes of each data block of the client files across the entire cluster, said each active namenode of the at least two active namenodes being configured to respond to any request from any client of the distributed file system to generate a new data block or enable a new data block to be stored on any of the data nodes while at least one other of the active namenodes is responding to any other request from the same or any other client to generate another new data block or enable another new data block to be stored on any of the data nodes, only one of the active namenodes being configured to respond to requests from any client to replicate or delete data blocks from the data nodes; receiving, by using a computer system, proposals from the active namenodes to change the state of the namespace, and generating, in response to receiving the proposals, an ordered set of agreements that specifies an order in which the active namenodes are to change the state of the namespace and assigning a unique global sequence number (GSN) to each agreement of the ordered set of agreements, the GSN specifying an order in which the active namenodes are to apply changes to the state of the namespace, such that the active namenodes delay making the changes to the state of the namespace until the active namenodes receive the ordered set of agreements and apply the changes to the state of the namespace in the order specified by the GSN of each received agreement of the received agreements. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18)
-
-
19. A non-transitory machine-readable medium having data stored thereon representing sequences of instructions which, when executed by computing devices, cause the computing devices to implement a distributed file system by:
-
coupling at least two active namenodes to a plurality of data nodes, each active namenode of the active namenodes being configured to store a state of a same namespace of an entire cluster, the namespace including at least a location within each data node of the plurality of data nodes of each data block of client files across the entire cluster, said each active namenode of the at least two active namenodes being configured to respond to any request from any client of the distributed file system to generate a new data block or enable a new data block to be stored on any of the data nodes while at least one other of the active namenodes is responding to any other request from the same or any other client to generate another new data block or enable another new data block to be stored on any of the data nodes, only one of the active namenodes being configured to respond to requests from any client to replicate or delete data blocks from the data nodes; receiving proposals from the active namenodes to change the state of the namespace, and generating, in response to receiving the proposals, an ordered set of agreements that specifies an order in which the active namenodes are to change the state of the namespace and assigning a unique global sequence number (GSN) to each agreement of the ordered set of agreements, the GSN specifying an order in which the active namenodes are to apply changes to the state of the namespace, such that the active namenodes delay making the changes to the state of the namespace until the active namenodes receive the ordered set of agreements and apply the changes to the state of the namespace in the order specified by the GSN of each received agreement of the received agreements. - View Dependent Claims (20)
-
Specification