Distributed file system using consensus nodes
First Claim
1. A computer-implemented method of implementing a distributed file system comprising a cluster comprising a plurality of data nodes configured to store data blocks of client files, the method comprising:
- receiving, by a computer via a network, from a first data node of the cluster, a first request to change a state of a namespace of the cluster, the namespace including at least a location within each data node of the plurality of data nodes of each data block of the client files across the cluster;
receiving, by the computer via the network, from a second data node of the cluster, a second request to change the state of the namespace of the cluster;
sending, by the computer, the first request to a selected first active namenode server computer of a plurality of active namenode server computers, each active namenode server computer of the plurality of active namenode server computers comprising persistent storage memory configured to store updates to the namespace of the cluster, the selected first active namenode server computer being configured to store a first replica of the namespace of the cluster;
sending, by the computer, the second request to a selected second active namenode server computer of the plurality of active namenode server computers, the selected second active namenode server computer being configured to store a second replica of the namespace of the cluster;
receiving, by the computer, a first proposal from the selected first active namenode server computer to change the state of the namespace of the cluster according to the first request and receiving a second proposal from the selected second active namenode server computer to change the state of the namespace of the cluster according to the second request, wherein the selected first and second active namenode server computers are configured to change the state of the namespace of the cluster while at least one other active namenode computer server of the plurality of active namenode server computers is also responding to another request to change the state of the namespace;
generating, by the computer, in response to receiving the first and second proposals from the selected first and second active namenode server computers, an ordered set of agreements that specifies an order in which the selected first and second active namenode server computers are to change the state of their respective first and second replicas;
sending, by the computer, the ordered set of agreements to the selected first and second active namenode server computers;
delaying, by the selected first and second active namenode server computers, making changes to the state of the first and second replicas of the namespace until the ordered set of agreements is received; and
storing updates, by the selected first and second active namenode server computers and based on the ordered set of agreements, in respective persistent storage memories of the selected first and second active namenode server computers, respectively, to maintain consistency of the state of the namespace of the cluster across the selected first and second active namenode server computers.
3 Assignments
0 Petitions
Accused Products
Abstract
A cluster of nodes in a distributed file system may include; at least two namenodes, each coupled to a plurality of data nodes and each configured to store a state of a namespace of the cluster and each being configured to respond to a request from a client while other(s) of the namenodes are responding to other requests from other clients; and a coordination engine coupled to each of the namenodes. The coordination engine may be configured to receive proposals from the namenodes to change the state of the namespace by replicating, deleting and/or adding data blocks stored in the data nodes and to generate, in response, an ordered set of agreements that specifies an order in which the namenodes are to change the state of the namespace. The namenodes are configured to delay making changes thereto until after the ordered set of agreements is received from the coordination engine.
-
Citations
20 Claims
-
1. A computer-implemented method of implementing a distributed file system comprising a cluster comprising a plurality of data nodes configured to store data blocks of client files, the method comprising:
-
receiving, by a computer via a network, from a first data node of the cluster, a first request to change a state of a namespace of the cluster, the namespace including at least a location within each data node of the plurality of data nodes of each data block of the client files across the cluster; receiving, by the computer via the network, from a second data node of the cluster, a second request to change the state of the namespace of the cluster; sending, by the computer, the first request to a selected first active namenode server computer of a plurality of active namenode server computers, each active namenode server computer of the plurality of active namenode server computers comprising persistent storage memory configured to store updates to the namespace of the cluster, the selected first active namenode server computer being configured to store a first replica of the namespace of the cluster; sending, by the computer, the second request to a selected second active namenode server computer of the plurality of active namenode server computers, the selected second active namenode server computer being configured to store a second replica of the namespace of the cluster; receiving, by the computer, a first proposal from the selected first active namenode server computer to change the state of the namespace of the cluster according to the first request and receiving a second proposal from the selected second active namenode server computer to change the state of the namespace of the cluster according to the second request, wherein the selected first and second active namenode server computers are configured to change the state of the namespace of the cluster while at least one other active namenode computer server of the plurality of active namenode server computers is also responding to another request to change the state of the namespace; generating, by the computer, in response to receiving the first and second proposals from the selected first and second active namenode server computers, an ordered set of agreements that specifies an order in which the selected first and second active namenode server computers are to change the state of their respective first and second replicas; sending, by the computer, the ordered set of agreements to the selected first and second active namenode server computers; delaying, by the selected first and second active namenode server computers, making changes to the state of the first and second replicas of the namespace until the ordered set of agreements is received; and storing updates, by the selected first and second active namenode server computers and based on the ordered set of agreements, in respective persistent storage memories of the selected first and second active namenode server computers, respectively, to maintain consistency of the state of the namespace of the cluster across the selected first and second active namenode server computers. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)
-
-
12. A cluster of nodes comprising computing devices configured to implement a distributed file system, comprising:
-
a plurality of data nodes connected to a network, each data node being configured to store data blocks of client files; a plurality of active namenode server computers, each active namenode server computer being coupled to the plurality of data nodes and said each active namenode server computer being configured to locally store a separate replica of a namespace of the cluster, the namespace including at least a location within each data node of the plurality of data nodes of each data block of the client files across the cluster, said each active namenode server computer of the plurality of active namenode server computers comprising persistent storage memory configured to store updates to a state of the namespace of the cluster and being configured to respond to requests to change the state of the namespace of the cluster while at least one other active namenode server computer of the plurality of active namenode server computers is also responding to other requests to change the state of the namespace; a proxy interface computer configured to receive client requests to change the state of the namespace of the cluster and to select an active namenode server computer from the plurality of active namenode server computers to which to send one or more of the received client requests; and a coordination engine computer configured to receive proposals to change the state of the namespace of the cluster from the selected active namenode server computers having received the one or more client requests from the proxy interface computer and, in response, to generate an ordered set of agreements that specifies an order in which the selected active namenode server computers are to change their respective locally-stored replicas to maintain the state of the namespace of the cluster consistent across the plurality of active namenode server computers and to send the ordered set of agreements to the selected active namenode server computers, wherein said each selected active namenode server computer of the plurality of active namenode server computers is further configured to delay making changes to the state of its respective locally-stored replica of the namespace until receipt of the ordered set of agreements from the coordination engine computer. - View Dependent Claims (13, 14, 15, 16, 17, 18, 19, 20)
-
Specification