DECENTRALIZING A HADOOP CLUSTER

• US 20140164452A1
• Filed: 12/06/2012
• Published: 06/12/2014
• Est. Priority Date: 12/06/2012
• Status: Active Grant

• Current Assignee: Xero Limited
• Original Assignee: Empire Technology Development LLC (Allied Inventors Management, LLC)
• Inventors: Ying, Lizhi, Song, Xuefeng
• Granted Patent: US 9,588,984 B2
• Time in Patent Office: Days
• US Class Current: 707/827
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/182 Distributed file systems

Abstract

In some examples, a distributed file system is described. The distributed file system may include multiple data nodes and a director unit. The multiple data nodes may each include one or more data blocks. The director unit may include multiple master nodes configured in a peer-to-peer distributed architecture and operably coupled to the multiple data nodes. Each of the master nodes may be configured to receive a task related to managing data with respect to the distributed file system; to manage a distribution of the task among one or more of the plurality of master nodes; and to communicate a task status and a status of at least a part of the distributed file system to each of the other master nodes.

Claims

1. A distributed file system comprising:

data nodes that each include one or more data blocks;

a director unit including master nodes configured in a peer-to-peer distributed architecture and operably coupled to the data nodes, each of the master nodes configured to;

receive a task related to managing data with respect to the distributed file system;

conduct a master node load balancing operation among at least some of the master nodes;

based on the load balancing operation, manage a distribution of the task among one or more of the master nodes and one or more of the data nodes; and

communicate a task status and a status of at least a part of the distributed file system to each of the master nodes not distributed the task.

2. The distributed file system of claim 1, wherein each master node is further configured to:

divide the task into sub-tasks;

distribute the sub-tasks among one or more of the master nodes; and

after the sub-tasks are performed by the one or more of the data nodes, collect results of the sub-tasks.

5. The distributed file system of claim 1, wherein the task includes one or more of:

writing a first file to the distributed file system, accessing a second file in the distributed file system, processing data in the distributed file system, changing a name of a third file in the distributed file system, deleting a fourth file in the distributed file system, and communicating a computing task to another master node of the master nodes.

6. The distributed file system of claim 1, wherein when the task includes accessing a file in the distributed file system, each of the master nodes is further configured to:

receive the task including file metadata identifying the file;

attempt to find the file based on the file metadata;

when the file is not found, distribute the task to another of the master nodes;

when the file is found, transmit system metadata indicating a location of the file and transmit a success message; and

when the file is not found by any of the master nodes, transmit a null message.

8. The distributed file system of claim 1, wherein when the task includes writing a file to the distributed file system, each of the master nodes is further configured to:

receive file metadata identifying the file to be written to the distributed file system;

attempt to find the file based on the file metadata;

when the file is not found, distribute the file metadata to another of the master nodes; and

when the file is not found by any of the master nodes, open a writing flow.

9. The distributed file system of claim 1, wherein when the task includes a computing task, a first master node of the master nodes is selected, and

each other of the master nodes is further configured to:

receive the computing task from the first master node;

obtain master node-specific results by performing the computing task locally;

determine whether a portion of the master nodes has received the computing task;

when the portion of the master nodes has not received the computing task, transmit the master node-specific results and the computing task to a second master node of the master nodes; and

when the portion of the master nodes has received the computing task, transmit the master node-specific results to the first master node.

10. The distributed file system of claim 9, wherein each of the master nodes is further configured to:

receive other master node-specific results from other master nodes of the master nodes;

combine the master node-specific results with the other master node-specific results;

when the portion of the master nodes has received the computing task, transmit a combination of the master node-specific result and the other master node-specific results to the second master node; and

when the portion of the master nodes has not received the computing task, transmit the combination of the master node-specific result and the other master node-specific results to the first master node.

11. The distributed file system of claim 1, wherein each of the master nodes is further configured to:

evaluate local computing power availability; and

when the local computing power availability is below a threshold computing power, transmit a message indicating insufficient computing power to another of the master nodes.

13. The distributed file system of claim 1, wherein the distribution of the task includes assigning the task to one of the master nodes, wherein each of the master nodes is further configured to:

receive the task;

evaluate whether redundancy of the master nodes is available;

when redundancy is available, accept the task; and

when redundancy is not available, transmit the task to another of the master nodes.

15. The distributed file system of claim 1, wherein when one of the master nodes fails, the master nodes remaining in operation are configured to reassign among the remaining master nodes a task assigned to the failed master node.

16. The distributed file system of claim 1, wherein the distribution includes randomly assigning the task to one of the master nodes.

19. The distributed file system of claim 1, wherein when the task includes writing a file to the distributed file system, a master node of the master nodes that receives the task is configured to:

conduct a data node load balancing operation amongst the data nodes;

based on the data node load balancing operation, select a data node of the data nodes to which to write the file;

open a writing flow; and

transmit a success message.

3-4. -4. (canceled)

7. (canceled)

12. (canceled)

14. (canceled)

17-18. -18. (canceled)

20. (canceled)

21. A method of managing a distributed file system comprising:

receiving a task related to managing data with respect to the distributed file system;

selecting one or more master nodes to manage the task, the master nodes configured in a peer-to-peer distributed architecture and operably coupled to data nodes;

conducting a master node load balancing operation among at least some of master nodes;

based on the load balancing operation, managing the distribution of the task among the data nodes and master nodes, the managing performed by the selected master node; and

communicating among at least some of the master nodes a task status and a status of at least a part of the distributed file system.

22. The method of claim 21, further comprising:

dividing the task into sub-tasks;

distributing the sub-tasks among one or more of the master nodes; and

following a performance by the one or more of the data nodes of the sub-tasks, collecting results of the sub-tasks.

23. The method of claim 21, wherein when the task includes accessing a file from the distributed file system, the method further comprises:

receiving the task including file metadata that identifies the file;

attempting to find the file based on the file metadata;

when the file is not found, distributing the task to another of the master nodes;

when the file is found, transmitting system metadata indicating a location of the file and transmitting a success message; and

transmitting a null message when the file is not found by any of the master nodes.

25. The method of claim 21, wherein when the task includes writing a file to the distributed file system, the method further comprises:

receiving file metadata that identifies the file to be written to the distributed file system;

attempting to find the file based on the file metadata;

when the file is not found, distributing the file metadata to another of the master nodes; and

when the file is not found by any of the master nodes, opening a writing flow.

26. The method of claim 21, wherein when the task includes a computing task, the method further comprises:

selecting a first master node of the master nodes;

receiving the computing task from the first master node;

obtaining master node-specific results by performing the computing task locally;

determining whether a portion of the master nodes has received the computing task;

when the portion of the master nodes has not received the computing task, transmitting the master node-specific results and the computing task to a second master node of the master nodes; and

when the portion of the master nodes has received the computing task, transmitting the master node-specific results to the first master node.

27. The method of claim 26, further comprising:

receiving other master node-specific results from other master nodes of the master nodes;

combining the master node-specific results with the other master node-specific results;

when the portion of the master nodes has received the computing task, transmitting a combination of the master node-specific result and the other master node-specific results to the second master node; and

when the portion of the master nodes has not received the computing task, transmitting the combination of the master node-specific result and the other master node-specific results to the first master node.

28. The method of claim 21, further comprising:

evaluating local computing power availability at one of the master nodes; and

when the local computing power availability is below a threshold computing power, transmitting a message indicating insufficient computing power to another of the master nodes.

30. The method of claim 21, wherein the conducting the master node load balancing operation includes:

receiving the task;

evaluating whether redundancy of the master nodes is available;

when redundancy is available, accepting the task; and

when redundancy is not available, transmitting the task to another of the master nodes.

31. The method of claim 21, wherein the conducting includes:

evaluating a number of tasks that each of the master nodes is performing; and

assigning the task to a first master node of the master nodes performing a lowest number of tasks.

32. The method of claim 21, further comprising:

when one of the master nodes fails, reassigning among the non-failed master nodes a task assigned to the failed master node.

33. The method of claim 21, wherein when the task includes writing a file to the distributed file system, the method comprises:

conducting a data node load balancing operation amongst the data nodes;

based on the data node load balancing operation, selecting a data node of the data nodes to which to write the file;

opening a writing flow; and

transmitting a success message.

35. The method of claim 21, further comprising randomly assigning the task to one of the master nodes.

24. (canceled)

29. (canceled)

34. (canceled)

36. (canceled)

37. A method of managing a distributed file system comprising:

receiving a task related to managing data with respect to the distributed file system;

selecting one or more master nodes to manage the task, the master nodes configured in a peer-to-peer distributed architecture and operably coupled to data nodes;

conducting a master node load balancing operation among at least some of the master nodes;

based on the load balancing operation, managing the distribution of the task among one the data nodes and the master nodes,

when the task includes accessing a file from the distributed file system, the managing including;

receiving the task including file metadata that identifies the file;

attempting to find the file based on the file metadata;

distributing the task to another of the master nodes when the file is not found; and

transmitting system metadata indicating a location of the file and transmitting a success message when the file is found, and

when the task includes a computing task, the managing includes;

receiving the computing task from the selected master node;

obtaining master node-specific results by performing the computing task locally;

determining whether a portion of the master nodes has received the computing task;

transmitting the master node-specific results and the computing task to a second master node of the master nodes when the portion of the master nodes has not received the computing task; and

transmitting the master node-specific results to the first master node when the portion of the master nodes has received the computing task.

38. The method of claim 37, further comprising communicating among at least some of the-master nodes a task status and a status of at least a part of the distributed file system.

39. The method of claim 37, further comprising:

receiving other master node-specific results from other master nodes;

combining the master node-specific results with the other master node-specific results;

when the portion of the master nodes has received the computing task, transmitting a combination of the master node-specific result and the other master node-specific results to the second master node; and

when the portion of the master nodes has not received the computing task, transmitting the combination of the master node-specific result and the other master node-specific results to the first master node.

40. The method of claim 37, further comprising:

evaluating local computing power availability at one of the master nodes; and

when the local computing power availability is below a threshold computing power, transmitting a message indicating insufficient computing power to another of the master nodes.