Distributed processing system including a name node and a plurality of data nodes, and method of operating the same

• US 9,633,040 B2
• Filed: 10/17/2014
• Issued: 04/25/2017
• Est. Priority Date: 02/04/2014
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Lee, Jong-Won
• Primary Examiner(s): WASEL, MOHAMED A
• Application Number:
  US14/516,692
• Publication Number:
  US 20150222695A1
• Time in Patent Office: 921 Days
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: H04L 2101/622 Layer-2 addresses, e.g. medium access control [MAC] addresses
• CPC Class Codes: H04L 41/0896 Bandwidth or capacity management, i.e. automatically increasing or decreasing capacities flow or congestion control using dynamic resource allocation, e.g. in-call renegotiation, H04L47/76
• CPC Class Codes: H04L 41/0897 by horizontal or vertical scaling of resources, or by migrating entities, e.g. virtual resources or entities
• CPC Class Codes: H04L 41/12 Discovery or management of network topologies
• CPC Class Codes: H04L 41/122 of virtualised topologies, e.g. software-defined networks [SDN] or network function virtualisation [NFV]
• CPC Class Codes: H04L 61/103 across network layers, e.g. resolution of network layer into physical layer addresses or address resolution protocol [ARP]

Abstract

A method of operating a distributed processing system including a name node and a plurality of data nodes includes transmitting an address request packet from the name node to each of the plurality of data nodes, transmitting an address packet from each of the plurality of data nodes to the name node upon receiving the address request packet at each of the plurality of data nodes and upon each of the plurality of data nodes storing its own media access control (MAC) address in a payload of the address packet as a data node MAC address, determining, by the name node, a node configuration of the distributed processing system based on the address packet received from each of the plurality of data nodes, and processing data using a selected data node from among the plurality of data nodes based on the node configuration.

Claims

1. A method of operating a distributed processing system including a name node and a plurality of data nodes, comprising:

transmitting an address request packet from the name node to each of the plurality of data nodes;

transmitting an address packet from each of the plurality of data nodes to the name node upon receiving the address request packet at each of the plurality of data nodes and upon each of the plurality of data nodes storing its own media access control (MAC) address in a payload of the address packet as a data node MAC address;

determining, by the name node, a node configuration of the distributed processing system based on the address packet received from each of the plurality of data nodes; and

processing data using a selected data node from among the plurality of data nodes based on the node configuration,

wherein determining the node configuration comprises;

reading a source name node MAC address and a name node MAC address from a payload of an address return packet and a source internet protocol (IP) address from a header of the address return packet; and

determining whether the name node operates on a virtual machine by comparing the source name node MAC address and the name node MAC address.

2. The method of claim 1, wherein at least one node from among the name node and the plurality of data nodes operates on the virtual machine.

3. The method of claim 1, wherein each of the plurality of data nodes reads its own MAC address from an internal register, stores the read MAC address in the payload of the address packet as the data node MAC address, and transmits the address packet to the name node upon receiving the address request packet from the name node.

4. The method of claim 1, wherein the name node determines the node configuration based on a source MAC address included in a header of the address packet and the data node MAC address included in the payload of the address packet.

9. The method of claim 1, further comprising:

selecting, by the name node, at least one data node from among the plurality of data nodes operating on the virtual machine as a virtual data node based on the node configuration;

transmitting the address request packet, repeatedly, from the name node to the at least one virtual data node;

transmitting the address packet from the at least one virtual data node to the name node upon receiving the address request packet at the at least one virtual data node and upon the at least one virtual data node storing its own MAC address in the payload of the address packet as the data node MAC address; and

updating, by the name node, the node configuration based on the address packet received from the at least one virtual data node.

10. The method of claim 1, further comprising:

transmitting an address return request packet from the name node to each of the plurality of data nodes upon the name node storing its own MAC address in a payload of the address return request packet as the name node MAC address;

transmitting the address return packet from each of the plurality of data nodes to the name node upon receiving the address return request packet at each of the plurality of data nodes, and upon each of the plurality of data nodes storing a source MAC address in the payload of the address return packet as the source name node MAC address and storing the name node MAC address in the payload of the address return packet,

wherein the source MAC address is included in a header of the address return request packet and the name node MAC address is included in the payload of the address return request packet; and

determining, by the name node, the node configuration based on the address return packet received from each of the plurality of data nodes.

11. The method of claim 10, wherein the name node determines that the name node operates on the virtual machine when the source name node MAC address and the name node MAC address are different from each other in at least one of the address return packets received from the plurality of data nodes.

12. The method of claim 10, wherein determining the node configuration based on the address return packet received from each of the plurality of data nodes further comprises:

determining that the name node and a data node from among the plurality of data nodes corresponding to the source IP address included in the header of the address return packet operate on different physical hosts when the source name node MAC address and the name node MAC address are different from each other in the address return packet; and

determining that the name node and the data node corresponding to the source IP address included in the header of the address return packet operate on a same physical host when the source name node MAC address and the name node MAC address are the same as each other in the address return packet.

13. The method of claim 10, wherein the name node transmits the address return request packet, repeatedly, to each of the plurality of data nodes upon storing its own MAC address in the payload of the address return request packet as the name node MAC address, and updates the node configuration based on the address return packet received from each of the plurality of data nodes.

14. The method of claim 1, wherein the distributed processing system uses a HADOOP framework.

5. A method of operating a distributed processing system including a name node and a plurality of data nodes, comprising:

transmitting an address request packet from the name node to each of the plurality of data nodes;

transmitting an address packet from each of the plurality of data nodes to the name node upon receiving the address request packet at each of the plurality of data nodes and upon each of the plurality of data nodes storing its own media access control (MAC) address in a payload of the address packet as a data node MAC address;

determining, by the name node, a node configuration of the distributed processing system based on the address packet received from each of the plurality of data nodes; and

processing data using a selected data node from among the plurality of data nodes based on the node configuration,

wherein determining the node configuration comprises;

reading a source MAC address and a source internet protocol (IP) address from a header of the address packet, and the data node MAC address from the payload of the address packet; and

determining whether a data node from among the plurality of data nodes corresponding to the source IP address operates on a virtual machine by comparing the source MAC address and the data node MAC address.

6. The method of claim 5, wherein the name node determines that the data node corresponding to the source IP address operates on the virtual machine when the source MAC address and the data node MAC address are different from each other.

7. The method of claim 5, wherein determining the node configuration based on the address packet received from each of the plurality of data nodes further comprises:

determining whether the data node corresponding to the source IP address operates on a physical host corresponding to the source MAC address.

8. The method of claim 7, wherein the name node determines that data nodes from among the plurality of data nodes corresponding to source IP addresses included in address packets having a same source MAC address operate on a same physical host.

15. A method of operating a distributed processing system including a name node and a plurality of data nodes, comprising:

transmitting a multiple address return request packet from the name node to each of the plurality of data nodes upon the name node storing its own media access control (MAC) address in a payload of the multiple address return request packet as a name node MAC address;

transmitting a multiple address return packet from each of the plurality of data nodes to the name node upon receiving the multiple address return request packet at each of the plurality of data nodes, upon each of the plurality of data nodes storing a source MAC address in a payload of the multiple address return packet as a source name node MAC address, upon each of the plurality of data nodes storing the name node MAC address in the payload of the multiple address return packet, and upon each of the plurality of data nodes storing its own MAC address in the payload of the multiple address return packet as a data node MAC address,

wherein the source MAC address is included in a header of the multiple address return request packet and the name node MAC address is included in the payload of the multiple address return request packet;

determining, by the name node, a node configuration of the distributed processing system based on the multiple address return packet received from each of the plurality of data nodes; and

processing data using a selected data node selected from among the plurality of data nodes based on the node configuration.

16. The method of claim 15, wherein determining the node configuration comprises:

reading the source name node MAC address, the name node MAC address, and the data node MAC address from the payload of the multiple address return packet, a source internet protocol (IP) address from a header of the multiple address return packet, and a source MAC address from the header of the multiple address return packet as a source data node MAC address;

determining whether the name node operates on a virtual machine by comparing the source name node MAC address and the name node MAC address; and

determining whether a data node corresponding to the source IP address operates on the virtual machine by comparing the source data node MAC address and the data node MAC address.

17. The method of claim 16, wherein the name node determines that the name node operates on the virtual machine when the source name node MAC address and the name node MAC address are different from each other in at least one of the multiple address return packets received from the plurality of data nodes, and determines that the data node corresponding to the source IP address operates on the virtual machine when the source data node MAC address and the data node MAC address are different from each other.

18. The method of claim 16, wherein determining the node configuration further comprises:

determining that the name node and the data node corresponding to the source IP address operate on different physical hosts when the source name node MAC address and the name node MAC address are different from each other in the multiple address return packet, wherein the source IP address is included in the header of the multiple address return packet;

determining that the name node and the data node corresponding to the source IP address operate on a same physical host when the source name node MAC address and the name node MAC address are the same as each other in the multiple address return packet; and

determining whether the data node corresponding to the source IP address operates on a physical host corresponding to the source data node MAC address.

19. The method of claim 15, wherein the name node transmits the multiple address return request packet, repeatedly, to each of the plurality of data nodes upon storing its own MAC address in the payload of the multiple address return request packet as the name node MAC address, and updates the node configuration based on the multiple address return packet received from each of the plurality of data nodes.