Modular architecture for extreme-scale distributed processing applications

• US 9,330,055 B2
• Filed: 06/04/2013
• Issued: 05/03/2016
• Est. Priority Date: 06/04/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Butt, Ali R., Sarkar, Prasenjit
• Primary Examiner(s): Vostal, O. C.
• Application Number:
  US13/909,767
• Publication Number:
  US 20140359050A1
• Time in Patent Office: 1,064 Days
• Field of Search: 709/214
• US Class Current: 1/1
• CPC Class Codes: G06F 15/17331 Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0655 Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals

Abstract

Embodiments of the present invention relate to a new data center architecture that provides for efficient processing in distributed analytics applications. In one embodiment, a distributed processing node is provided. The node comprises a plurality of subnodes. Each subnode includes at least one processor core operatively connected to a memory. A first interconnect operatively connects each of the plurality of subnodes within the node. A second interconnect operably connects each of the plurality of subnodes to a storage. A process runs on a first of the plurality of subnodes, the process being operative to retrieve data from the memory of the first subnode. The process interrogates the memory of the first subnode for requested data. If the requested data is not found in the memory of the first subnode, the process interrogates the memory of at least one other subnode of the plurality of subnodes via the first interconnect. If the requested data is found in the memory of the other subnode, the process copies the requested data to the memory of the first subnode. If the requested data is not found in the memory of the first subnode or the memories of at least one subnode of the plurality of subnodes, the process interrogates the storage via the second interconnect.

Claims

1. A system comprising:

a distributed processing node comprising a plurality of subnodes, each subnode including;

at least one processor core operatively connected to a memory;

a first interconnect operatively connected to each of the plurality of subnodes;

a second interconnect operatively connected to each of the plurality of subnodes and to a storage, the storage comprising a first storage unit and a second storage unit, the second storage unit having lower access time and latency than the first storage unit;

a process running on a first of the plurality of subnodes, the process being operative to retrieve data from the memory of the first subnode;

wherein;

the process interrogates the memory of the first subnode for requested data;

if the requested data is not found in the memory of the first subnode, the process interrogates the memory of at least one other subnode of the plurality of subnodes via the first interconnect;

if the requested data is found in the memory of the other subnode, the process copies the requested data to the memory of the first subnode; and

if the requested data is not found in the memory of the first subnode or the memory of at least another subnode of the plurality of subnodes, the process interrogates the storage via the second interconnect;

a storage manager operative to allocate data between the first and second storage units based on access patterns, the storage manager preferentially relocating non-sequentially accessed data to the second storage unit from the first storage unit.

2. The system of claim 1, wherein the distributed processing node is one of a plurality of distributed processing nodes forming a distributed processing cluster, each of the plurality of distributed processing nodes being operatively connected to a third interconnect.

3. The system of claim 1, wherein the first storage unit comprises a hard disk drive and the second storage unit comprises a solid-state drive, the hard disk drive including sequential data and the solid-state drive including non-sequential data.

4. The system of claim 1, wherein the first interconnect has higher bandwidth than the second interconnect.

5. The system of claim 2, wherein the first interconnect has higher bandwidth than the third interconnect.

6. The system of claim 1, wherein the memories of each of the plurality of subnodes form a cooperative cache or a shared memory.

7. The system of claim 1, wherein the storage is accessed via a Hadoop Distributed File System.

8. The system of claim 1, wherein the distributed processing node is a Hadoop node.

9. The system of claim 2, wherein the cluster is a Hadoop cluster.

10. The system of claim 1, wherein the first interconnect is overprovisioned.

11. The system of claim 1, wherein the memories of the plurality of subnodes are managed by Memcached.

12. The system of claim 1, wherein if the requested data is found in the storage, the process copies the requested data to the memory of the first subnode.

13. The system of claim 1, wherein the process is further operative to propagate changed data between the memories of the plurality of subnodes and the storage.

14. The system of claim 1, further comprising a task scheduler operative to allocate the process to the subnode of the plurality of subnodes that has the most requested data in its memory.

15. The system of claim 1, wherein the first storage unit comprises a hard disk drive and the second storage unit comprises a solid-state drive, the hard disk drive including sequential data and the solid-state drive including non-sequential data.

16. A method comprising:

receiving a task at a first distributed processing node;

allocating the task to a first subnode of the first distributed processing node, the subnode including at least one processor core operatively connected to a memory;

determining data requested by the task;

interrogating the memory of the first subnode for the requested data;

if the requested data is not found in the memory of the first subnode, interrogating the memory of at least another subnode of the first distributed processing node via a first interconnect;

if the requested data is found in the memory of the other subnode, copying the requested data from the memory of the other subnode to the memory of the first subnode;

if the requested data is not found in the memory of the first subnode or the memory of at least another subnode of the first distributed processing node, interrogating a storage via a second interconnect, the storage comprising a first storage unit and a second storage unit, the second storage unit having lower access time and latency than the first storage unit; and

processing the task on the at least one processor core of the first subnode;

allocating data between the first and second storage units based on access patterns, preferentially relocating non-sequentially accessed data to the second storage unit from the first storage unit.

17. The method of claim 16, wherein the first distributed processing node is one of a plurality of distributed processing nodes forming a distributed processing cluster, each of the plurality of distributed processing nodes being operatively connected to a third interconnect.

18. A computer program product for distributed data processing, the computer program product comprising a non-transitory computer readable storage medium having program code embodied therewith, the program code executable by a processor to:

receive a task at a first distributed processing node;

allocate the task to a first subnode of the first distributed processing node, the subnode including at least one processor core operatively connected to a memory;

determine data requested by the task;

interrogate the memory of the first subnode for the requested data;

if the requested data is not found in the memory of the first subnode, interrogate the memory of at least another subnode of the first distributed processing node via a first interconnect;

if the requested data is found in the memory of the other subnode, copy the requested data from the memory of the other subnode to the memory of the first subnode;

if the requested data is not found in the memory of the first subnode or the memory of at least another subnode of the first distributed processing node, interrogate a storage via a second interconnect, the storage comprising a first storage unit and a second storage unit, the second storage unit having lower access time and latency than the first storage unit; and

process the task on the at least one processor core of the first subnode;

allocate data between the first and second storage units based on access patterns, preferentially relocating non-sequentially accessed data to the second storage unit from the first storage unit.

19. The computer program product of claim 18, wherein the first distributed processing node is one of a plurality of distributed processing nodes forming a distributed processing cluster, each of the plurality of distributed processing nodes being operatively connected to a third interconnect.

20. The computer program product of claim 18, wherein the first distributed processing node is one of a plurality of distributed processing nodes forming a distributed processing cluster, each of the plurality of distributed processing nodes being operatively connected to a third interconnect.