Scalable method for optimizing information pathway

• US 10,169,083 B1
• Filed: 06/28/2012
• Issued: 01/01/2019
• Est. Priority Date: 03/30/2012
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Florissi, Patricia G. S., Vijendra, Sudhir
• Primary Examiner(s): Bullock, Jr., Lewis A
• Assistant Examiner(s): Lu, Kevin X
• Application Number:
  US13/535,760
• Time in Patent Office: 2,378 Days
• CPC Class Codes: G06F 2209/485 Resource constraint
• CPC Class Codes: G06F 2209/5015 Service provider selection
• CPC Class Codes: G06F 2209/5017 Task decomposition
• CPC Class Codes: G06F 2209/5019 Workload prediction
• CPC Class Codes: G06F 2209/509 Offload
• CPC Class Codes: G06F 9/4843 by program, e.g. task dispatcher, supervisor, operating system
• CPC Class Codes: G06F 9/50 Allocation of resources, e.g. of the central processing unit [CPU]
• CPC Class Codes: G06F 9/5033 considering data affinity
• CPC Class Codes: G06F 9/5044 considering hardware capabilities
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04W 28/00 Network traffic management; Network resource management

Abstract

A method, system and process for receiving a request for task execution at a central processing node for the world wide data; wherein the central processing node is connected to sub-processing nodes, dividing the request for task execution to be distributed to a set of the sub-processing nodes; wherein the set of processing nodes manages a portion of the world wide data, transmitting to each of the set of the sub-processing nodes the respective portion of the divided task execution.

Claims

1. An apparatus comprising:

a receiving module configured to receive a request for task execution at a central processing node for worldwide data;

wherein the central processing node is connected to sub-processing network nodes;

wherein the sub-processing network nodes are grouped into clusters;

wherein each cluster has a distributed file system mapping out network nodes for each respective cluster;

wherein each cluster stores a subset of the worldwide data; and

wherein each cluster is enabled to use the network nodes of the cluster to perform parallel processing;

wherein the central processing node is communicatively coupled to a global distributed file system that maps over each of the cluster'"'"'s distributed file systems to enable orchestration between the clusters;

a dividing module configured to divide by a worldwide job tracker the request for task execution into worldwide task trackers to be distributed to sub-processing network nodes of the clusters;

wherein the network sub-nodes manages a portion of the worldwide data for each respective cluster;

wherein each worldwide task tracker maintains records of sub-activities executed as part of the worldwide job;

a transmitting module configured to transmit to each of the sub-processing network nodes for each respective cluster the respective portion of the divided task execution by assigning each worldwide task tracker corresponding to the respective portion to the respective each cluster; and

a leveraging module configured to generate a graph layout of data pathways, the pathways calculated based upon physical distance between the processing nodes and bandwidth constraints, the leveraging module further configured to distribute task execution based upon the processing power of the processing nodes, graph layout, and the size of data processed by the sub-processing network nodes to reduce data movement between the central processing node and the sub-processing nodes.

2. The apparatus of claim 1 further comprising:

an execution module, configured to coordinate, for each of the sub-processing network nodes for each respective cluster, execution of the respective portion of the divided task execution between the respective cluster of the data network nodes;

a consolidating module configured to consolidate, at each of the sub-processing network nodes for each respective cluster, the coordinated execution of the respective portion of the divided task execution;

a transmitter module configured to transmit the consolidated result of the respective portion of the divided task execution from each of the sub-processing network nodes for each respective cluster to the central processing network nodes; and

a consolidater module configured to consolidate the results of divided task execution from each of the sub-processing network nodes for each respective cluster.

3. The apparatus of claim 2 wherein the sub-processing network nodes for each respective cluster are located on a path between the data and the central processing node.

4. The apparatus of claim 3 further comprising:

a designating configured to designate, for each sub-processing network node for each respective cluster, a second set of sub-processing network nodes between the central processing network node and the data;

a divider module configured to divide the request for task execution to be distributed to the second set of the sub-processing network nodes;

a coordinate module configured to coordinate, for each of the second set of sub-processing network nodes, execution of the respective portion of the divided task execution between a respective cluster; and

consolidating, at each of the second set of sub-processing network nodes, the coordinated execution of the respective portion of the divided task execution.

5. A computer program product comprising:

a non-transitory computer readable medium encoded with computer executable program code, the code configured to enable the execution of;

receiving a request for task execution at a central processing node for worldwide data;

wherein the central processing node is connected to sub-processing network nodes;

wherein the sub-processing network nodes are grouped into clusters;

wherein each cluster has a distributed file system mapping out network nodes for each respective cluster;

wherein each cluster stores a subset of the worldwide data; and

wherein each cluster is enabled to use the network nodes of the cluster to perform parallel processing;

wherein the central processing node is communicatively coupled to a global distributed file system that maps over each of the cluster'"'"'s distributed file systems to enable orchestration between the clusters;

dividing by a worldwide job tracker the request for task execution into worldwide task trackers to be distributed sub-processing network nodes of the clusters;

wherein the sub- processing network nodes manages a portion of the worldwide data for each respective cluster;

wherein each worldwide task tracker maintains records of sub-activities executed as part of the worldwide job;

transmitting to each of the sub-processing network nodes for each respective cluster the respective portion of the divided task execution by assigning each worldwide task tracker corresponding to the respective portion to the respective each cluster;

generating a graph layout of data pathways, the pathways calculated based upon physical distance between the processing nodes and bandwidth constraints; and

distributing task execution among processing nodes based upon the graph layout, the processing power of the processing nodes, and the size of data processed by the sub-processing network nodes to reduce the amount and duration of data movement between the central processing node and the sub-processing nodes.

6. The computer program product of claim 5 the code further configured to enable:

for each of the sub-processing network nodes, coordinating execution of the respective portion of the divided task execution between the respective cluster of the data network nodes;

consolidating, at each of the subprocessing network nodes for each respective cluster, the coordinated execution of the respective portion of the divided task execution;

transmitting the consolidated result of the respective portion of the divided task execution from each of the sub-processing network nodes for each respective cluster to the central processing nodes; and

consolidating the results of divided task execution from each of the sub-processing network nodes for each respective cluster.

7. The method of claim 6 wherein the sub-processing network nodes for each respective cluster are located on a path between the data and the central processing node.

8. The computer program product of claim 7 the code further configured to enable:

designating, for each sub-processing network node of the sub-processing network nodes for each respective cluster, a second set of sub-processing network nodes between the central processing node and the data;

dividing the request for task execution to be distributed to the second set of the sub-processing network nodes;

for each of the second set of sub-processing network nodes, coordinating execution of the respective portion of the divided task execution between a respective cluster; and

consolidating, at each of the second set of sub-processing network nodes, the coordinated execution of the respective portion of the divided task execution.

9. The computer program product of claim 7 the code further configured to enable:

leveraging processing power of the sub-processing network nodes for each respective cluster to minimize data flow to the worldwide data node.

10. The computer program product of claim 5 wherein the dividing the request for task execution uses the global distributed file system to divide task.

11. The computer program product of claim 5 wherein transmitting to each of the sub-processing nodes for each respective cluster uses the global distributed file system to find the location of each sub-processing node.

12. A computer method for processing execution of a task in a worldwide data environment comprising:

receiving a request for task execution at a central processing node for worldwide data;

wherein the central processing node is connected to sub-processing network nodes;

wherein the sub-processing network nodes are grouped into clusters;

wherein each cluster has a distributed file system mapping out network nodes for each respective cluster;

wherein each cluster stores a subset of the worldwide data; and

wherein each cluster is enabled to use the network nodes of the cluster to perform parallel processing;

wherein the central processing node is communicatively coupled to a global distributed file system that maps over each of the cluster'"'"'s distributed file systems to enable orchestration between the clusters;

dividing by a worldwide job tracker the request for task execution to be distributed to the sub-processing network nodes of the clusters;

wherein the processing network nodes manages a portion of the worldwide data for each respective cluster;

wherein each worldwide task tracker maintains records of sub-activities executed as part of the worldwide job;

transmitting to each of the sub-processing network nodes for each respective cluster the respective portion of the divided task execution by assigning each worldwide task tracker corresponding to the respective portion to the respective each cluster;

generating a graph layout of data pathways, the pathways calculated based upon physical distance between the processing nodes and bandwidth constraints; and

distributing task execution based upon the graph layout, the processing power of the processing nodes, proximity between the central processing node and the sub-processing nodes, and the size of data processed by the sub-processing network nodes to reduce the amount and duration of data movement between the central processing node and the sub-processing nodes.

13. The method of claim 12 further comprising:

for each of the sub-processing network nodes, coordinating execution of the respective portion of the divided task execution between the respective cluster of the data network nodes;

consolidating, at each of the sub-processing network nodes for each respective cluster, the coordinated execution of the respective portion of the divided task execution;

transmitting the consolidated result of the respective portion of the divided task execution from each of the sub-processing network nodes for each respective cluster to the central processing nodes; and

consolidating the results of divided task execution from each of the sub-processing network nodes for each respective cluster.

14. The method of claim 13 wherein the sub-processing network nodes for each respective cluster are located on a path between the data and the central processing node.

15. The method of claim 14 further comprising:

designating, for each sub-processing network node of the sub-processing network nodes for each respective cluster, a second set of sub-processing network nodes between the central processing node and the data;

dividing the request for task execution to be distributed to the second set of the sub-processing network nodes;

for each of the second set of sub-processing network nodes, coordinating execution of the respective portion of the divided task execution between a respective cluster; and

consolidating, at each of the second set of sub-processing network nodes, the coordinated execution of the respective portion of the divided task execution.

16. The method of claim 13 wherein each cluster is a Hadoop cluster.

17. The method of claim 12 wherein each cluster has an intermediate worldwide task tracker to track the divided task at each cluster.

18. The method of claim 12 wherein a worldwide data directory is layered over the clusters to create a namespace for the worldwide data.

19. The method of claim 18 further comprising:

automatically browsing each cluster to create the worldwide directory.

20. The method of claim 18 wherein a Worldwide Job Tracker decomposes a Worldwide Job into Worldwide Tasks tracked by the intermediate worldwide task tracker of each cluster.