Worldwide distributed job and tasks computational model

• US 9,747,127 B1
• Filed: 06/28/2012
• Issued: 08/29/2017
• Est. Priority Date: 03/30/2012
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Florissi, Patricia G. S., Vijendra, Sudhir
• Primary Examiner(s): Patel, Hiren
• Application Number:
  US13/535,814
• Time in Patent Office: 1,888 Days
• Field of Search: None
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 2209/5017 Task decomposition
• CPC Class Codes: G06F 9/46 Multiprogramming arrangements
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals
• CPC Class Codes: G06F 9/5072 Grid computing
• CPC Class Codes: G06F 9/54 Interprogram communication
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

Example embodiments of the present invention relate to a method, an apparatus, and a computer program product for performing file system activities across administrative boundaries between a plurality of file systems. The method includes receiving a worldwide job to perform on a plurality of file systems, managing the worldwide job, and receiving results of the worldwide job from the plurality of file systems.

Claims

1. A method comprising:

receiving a worldwide job configured to be performed on a heterogeneous worldwide distributed file system across administrative boundaries between data sets stored on and associated with a respective distribute file system of a cluster of a plurality of nodes;

splitting, by a worldwide job tracker, the worldwide job into worldwide tasks configured to be performed on respective data sets;

assigning, by the worldwide job tracker, the worldwide tasks to worldwide task trackers at the respective clusters, wherein each worldwide task tracker maintains records of all sub-activities executed as part of the of the worldwide job and edit logs to capture the activities;

submitting the worldwide tasks as jobs from the worldwide task trackers to job trackers at the respective clusters;

splitting, by the job trackers, each of the jobs into tasks each configured to be performed on a portion of the data set stored on the distributed file system of the respective cluster;

assigning the tasks to task trackers at the respective clusters; and

performing parallel processing of the worldwide job on the heterogeneous worldwide distributed file system by executing the tasks across nodes of the respective clusters, wherein the world wide task tracker is a slave of the world wide job tracker, and further wherein the world wide task tracker is a master of the job tracker.

2. The method of claim 1 wherein the worldwide job comprises at least one worldwide task.

3. The method of claim 2

wherein the worldwide job represents an activity that may be executed on a domain;

and

wherein each worldwide task of the worldwide job executes a function on a respective part of the domain.

4. The method of claim 2 further comprising:

submitting each worldwide task of the worldwide job for execution; and

receiving results of execution of each worldwide task as results of the worldwide job.

5. The method of claim 4 wherein each worldwide task of the worldwide job comprises at least one job.

6. The method of claim 5

wherein submitting each worldwide task of the worldwide job for execution comprises submitting each job comprising each respective worldwide task for execution;

and

wherein receiving results of execution of each worldwide task as results of the worldwide job comprises receiving results of execution of each respective job of each respective worldwide task as results of the worldwide job.

7. The method of claim 6 wherein each job comprises a plurality of tasks.

8. The method of claim 7

wherein submitting each job comprising each respective worldwide task for execution comprises submitting each task comprising each job for execution;

and

wherein receiving results of execution of each respective job of each respective worldwide task as results of the worldwide job comprises receiving results of each respective task as results of each respective job.

9. A system comprising:

one or more processors; and

memory storing computer program code that when executed on one or more of the one or more processors causes the system to perform the operation of;

receiving a worldwide job configured to be performed on a heterogenous worldwide distributed file system across administrative boundaries between data sets stored on and associated with a respective distribute file system of a cluster of a plurality of nodes;

splitting, by a worldwide job tracker, the worldwide job into worldwide tasks configured to be performed on respective data sets;

assigning, by the worldwide job tracker, the worldwide tasks to worldwide task trackers at the respective clusters, wherein each worldwide task tracker maintains records of all sub-activities executed as part of the of the worldwide job and edit logs to capture the activities; and

submitting the worldwide tasks as jobs from the worldwide task trackers to job trackers at the respective clusters;

splitting, by the job trackers, each of the jobs into tasks each configured to be performed on a portion of the data set stored on the distributed file system of the respective cluster;

assigning the tasks to task trackers at the respective clusters; and

performing parallel processing of the worldwide job on the heterogeneous worldwide distributed file system by executing the tasks across nodes of the respective clusters, wherein the world wide task tracker is a slave of the world wide job tracker, and further wherein the world wide task tracker is a master of the job tracker.

10. The system of claim 9 wherein the worldwide job comprises at least one worldwide task.

11. The system of claim 10

wherein the worldwide job represents an activity that may be executed on a domain;

and

wherein each worldwide task of the worldwide job executes a function on a respective part of the domain.

12. The system of claim 10 further comprising:

submitting each worldwide task of the worldwide job for execution; and

receiving results of execution of each worldwide task as results of the worldwide job.

13. The system of claim 12 wherein each worldwide task of the worldwide job comprises at least one job.

14. The system of claim 13

wherein submitting each worldwide task of the worldwide job for execution comprises submitting each job comprising each respective worldwide task for execution;

and

wherein receiving results of execution of each worldwide task as results of the worldwide job comprises receiving results of execution of each respective job of each respective worldwide task as results of the worldwide job.

15. The system of claim 14 wherein each job comprises a plurality of tasks.

16. The system of claim 15

wherein submitting each job comprising each respective worldwide task for execution comprises submitting each task comprising each job for execution;

and

wherein receiving results of execution of each respective job of each respective worldwide task as results of the worldwide job comprises receiving results of each respective task as results of each respective job.

17. A computer program product having a non-transitory computer readable storage medium with instructions encoded thereon that, when executed by a processor of a computer, causes the computer to perform the operations of:

receiving a worldwide job configured to be performed on a heterogeneous worldwide distributed file system across administrative boundaries between data sets stored on and associated with a respective distribute file system of a cluster of a plurality of nodes;

splitting, by a worldwide job tracker, the worldwide job into worldwide tasks configured to be performed on respective data sets;

assigning, by the worldwide job tracker, the worldwide tasks to worldwide task trackers at the respective clusters, wherein each worldwide task tracker maintains records of all sub-activities executed as part of the of the worldwide job and edit logs to capture the activities;

submitting the worldwide tasks as jobs from the worldwide task trackers to job trackers at the respective clusters;

splitting, by the job trackers, each of the jobs into tasks each configured to be performed on a portion of the data set stored on the distributed file system of the respective cluster;

assigning the tasks to task trackers at the respective clusters; and

performing parallel processing of the worldwide job on the heterogeneous worldwide distributed file system by executing the tasks across nodes of the respective clusters, wherein the world wide task tracker is a slave of the world wide job tracker, and further wherein the world wide task tracker is a master of the job tracker.

18. The computer program product of claim 17 wherein the worldwide job comprises at least one worldwide task.

19. The computer program product of claim 18 wherein each worldwide task of the worldwide job comprises at least one job.

20. The computer program product of claim 19 wherein each job comprises a plurality of tasks.