Maximizing processor utilization and minimizing network bandwidth requirements in throughput compute clusters

• US 20050060608A1
• Filed: 07/16/2004
• Published: 03/17/2005
• Est. Priority Date: 05/23/2002
• Status: Abandoned Application

• Current Assignee: eXludus Technologies, Inc.
• Original Assignee: eXludus Technologies, Inc.
• Inventors: Marchand, Benoit
• Application Number:
  US10/893,752
• Publication Number:
  US 20050060608A1
• Time in Patent Office: Days
• US Class Current: 714/18
• CPC Class Codes: H04L 12/1877 Measures taken prior to transmission
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

Exemplary methods and apparatus for improving speed, scalability, robustness and dynamism of data transfers and workload distribution to remote computers are provided. Computing applications, such as Genomics, Proteomics, Seismic, Risk Management require a priori or on-demand transfer of sets of files or other data to remote computers prior to processing taking place. The fully distributed data transfer and data replication protocol of the present invention permits transfers which minimize processing requirements on master transfer nodes by spreading work across the network and automatically synchronizing the enabling and disabling of job dispatch functions with workload distribution mechanisms to enable/disable job dispatch activities resulting in higher scalability than current methods, more dynamism and allowing fault-tolerance by distribution of functionality. Data transfers occur asynchronously to job distribution allowing full utilization of remote system resources to receive data for job queues while processing jobs for previously transferred data. Processor utilization is further increased as file accesses are local to systems and bear no additional network latencies that reduce processing efficiency.

Claims

1. A method comprising:

transferring data with a workload distribution mechanism between at least two computing devices using a transfer protocol; and

synchronizing workload distribution mechanisms with a synchronizer wherein job dispatch functions of at least two computing devices are enabled or disabled.

2. The method of claim 1 wherein the transfer protocol comprises a multicast protocol.

3. The method of claim 1 wherein the transfer protocol comprises a broadcast protocol.

4. The method of claim 1 wherein transferring data is used for transferring already transferred data from one of the at least two computing devices to a newly connected computing device.

5. The method of claim 1 wherein transferring data is used for completing interrupted data transfers.

6. The method of claim 1 wherein the transferred data comprises segments of a file.

7. The method of claim 1, further comprising recording received data and received jobs in a log at each computing device of said at least two computing devices.

8. The method of claim 1, further comprising performing a security check on a job description file to validate a request.

9. The method of claim 8 wherein validation comprises file access permissions.

10. The method of claim 8 wherein validation comprises execution permissions.

11. A computing device for transferring data and synchronizing workload distributions comprising:

a data transfer module configured for transferring data to a second computing device using a transfer protocol; and

a synchronization module configured for synchronizing work load distribution mechanisms and enabling or disabling a job dispatch function.

12. The computing device of claim 11 wherein the protocol comprises a broadcast protocol.

13. The computing device of claim 11 wherein the protocol comprises a multicast protocol.

14. The computing device of claim 11 further comprising a security module for performing a security check on a job description file to validate a request.

15. The computing device of claim 14 wherein the security module validates file access permissions.

16. The computing of claim 14 wherein the security module validates execution permissions.

17. A computer readable medium having embodied thereon a program, the program being executable by a machine to perform a method of transferring data and synchronizing workload distributions, the method comprising:

transferring data based on a data transfer phase between at least two computing devices using a transfer protocol; and

synchronizing workload distribution mechanisms based on a synchronization phase wherein job dispatch functions of at least two computing devices are enabled or disabled.

18. The computer readable medium of claim 17 wherein the computer readable medium is executed by an electronic appliance.

19. The computer readable medium of claim 18 wherein the electronic appliance is a personal computer.

20. The computer readable medium of claim 18 wherein the electronic appliance is a cellular phone.

21. The computer readable medium of claim 18 wherein the electronic appliance is a PDA.