OPTIMIZED PEER-TO-PEER FILE TRANSFERS ON A MULTI-NODE COMPUTER SYSTEM

• US 20130198343A1
• Filed: 03/06/2013
• Published: 08/01/2013
• Est. Priority Date: 11/12/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Barsness, Eric L., Darrington, David L., Peters, Amanda, Santosuosso, John M.
• Granted Patent: US 8,856,286 B2
• Time in Patent Office: Days
• US Class Current: 709/219
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/104 Peer-to-peer [P2P] networks
• CPC Class Codes: H04L 67/1063 Discovery through centralising entities
• CPC Class Codes: H04L 67/1076 Resource dissemination mechanisms or network resource keeping policies for optimal resource availability in the overlay network

Abstract

A method and apparatus performs peer-to-peer file transfers on a High Performance Computing (HPC) cluster such as a Beowulf cluster. A peer-to-peer file tracker (PPFT) allows operating system, application and data files to be moved from a pre-loaded node to another node of the HPC cluster. A peer-to-peer (PTP) client is loaded into the nodes to facilitate PTP file transfers to reduce loading on networks, network switches and file servers to reduce the time needed to load the nodes with these files to increase overall efficiency of the multi-node computing system. The selection of the nodes participating in file transfers can be based on network topology, network utilization, job status and predicted network/computer utilization. This selection can be dynamic, changing during the file transfers as resource conditions change. The policies used to choose resources can be configured by an administrator.

Claims

1. A computer implemented method for a peer-to-peer file transfer in a plurality of nodes in a multi-node computer cluster, the method comprising the steps of:

reading file server records for a requested file to peer-to-peer transfer between the plurality of nodes in the multi-node computer cluster;

selecting a plurality of participating nodes from the plurality of nodes for the peer-to-peer transfer based on resource attributes;

transferring file segments of the requested file from the plurality of participating nodes based on the file server records;

a peer-to-peer client dynamically adjusting the nodes participating in the file transfer in response to a file tracker update with updated resource attributes by restarting segments of the transfer from a new source node; and

monitoring network utilization of networks of the multi-node computer system and where high utilization of a network is detected disabling peer-to-peer traffic for nodes using the detected network.

2. The computer implemented method of claim 1 wherein the peer-to-peer transfer copies an operating system kernel from a source node to a destination node on the cluster in a process of booting the destination node.

3. The computer implemented method of claim 1 wherein the resource attributes include the following:

network topology, network utilization, network switch loading, file server loading, job status and historical information related to the resource attributes.

4. The computer implemented method of claim 1 wherein the resource attributes are used in conjunction with policies determined by a system administrator.

5. The computer implemented method of claim 1 wherein the peer-to-peer client dynamically adjusts the nodes participating in the file transfer while the file transfer is in progress.