Optimized peer-to-peer file transfers on a multi-node computer system

• US 8,856,275 B2
• Filed: 03/06/2013
• Issued: 10/07/2014
• Est. Priority Date: 11/12/2007
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Barsness, Eric L., Darrington, David L., Randles, Amanda, Santosuosso, John M.
• Primary Examiner(s): Kim, Hee Soo
• Application Number:
  US13/787,728
• Publication Number:
  US 20130191504A1
• Time in Patent Office: 580 Days
• Field of Search: 709/217, 709/219, 709/220
• US Class Current: 709/217
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/104 Peer-to-peer [P2P] networks
• CPC Class Codes: H04L 67/1063 Discovery through centralising entities
• CPC Class Codes: H04L 67/1076 Resource dissemination mechanisms or network resource keeping policies for optimal resource availability in the overlay network

Abstract

A method and apparatus performs peer-to-peer file transfers on a High Performance Computing (HPC) cluster such as a Beowulf cluster. A peer-to-peer file tracker (PPFT) allows operating system, application and data files to be moved from a pre-loaded node to another node of the HPC cluster. A peer-to-peer (PTP) client is loaded into the nodes to facilitate PTP file transfers to reduce loading on networks, network switches and file servers to reduce the time needed to load the nodes with these files to increase overall efficiency of the multi-node computing system. The selection of the nodes participating in file transfers can be based on network topology, network utilization, job status and predicted network/computer utilization. This selection can be dynamic, changing during the file transfers as resource conditions change. The policies used to choose resources can be configured by an administrator.

Claims

1. A multi-node computer cluster comprising:

a plurality of nodes that each comprise a processor and memory;

a network with network switches connecting the plurality of nodes;

a service node connected to the plurality of nodes, wherein the service node includes a resource manager that manages and monitors storage and network resources used by the system, and a scheduler that handles allocating and scheduling work and data placement on the compute nodes, wherein the resource manager and the resource scheduler provide resource attributes to dynamically adjust which nodes of the plurality of nodes participate in a peer-to-peer file transfer;

a peer-to-peer file tracker in the service node that manages the plurality of nodes to accomplish a the peer-to-peer file transfer between at least two of the plurality of nodes over the network, wherein the peer-to-peer transfer copies an operating system kernel from a source node to a destination node on the cluster in a process of booting the destination node;

a peer-to-peer client residing in the at least two of the plurality of nodes, where the peer-to-peer client manages data flow in the peer-to-peer file transfer, wherein the peer-to-peer client uses the resource attributes from the resource manager and from the scheduler received in a file tracker update sent from the peer-to-peer file tracker to dynamically adjust which nodes of the plurality of nodes participate in the peer-to-peer file transfer while the file transfer is in progress by restarting segments of the transfer from a new source node;

policies for the resource attributes that are set by a system administrator to indicate how to manage the peer-to-peer file transfer;

wherein the resource attributes include the following;

network topology, network utilization, network switch loading, file server loading, job status and historical information related to the resource attributes; and

wherein the peer-to-peer client dynamically adjusts the nodes participating in the peer-to-peer file transfer while the file transfer is in progress.

2. The multi-node computer cluster of claim 1 wherein the multi-node computer cluster is diskless cluster computer system with the network being a private network.

3. A computer-readable article of manufacture comprising:

a peer-to-peer file tracker for execution in a service node connected to a plurality of nodes in a computer cluster, wherein the service node includes a resource manager that manages and monitors storage and network resources used by the system, and a scheduler that handles allocating and scheduling work and data placement on the compute nodes, wherein the resource manager and the resource scheduler provide resource attributes to dynamically adjust which nodes of the plurality of nodes participate in a peer-to-peer file transfer, the peer-to-peer tracker managing the plurality of nodes to accomplish a the peer-to-peer file transfer between at least two of the plurality of nodes over a private network, wherein the peer-to-peer transfer copies an operating system kernel from a source node to a destination node on the cluster in a process of booting the destination node;

a peer-to-peer client residing in the at least two of the plurality of nodes, where the peer-to-peer client manages data flow in the peer-to-peer transfer, wherein the peer-to-peer client uses the resource attributes from the resource manager and from the scheduler received in a file tracker update sent from the peer-to-peer file tracker and in response to the file tracker update the peer-to-peer client dynamically adjusts nodes participating in the file transfer;

policies for the resource attributes that are set by a system administrator to indicate how to manage the peer-to-peer file transfer;

wherein the resource attributes include the following;

network topology, network utilization, network switch loading, file server loading, job status and historical information related to the resource attributes;

wherein the peer-to-peer client dynamically adjusts the nodes participating in the peer-to-peer file transfer while the file transfer is in progress; and

non-transient computer recordable media bearing the peer-to-peer file tracker and the peer-to-peer client.

4. The article of manufacture of claim 3 wherein the peer-to-peer client determines the nodes to participate in the peer-to-peer file transfer by selecting a plurality of file server records to send to the peer-to-peer client that contain information for the selected nodes.