Optimized peer-to-peer file transfers on a multi-node computer system
First Claim
1. A multi-node computer cluster comprising:
- a plurality of nodes that each comprise a processor and memory;
a network with network switches connecting the plurality of nodes;
a service node connected to the plurality of nodes;
a peer-to-peer file tracker in the service node that manages the plurality of nodes to accomplish a peer-to-peer file transfer between at least two of the plurality of nodes over the network;
a peer-to-peer client residing in the at least two of the plurality of nodes, where the peer-to-peer client manages data flow in the peer-to-peer file transfer; and
wherein the peer-to-peer client uses resource attributes from a resource manager and from a scheduler received in a file tracker update to dynamically adjust which nodes of the plurality of nodes participate in the file transfer while the file transfer is in progress by restarting segments of the transfer from a new source node.
1 Assignment
0 Petitions
Accused Products
Abstract
A method and apparatus performs peer-to-peer file transfers on a High Performance Computing (HPC) cluster such as a Beowulf cluster. A peer-to-peer file tracker (PPFT) allows operating system, application and data files to be moved from a pre-loaded node to another node of the HPC cluster. A peer-to-peer (PTP) client is loaded into the nodes to facilitate PTP file transfers to reduce loading on networks, network switches and file servers to reduce the time needed to load the nodes with these files to increase overall efficiency of the multi-node computing system. The selection of the nodes participating in file transfers can be based on network topology, network utilization, job status and predicted network/computer utilization. This selection can be dynamic, changing during the file transfers as resource conditions change. The policies used to choose resources can be configured by an administrator.
5 Citations
17 Claims
-
1. A multi-node computer cluster comprising:
-
a plurality of nodes that each comprise a processor and memory; a network with network switches connecting the plurality of nodes; a service node connected to the plurality of nodes; a peer-to-peer file tracker in the service node that manages the plurality of nodes to accomplish a peer-to-peer file transfer between at least two of the plurality of nodes over the network; a peer-to-peer client residing in the at least two of the plurality of nodes, where the peer-to-peer client manages data flow in the peer-to-peer file transfer; and wherein the peer-to-peer client uses resource attributes from a resource manager and from a scheduler received in a file tracker update to dynamically adjust which nodes of the plurality of nodes participate in the file transfer while the file transfer is in progress by restarting segments of the transfer from a new source node. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A computer implemented method for a peer-to-peer file transfer in a plurality of nodes in a multi-node computer cluster, the method comprising the steps of:
-
reading file server records for a requested file to peer-to-peer transfer between the plurality of nodes in the multi-node computer cluster; selecting a plurality of participating nodes from the plurality of nodes for the peer-to-peer transfer based on resource attributes; transferring file segments of the requested file from the plurality of participating nodes based on the file server records; a peer-to-peer client dynamically adjusting the nodes participating in the file transfer in response to a file tracker update with updated resource attributes by restarting segments of the transfer from a new source node. - View Dependent Claims (9, 10, 11)
-
-
12. A computer-readable article of manufacture comprising:
-
a peer-to-peer file tracker for execution in a service node connected to a plurality of nodes in a computer cluster, the peer-to-peer tracker managing the plurality of nodes to accomplish a peer-to-peer file transfer between at least two of the plurality of nodes over a private network; a peer-to-peer client residing in the at least two of the plurality of nodes, where the peer-to-peer client manages data flow in the peer-to-peer transfer; wherein the peer-to-peer client uses resource attributes from a resource manager and from a scheduler received in a file tracker update and in response to the file tracker update the peer-to-peer client dynamically adjusts nodes participating in the file transfer; and tangible computer recordable media bearing the peer-to-peer file tracker and the peer-to-peer client. - View Dependent Claims (13, 14, 15, 16, 17)
-
Specification